Run federated searches

A federated search lets you search across specific datasets on multiple Splunk platform deployments. These deployments include your local Splunk platform deployment and remote Splunk platform deployments that you designate as federated providers. A federated search is processed partly on your local instance and partly on the federated providers. The remote and local results are then brought to the federated search head of your local Splunk platform deployment, where they are aggregated together to produce a final result set.

The federated search process includes search optimizations that help federated searches run as efficiently as possible. These optimizations filter results on the remote search heads, which reduces the amount of data that must be sent over the network and processed on the federated search head, improving overall search performance.

The experience of writing federated searches differs depending on whether you are searching over standard mode or transparent mode federated providers.

For an overview of federated search and federated search terminology, see About federated search.

General requirements for federated search

Before you can run a federated search, you must meet these requirements:

• Your local Splunk platform instance must have the federated search feature enabled in server.conf. If it is not enabled, your administrator must enable it.
• Your administrator must designate one or more remote Splunk platform deployments as federated providers. See Define a federated provider.

Search over a standard mode federated provider

When you run federated searches over a standard mode federated provider, you must follow a special search syntax. There are also restrictions specific to standard mode federated searches to be aware of.

Additional requirements for standard mode federated search

• Your administrator must create one or more federated indexes on your local federated search head. Each federated index maps to a specific remote dataset on a federated provider. See Create a federated index.
• Your role must have permissions for the federated indexes on your local federated search head that you intend to search. See the section on federated indexes in Service accounts and federated search security.

Write standard mode federated searches

The basic syntax for a standard mode federated search differs depending on the type of remote dataset that you are referencing in the search. A federated search of an index requires different syntax than a federated search of a saved search dataset.

In both cases, you specify a federated index that you have defined on your federated search head. This federated index maps to a remote dataset on the federated provider such as an events index or saved search. See Create a federated index.

You can reference multiple federated indexes in a subsearch using Boolean operators such as AND and OR.

If your role has the admin_all_objects capability, you can view the federated indexes to which you have access and the remote datasets that those federated indexes map to on the Federated Indexes listing page at Settings > Federated Search > Federated Indexes. If your role does not have this capability, get the names of the federated indexes that you can search from your administrator.

Search a remote events index dataset

To search a remote events index dataset on a standard mode federated provider, use the search command in conjunction with a reference to a federated index.

search index=federated:<federated_index_for_remote_events_index>

The prefix federated: before the federated index name indicates that you are referring to a federated index, and <federated_index_for_remote_events_index> is the name of a federated index that maps to a remote index on a standard mode federated provider.

Search a remote saved search dataset

To search a remote saved search dataset on a standard mode federated provider, use the from command in conjunction with a reference to a federated index.

| from federated:<federated_index_for_remote_saved_search>

The prefix federated: before the index name indicates that you are referring to a federated index, and <federated_index_for_saved_search> is the name of a federated index that maps to a saved search dataset on a federated provider.

Saved search datasets always use knowledge objects from the remote search head on the federated provider, even when Local knowledge objects is enabled for the federated provider.

Example: Simple search of a large remote events index dataset

Say you have a federated index on your local search head named airline_flights_SF. This federated index is mapped to an events index on a remote Splunk platform deployment that is defined as a federated provider to your Splunk platform deployment. The remote events index contains 2,000,000 events relating to airline flight departures and arrivals at a San Francisco Airport. To return this large event set from the remote deployment, run this search:

index = federated:airline_flights_SF

As a best practice, run a simple search like this on a federated index that you're unfamiliar with. This practice helps you determine whether the federated index name is valid, since you won't see data if the federated index is invalid. It also helps you understand whether the dataset that the federated index is mapped to holds data that is worth searching or correlating with information on your local Splunk platform deployment.

Example: Combine local and federated search results for comparative analysis

Now, say you want to investigate the performance of specific airline carriers at competing airports. The following search uses the union command to combine a remote saved search dataset of San Francisco flight data and a local events index dataset of New York City flight data. Then it finds the average arrival delay between airports in the combined dataset, broken out by airline.

| union [|from federated:search_of_airline_flights_SF] [search index = airlinedata_NYC] | stats avg (ArrDelay) by UniqueCarrier

If you want to search a local events index and a remote events index, you can use an OR operator to combine the results:

index = airlinedata_NYC OR index = federated:airline_flights_SF | stats avg (ArrDelay) by UniqueCarrier

Example: Aggregate results with join for a count of unique values

The following search uses the join command to aggregate results from the remote and local deployments. It returns the total count of airline carriers running flights out of San Francisco and New York City.

index = federated:airline_flights_SF | join left = L right = R where L.UniqueCarrier = R.UniqueCarrier [search index = airlinedata_NYC] | stats count

Restrictions for standard mode federated search

Standard mode federated search does not support the following:

• Generating commands other than search and from. For example, federated searches cannot include the datamodel, inputlookup, or tstats commands. To see which commands are classified as generating commands, see Command types in the Search Reference.
• Lookups.
• Using subsearches purely as search filters, especially when they do not include references to federated indexes.
• The from command can reference only datasets of the saved search type.
• The verbose and smart search modes. You can run federated searches only in the fast search mode. This can cause complications if you need your searches to provide search-time field extraction. See Federated searches and search-time field extraction.
• Real-time search.
• Usage of wildcard symbols (*) to reference multiple federated indexes.
• Metrics indexes and related metrics-specific search commands, such as mpreview or mstats. If you must include metrics data in a federated search, consider mapping a federated index to a saved search dataset that contains metric data. See Create a federated index.

In addition, you cannot use the from command to reference saved searches that include references to federated indexes. For example, say you save | from federated:saved_search_1 | stats count by field_1m as a saved search named fed_search_1. If you then try to run the search | from savedsearch:fed_search_1, that search will fail, because the saved search you are referencing contains a reference to a federated index.

You can still run saved searches that reference federated indexes using other ad-hoc search methods, such as running them from the Reports list page, and you can arrange for such searches to be scheduled searches.

Search over a transparent mode federated provider

When you run searches over a transparent mode federated provider, you can search as if you were searching over your local Splunk deployment. No special syntax is required. The search permissions associated with your role govern what you can search on the federated provider.

Restrictions for transparent mode federated search

Transparent mode federated search does not support the following:

• The verbose and smart search modes. You can run federated searches only in the fast search mode. This can cause complications if you need your searches to provide search-time field extraction. See Federated searches and search-time field extraction.
• Real-time search.
• Using from to search saved search datasets on the federated provider. You can use from to search saved search datasets on your local Splunk platform deployment.
• References to federated indexes with federated:. Only standard mode federated searches refer to federated indexes.

Federated searches and search-time field extraction

Ordinarily, if you want to to see field discovery in a search, you need to run the search in the verbose or smart search modes. Unfortunately, at this time federated searches support only the fast search mode.

You can ensure that search-time field extraction takes place for federated searches by appending | fields * to the ends of your searches.

Troubleshooting federated searches

Federated searches can fail to return events for a variety of reasons. The following table covers the most common error messages and conditions, and gives you some ways to resolve them.

Error message or condition	Resolution
Invalid federated index specified.	Provide the name of a federated index that exists on your local deployment.
Invalid federated provider specified.	Your search references a federated index which specifies a nonexistent federated provider in its definition. Contact your administrator and have them correct the federated index definition.
Unauthorized.	This can happen for a variety of reasons. Try these possible solutions: The service account credentials that have been set up for a federated provider in this search may be incorrect. Verify that the Service Account Username and Service Account Password that have been designated for the federated provider are correct. See Define a federated provider. If Local knowledge objects is enabled for the federated provider and the federated provider service account does not have a role with a name that matches the name of a role on your local Splunk platform deployment, searches may not run. The service account for the federated provider should have permission to search the datasets on the federated provider that are involved in the search. For example, if the search references an index dataset on the federated provider, the service account for that federated provider should have the ability to search that index dataset. This ability is associated with the service account role.
Zero results returned.	This can happen for a variety of reasons. Here is a checklist of possible solutions for your administrator to run through. Verify that your role permissions give you access to the federated indexes referenced in the search. Verify that the federated index is mapped to a valid remote dataset. If the remote dataset you are trying to search is an index, verify that the index is not disabled. If the remote dataset you are trying to search is a saved search, verify that the saved search permissions enable you to search it. If possible, run a search job directly on the remote search head to verify that it is working correctly. Verify that the dataset that the federated index maps to actually has more than zero events.
Knowledge-object issues	If you get errors telling you that lookups, event types, tags, calculated fields, or similar knowledge objects aren't found, there are two likely reasons for this that your administrator can investigate: The knowledge objects aren't where the federated provider definition indicates they will be. Verify that the knowledge objects belong to the application identified by Application Short Name, and that the application is installed on the local deployment if Local Knowledge Objects is selected, or on the federated provider if Local Knowledge Objects is not selected. When Local Knowledge Objects is disabled, permissions of knowledge objects on the remote deployment should be set so that users running federated searches from the local deployment can access them. See Manage knowledge object permissions in the Knowledge Manager Manual. Was Local Knowledge Objects enabled for the federated provider within the last few minutes? The process by which knowledge objects are transferred from local deployments to the federated provider takes a few minutes. This latency can vary depending on your network bandwidth and the size of the knowledge object bundle. Wait and try your search again.