Run federated searches

A federated search allows you to run queries across specific datasets on multiple Splunk platform deployments: your local Splunk platform deployment, and remote Splunk platform deployments that you have designated as federated providers. A federated search is processed partly on your local instance and partly on the federated providers. The remote and local results are then brought to the federated search head of your local Splunk platform deployment, where they are aggregated together to produce a final result set.

The federated search process includes search optimizations that help federated searches run as efficiently as possible. These optimizations filter results on the remote search heads, which reduces the amount of data that must be sent over the network and processed on the federated search head, improving overall search performance.

For an overview of the federated search feature and definitions of federated search terminology, see About federated search.

Federated search requirements

Before you can run a federated search there are requirements that you must meet.

• Your Splunk platform instance must have the federated search feature enabled. If it is not enabled, your administrator must enable it.
• Your administrator must have designated one or more remote Splunk platform deployments as federated providers. See Define a federated provider.
• Your administrator must have created one or more federated indexes on your local federated search head. Each federated index maps to a specific remote dataset on a federated provider. See Create a federated index.
• Your role must have permissions for the federated indexes on your local federated search head that you intend to search.
  • If you use Splunk Cloud Platform, see Documentation:SplunkCloud:Security:ManageSplunkCloudusersandroles in Securing the Splunk Platform.
  • If you use Splunk Enterprise, see Create and manage roles with Splunk Web in Securing the Splunk Platform.

Writing a federated search

When you write a federated search, you specify a federated index that has been created on your federated search head. The federated index maps to a remote dataset on the federated provider that you want to query. In this version of the Splunk platform, federated indexes can map only to index datasets on federated providers.

The syntax for the federated index reference follows this pattern: index = federated:<name_of_federated_index>. The prefix federated: before the index name indicates that you are referring to a federated index.

This means that if you want to run federated searches, you need to know the names of the federated indexes that your role has permission to search. If your role has the admin_all_objects and indexes_edit capabilities, you can see a list of available federated indexes and the datasets to which they map on the Federated Indexes listing page at Settings > Federated Search > Federated Indexes. If your role does not have this capability you will need to get names of federated indexes that you can search from your administrator.

Example: Simple search of a large remote dataset

Say you have a federated index on your local search head named airline_flights_SF. This federated index is mapped to an index on a remote Splunk platform deployment that has been defined as a federated provider to your Splunk platform deployment. The remote index contains two million events relating to airline flight departures and arrivals at a San Francisco Airport. To quickly return this large event set from the remote deployment, you would run this search:

search index = federated:airline_flights_SF

It is always a good idea to run a simple search like this on a federated index with which you are unfamiliar. This practice helps you determine whether the index name is valid: you will not see data if the index is invalid. It also helps you determine whether the index holds data that is worth searching or correlating with information on your local Splunk platform deployment.

Example: Combine local and federated search results for comparative analysis

Now, say you want to investigate the performance of specific airline carriers at competing airports. The following search uses the union command to combine a remote dataset of San Francisco flight data and a local dataset of New York City flight data. Then it finds the average arrival delay between airports in the combined dataset, broken out by airline.

| union [ search index = federated:airline_flights_SF ] [ search index = airlinedata_NYC ] | stats avg (ArrDelay) by UniqueCarrier

Example: Aggregate results with join for a count of unique values

The following search uses the join command to aggregate results from the remote and local deployments. It returns the total count of airline carriers running flights out of San Francisco and New York City.

index = federated:airline_flights_SF | join left = L right = R where L.UniqueCarrier = R.UniqueCarrier [ search index = airlinedata_NYC ] | stats count

Federated search restrictions

In this version of the Splunk platform, federated search has the following restrictions. Some of these restrictions may be removed in upcoming versions.

Federated search does not support the following:

• Generating commands other than search and from. For example, datamodel, inputlookup, and tstats are not supported. To see which commands are classified as generating commands, see Command types in the Search Reference.
• The verbose and smart search modes. Federated searches support only the fast search mode.
• Real-time search.
• Usage of wildcard symbols (*) to reference multiple federated indexes.
• Metric indexes and related metrics-specific search commands, such as mpreview or mstats.

Federated Search and Splunk Premium Solutions

You can run federated searches over Splunk Premium Solutions such as Enterprise Security or IT Service Intelligence, but the federated search restrictions described in this section apply to those searches.

For example, federated search currently does not support using tstats to search accelerated data model datasets, nor does it support the search of KV Store collection datasets. This is because tstats is a generating command and because federated search currently supports only the index dataset type.

You can run federated searches across any event index dataset, including notable and risk indexes.

Troubleshooting federated searches

Federated searches can fail to return events for a variety of reasons. The following table covers the most common error messages and conditions and gives you some ways to resolve them.

Error message or condition	Resolution
Invalid federated index specified.	Provide a valid federated index name.
Invalid federated provider specified.	The definition of a federated index that you are referencing in your search specifies an invalid federated provider. Contact your administrator and have them fix the index definition.
Unauthorized.	This can happen for a variety of reasons. Here is a checklist of possible solutions for your administrator to run through. The service account credentials that have been set up for a federated provider in this search may be incorrect. Verify that the Service Account Username and Service Account Password that have been designated for the federated provider are correct. See Define a federated provider. The service account for the federated provider should have permission to search the datasets on the federated provider that are involved in the search. For example, if the search references an index dataset on the federated provider, the service account for that federated provider should have the ability to search that index dataset. This ability is associated with the service account role.
Zero results returned.	This can happen for a variety of reasons. Here is a checklist of possible solutions for your administrator to run through. Verify that your role permissions give you access to the federated indexes referenced in the search. Verify that the federated index is mapped to a valid remote dataset. If the remote dataset you are trying to search is an index, verify that the index is not disabled. If possible, run a search job directly on the remote search head to verify that it is working correctly. Verify that the dataset that the federated index maps to actually has > 0 events.
Knowledge-object issues	If you get errors telling you that lookups, event types, tags, calculated fields, or similar knowledge objects aren't found, there are two likely reasons for this that your administrator can investigate: The knowledge objects aren't where the federated provider definition indicates they will be. Verify that the knowledge objects belong to the application identified by Application Short Name, and that the application is installed on the local deployment if Local Knowledge Objects is selected, or on the federated provider if Local Knowledge Objects is not selected. Was Local Knowledge Objects just selected for the federated provider? The process by which knowledge objects are transferred from local deployments to the federated provider takes a few minutes. This latency can vary depending on your network bandwidth and the size of the knowledge object bundle. Wait and try your search again.