Splunk Cloud Platform

Search Manual

Acrobat logo Download manual as PDF

This documentation does not apply to the most recent version of SplunkCloud. Click here for the latest version.
Acrobat logo Download topic as PDF

Run federated searches

A federated search allows you to search across specific datasets on multiple Splunk platform deployments. These deployments include your local Splunk platform deployment and remote Splunk platform deployments that you designate as federated providers. A federated search is processed partly on your local instance and partly on the federated providers. The remote and local results are then brought to the federated search head of your local Splunk platform deployment, where they are aggregated together to produce a final result set.

The federated search process includes search optimizations that help federated searches run as efficiently as possible. These optimizations filter results on the remote search heads, which reduces the amount of data that must be sent over the network and processed on the federated search head, improving overall search performance.

For an overview of federated search and federated search terminology, see About federated search.


Before you can run a federated search, you must meet these requirements:

  • Your Splunk platform instance must have the federated search feature enabled. If it is not enabled, your administrator must enable it.
  • Your administrator must designate one or more remote Splunk platform deployments as federated providers. See Define a federated provider.
  • Your administrator must create one or more federated indexes on your local federated search head. Each federated index maps to a specific remote dataset on a federated provider. See Create a federated index.
  • Your role must have permissions for the federated indexes on your local federated search head that you intend to search.

Writing a federated search

The basic syntax for a federated search differs depending on the type of remote dataset that you are referencing in the search. A federated search of an index requires different syntax than a federated search of a saved search dataset.

In both cases, you specify a federated index that you have defined on your federated search head. This federated index maps to a remote dataset on the federated provider such as an events index or saved search. See Create a federated index.

You can reference multiple federated indexes in a subsearch using Boolean operators such as AND and OR.

If your role has the admin_all_objects capability, you can view the available federated indexes and the remote datasets to which they map on the Federated Indexes listing page at Settings > Federated Search > Federated Indexes. If your role does not have this capability, you need to get names of the federated indexes that you can search from your administrator.

Searching a remote index dataset

To search an remote index dataset on a federated provider, use the search command in conjunction with a reference to a federated index that maps to the index dataset.

search index=federated:<federated_index_for_remote_index>

The prefix federated: before the index name indicates that you are referring to a federated index, and <federated_index_for_remote_index> is the name of a federated index that maps to a remote index on a federated provider.

Searching a remote saved search dataset

To search a remote saved search dataset on a federated provider, use the from command in conjunction with a reference to a federated index that maps to the saved search dataset.

| from federated:<federated_index_for_remote_saved_search>

The prefix federated: before the index name indicates that you are referring to a federated index, and <federated_index_for_saved_search> is the name of a federated index that maps to a saved search on a federated provider.

Example: Simple search of a large remote index dataset

Say you have a federated index on your local search head named airline_flights_SF. This federated index is mapped to an index on a remote Splunk platform deployment that is defined as a federated provider to your Splunk platform deployment. The remote index contains 2,000,000 events relating to airline flight departures and arrivals at a San Francisco Airport. To return this large event set from the remote deployment, run this search:

index = federated:airline_flights_SF

As a best practice, run a simple search like this on a federated index that you're unfamiliar with. This practice helps you determine whether the federated index name is valid, since you won't see data if the index is invalid. It also helps you understand whether the dataset that the federated index is mapped to holds data that is worth searching or correlating with information on your local Splunk platform deployment.

Example: Combine local and federated search results for comparative analysis

Now, say you want to investigate the performance of specific airline carriers at competing airports. The following search uses the union command to combine a remote saved search dataset of San Francisco flight data and a local index dataset of New York City flight data. Then it finds the average arrival delay between airports in the combined dataset, broken out by airline.

| union [|from federated:search_of_airline_flights_SF] [search index = airlinedata_NYC] | stats avg (ArrDelay) by UniqueCarrier

If you want to search a local index and a remote index, you can use an OR operator to combine the results:

index = airlinedata_NYC OR index = federated:airline_flights_SF | stats avg (ArrDelay) by UniqueCarrier

Example: Aggregate results with join for a count of unique values

The following search uses the join command to aggregate results from the remote and local deployments. It returns the total count of airline carriers running flights out of San Francisco and New York City.

index = federated:airline_flights_SF | join left = L right = R where L.UniqueCarrier = R.UniqueCarrier [search index = airlinedata_NYC] | stats count

Federated search restrictions

In this version of the Splunk platform, federated search has several restrictions.

Federated search does not support the following:

  • Generating commands other than search and from. For example, datamodel, inputlookup, and tstats are not supported. To see which commands are classified as generating commands, see Command types in the Search Reference.
  • The verbose and smart search modes. Federated searches support only the fast search mode.
  • Real-time search.
  • Usage of wildcard symbols (*) to reference multiple federated indexes.
  • Metrics indexes and related metrics-specific search commands, such as mpreview or mstats. If you must include metrics data in a federated search, consider mapping a federated index to a saved search dataset that contains metric data. See Create a federated index.

In addition, you cannot use the from command to reference saved searches that include references to federated indexes. For example, say you save | from federated:saved_search_1 | stats count by field_1m as a saved search named fed_search_1. If you then try to run the search | from savedsearch:fed_search_1, that search will fail, because the saved search you are referencing contains a reference to a federated index.

You can still run saved searches that reference federated indexes using other ad-hoc search methods, such as running them from the Reports list page, and you can arrange for such searches to be scheduled searches.

Federated Search and Splunk Premium Solutions

You can run federated searches over Splunk Premium Solutions such as Enterprise Security or IT Service Intelligence, but the federated search restrictions described in this section apply to those searches.

For example, federated search currently does not support using tstats to search accelerated data model datasets, nor does it support search of KV Store collection datasets. This is because tstats is a generating command and because federated search currently supports only the index and saved search dataset types.

You can run federated searches across any events index dataset, including notable and risk indexes.

Troubleshooting federated searches

Federated searches can fail to return events for a variety of reasons. The following table covers the most common error messages and conditions, and gives you some ways to resolve them.

Error message or condition Resolution
Invalid federated index specified. Provide a valid federated index name.
Invalid federated provider specified. The definition of a federated index that you are referencing in your search specifies an invalid federated provider. Contact your administrator to fix the index definition.
Unauthorized. This can happen for a variety of reasons. Try these possible solutions:
  • The service account credentials that have been set up for a federated provider in this search may be incorrect. Verify that the Service Account Username and Service Account Password that have been designated for the federated provider are correct. See Define a federated provider.
  • The service account for the federated provider should have permission to search the datasets on the federated provider that are involved in the search. For example, if the search references an index dataset on the federated provider, the service account for that federated provider should have the ability to search that index dataset. This ability is associated with the service account role.
Zero results returned. This can happen for a variety of reasons. Here is a checklist of possible solutions for your administrator to run through.
  • Verify that your role permissions give you access to the federated indexes referenced in the search.
  • Verify that the federated index is mapped to a valid remote dataset.
  • If the remote dataset you are trying to search is an index, verify that the index is not disabled.
  • If the remote dataset you are trying to search is a saved search, verify that the saved search permissions enable you to search it.
  • If possible, run a search job directly on the remote search head to verify that it is working correctly.
  • Verify that the dataset that the federated index maps to actually has more than zero events.
Knowledge-object issues If you get errors telling you that lookups, event types, tags, calculated fields, or similar knowledge objects aren't found, there are two likely reasons for this that your administrator can investigate:
  • The knowledge objects aren't where the federated provider definition indicates they will be. Verify that the knowledge objects belong to the application identified by Application Short Name, and that the application is installed on the local deployment if Local Knowledge Objects is selected, or on the federated provider if Local Knowledge Objects is not selected.
  • Was Local Knowledge Objects selected for the federated provider within the last few minutes? The process by which knowledge objects are transferred from local deployments to the federated provider takes a few minutes. This latency can vary depending on your network bandwidth and the size of the knowledge object bundle. Wait and try your search again.
Last modified on 11 August, 2021
Create a federated index
Export search results

This documentation applies to the following versions of Splunk Cloud Platform: 8.2.2105 (latest FedRAMP release), 8.2.2106

Was this documentation topic helpful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters