Run federated searches
A federated search lets you search across specific datasets on multiple Splunk platform deployments. These deployments include your local Splunk platform deployment and remote Splunk platform deployments that you designate as federated providers. A federated search is processed partly on your local instance and partly on the federated providers. The remote and local results are then brought to the federated search head of your local deployment, where they are aggregated together to produce a final result set.
The federated search process includes search optimizations that help federated searches run as efficiently as possible. These optimizations filter results on the remote search heads, which reduces the amount of data that must be sent over the network and processed on the federated search head, improving overall search performance.
The experience of writing and running federated searches differs depending on whether your federated providers use standard mode or transparent mode.
For an overview of federated search and federated search terminology, see About federated search.
General requirements for federated search
Before you can run a federated search, your administrator must designate one or more remote deployments as federated providers. See Define a federated provider. Find out whether the providers you are searching over use standard mode or transparent mode.
Federated provider names added to search results
When you run federated searches, the names of the federated providers you search are added to your search results as the values of a field named splunk_federated_provider. You can use the splunk_federated_provider field to group or filter results by the federated providers that produced them.
For example, if you run transparent mode federated searches over multiple federated providers, you can use the splunk_federated_provider field to discern how many results are coming from each provider. Without this field it can be hard to tell where your results are coming from. The following transparent mode federated search breaks out result counts by provider name and then again by product name.
sourcetype=access_* | stats count(eval(splunk_federated_provider="ProviderWest")) AS Provider_West, count(eval(splunk_federated_provider="ProviderEast")) AS Provider_East BY productName
This feature requires that your local Splunk platform deployment and all remote deployments set up as federated providers be upgraded to the following Splunk platform versions:
- Splunk Cloud Platform 9.0.2305 or higher.
- Splunk Enterprise 9.0.6 or higher Splunk Enterprise 9.0.x versions.
- Splunk Enterprise 9.1.1 or higher.
Search over a standard mode federated provider
Federated searches that run over a standard mode federated provider must use a special search syntax to invoke the federated search. If you are only working with standard mode federated providers and you run a search string that does not include this syntax, you are running a local search over your local deployment.
Standard mode federated searches are subject to specific requirements and restrictions.
Additional requirements for standard mode federated search
- Your administrator must create one or more federated indexes on your local federated search head. Each federated index maps to a specific remote dataset on a federated provider. See Create a federated index.
- Your role must have permissions for the federated indexes on your local federated search head that you intend to search. See the section on federated indexes in Service accounts and federated search security.
- Any custom knowledge objects in your search such as lookups, calculated fields, or event types must be present on the local search head and the remote search head. If this duplication of knowledge objects is not present or is incorrectly applied, searches may fail or return incorrect results. See Custom knowledge object coordination for standard mode federated providers.
Write standard mode federated searches
The basic syntax for a standard mode federated search differs depending on the type of remote dataset that you are referencing in the search. A federated search of an events index dataset requires different syntax than a federated search of a saved search or data model dataset.
However, all standard mode federated searches require a reference to at least one federated index that you have defined on your federated search head. This federated index maps to a remote dataset on the federated provider such as an events index, saved search, or data model. See Create a federated index.
You can use Boolean operators such as AND and OR to reference multiple federated indexes in a subsearch.
If your role has the admin_all_objects and indexes_edit capabilities, you can view the federated indexes to which you have access and the remote datasets that those federated indexes map to on the Federated Indexes listing page at Settings > Federated Search > Federated Indexes. If your role does not have this capability, get the names of the federated indexes that you can search from your administrator.
Syntax for standard mode federated searches of remote datasets
The following table provides the search commands and syntax required to search various dataset types on a standard mode federated provider. All of the examples use the federated: prefix to invoke a federated index on the federated search head that maps to a dataset on the remote search head.
| Remote dataset type | Required syntax | More information |
|---|---|---|
| events index |
|
|
| saved search |
|
A saved search dataset is composed of the results of an ad-hoc run of a saved search on the remote search head. For a comparison of saved search and last job datasets, see Create a federated index. |
| data model |
|
Searches that use tstats in conjunction with prestats=t are not supported.
|
Restrictions for standard mode federated search
Standard mode federated search does not support the following:
- Generating commands other than
search,from, ortstats. For example, federated searches cannot include thedatamodelorinputlookupcommands. You can find a list of generating commands in Command types, in the Search Reference. - Using
fromto reference datasets that are not saved searches or data models. - Searches of accelerated data models where
tstatshasprestats=t. - Real-time search.
- Usage of wildcard symbols (
*) to reference multiple federated indexes. - Metrics indexes and related metrics-specific search commands, such as
mpreviewormstats. If you must include metrics data in a federated search, consider mapping a federated index to a saved search dataset that contains metric data. See Create a federated index.
In addition, standard mode federated search supports the verbose and smart search modes only for generating searches: searches that generate events without transforming them. You can run other kinds of federated searches only in the fast search mode.
The fast search mode does not allow search-time field extraction. You can ensure that search-time field extraction takes place for fast mode federated searches by appending | fields * to the ends of those searches.
Standard mode federated search examples
Here are examples of federated searches of remote datasets on standard mode federated providers.
Simple search of a large remote events index dataset
Say you have a federated index on your local search head named airline_flights_SF. This federated index is mapped to an events index on a remote deployment that is defined as a federated provider for your local deployment. The remote events index contains 2,000,000 events relating to airline flight departures and arrivals at a San Francisco Airport. To return the first 100 events of this large remote dataset, run this search:
index = federated:airline_flights_SF | head 100
As a best practice, run a simple search like this on a federated index that you're unfamiliar with. This practice helps you determine whether the federated index name is valid, since you won't see data if the federated index is invalid. It also helps you understand whether the dataset that the federated index is mapped to holds data that is worth searching or correlating with information on your local deployment.
Combine local and standard mode federated search results for comparative analysis
Now, say you want to investigate the performance of specific airline carriers at competing airports. The following search uses the union command to combine a remote saved search dataset of San Francisco flight data and a local events index dataset of New York City flight data. Then it finds the average arrival delay between airports in the combined dataset, broken out by airline.
| union
[|from federated:search_of_airline_flights_SF]
[search index = airlinedata_NYC]
| stats avg (ArrDelay) by UniqueCarrier
If you want to search a local events index and a remote events index, you can use an OR operator to combine the results:
index = airlinedata_NYC OR index = federated:airline_flights_SF
| stats avg (ArrDelay) by UniqueCarrier
Aggregate results with join for a count of unique values
The following standard mode federated search uses the join command to aggregate results from the remote and local deployments. It returns the total count of airline carriers running flights out of San Francisco and New York City.
index = federated:airline_flights_SF
| join left = L right = R where L.UniqueCarrier = R.UniqueCarrier
[search index = airlinedata_NYC]
| stats count
Filter a remote data model dataset search by one or more index names
When you run a tstats search of a remote data model dataset on a standard mode federated provider, it is often helpful to filter the results of the search by one or more indexes. Indexes referenced by the WHERE keyword must be present on the remote search head.
| tstats count from datamodel = federated:remote_dm_1
WHERE index=index_fs_1 OR index=index_fs_2
You can also run tstats searches that mix local and federated indexes.
|tstats count where index=local_index_1 OR index=federated:remote_dm_1
Filter a remote data model dataset search on child data model datasets
When you run a tstats search of a remote data model dataset on a standard mode federated provider, you can filter the results on one or more of the child datasets within the data model by referencing the nodename of the child dataset.
| tstats count from datamodel = federated:remote_dm_1
WHERE nodename = DM_1_DS.DM_1_DS_CHILD_1
See the tstats topic in the Search Reference for more information about using nodename to filter tstats searches on child data model datasets.
Search over a transparent mode federated provider
When you run searches over a transparent mode federated provider, you can search as if you were searching over your local deployment. No special syntax is required. The search permissions associated with your role govern what you can search on the federated provider.
Restrictions for transparent mode federated search
Transparent mode federated search does not support the following things:
- Real-time search.
- Searching accelerated data models.
- Using
fromto search saved search datasets on the federated provider. You can usefromto search saved search datasets on your local deployment. - Usage of the
federated:syntax to refer to specific federated indexes. Transparent mode searches do not use federated indexes.
Troubleshooting federated searches
Federated searches can fail to return events for a variety of reasons. The following table covers the most common error messages and conditions, and gives you some ways to resolve them.
| Error message or condition | Resolution | Link to more information |
|---|---|---|
| Invalid federated index specified | Provide the name of a federated index that exists on your local deployment. | Create a federated index |
| Invalid federated provider specified. | Your search references a federated index which specifies a nonexistent federated provider in its definition. Contact your administrator and have them correct the federated index definition. | Create a federated index |
| Unauthorized. | The service account credentials that have been set up for a federated provider in this search may be incorrect. Verify that the Service Account Username and Service Account Password that have been designated for the federated provider are correct. | Service accounts and federated search security |
| If the federated provider uses transparent mode and the service account user for the provider has a role that does not have the fsh_manage capability, searches might not run. | Service accounts and federated search security | |
| If the federated provider uses standard mode, the service account user for the federated provider should have permission to search the index, saved search, and data model datasets on the federated provider that are involved in the search. For example, if the search references an index dataset on the federated provider, the service account for that federated provider should have the ability to search that index dataset. This ability is associated with the service account role. |
Manage knowledge object permissions in the Knowledge Manager Manual | |
| Zero results returned. | If the federated provider uses standard mode, verify that your role's permissions give you access to the federated indexes referenced in the search. | Service accounts and federated search security |
| If the federated provider uses standard mode, verify that the federated index is mapped to a valid remote dataset. | Create a federated index | |
| If the remote dataset you are trying to search is an events index, verify that the events index is not disabled. | If you need to verify whether an index is disabled:
| |
| If you are using a standard mode federated provider and the remote dataset you are trying to search is a knowledge object such as a saved search or data model, verify that the knowledge object's permissions enable you to search it. | Manage knowledge object permissions in the Knowledge Manager Manual | |
| If possible, run a search job directly on the remote search head to verify that the search head is working correctly. | ||
| If you are using a standard mode federated provider, verify that the remote dataset that the federated index maps to contains more than zero events. | ||
| Knowledge object issues for standard mode federated providers | The knowledge objects aren't where the federated provider definition indicates they will be. Verify that the knowledge objects belong to the application identified by Application Short Name, and that the application is installed on the federated provider. | Set the app context for standard mode federated providers |
| Your search includes custom knowledge objects that are not duplicated on the remote search head of the federated provider. | Custom knowledge object coordination for standard mode federated providers | |
| Verify that the permissions of knowledge objects on the remote search head are set so that users running federated searches from the local search head can access them. | Manage knowledge object permissions in the Knowledge Manager Manual. | |
| Knowledge object issues for transparent mode federated providers | Did you run a search within minutes of setting up a transparent mode federated provider? The process by which knowledge objects are transferred from your local deployment to a transparent mode federated provider takes a few minutes. You might be encountering issues because the knowledge objects upon which your search depends aren't yet present on the remote search head. This latency can vary depending on your network bandwidth and the size of the knowledge object bundle. Wait and try your search again. |
| Create a federated index | Export search results |
This documentation applies to the following versions of Splunk® Enterprise: 9.0.6, 9.0.7, 9.0.8, 9.0.9
Download manual
Feedback submitted, thanks!