Run federated searches over remote Splunk platform deployments

Federated Search for Splunk lets you search across specific datasets on multiple Splunk platform deployments. These deployments include your local Splunk platform deployment and remote Splunk platform deployments that you designate as federated providers. A federated search is processed partly on your local instance and partly on the federated providers. The remote and local results are then brought to the federated search head of your local deployment, where they are aggregated together to produce a final result set.

The federated search process includes search optimizations that help federated searches run as efficiently as possible. These optimizations filter results on the remote search heads, which reduces the amount of data that must be sent over the network, and then process the results on your federated search head, improving overall search performance.

The experience of writing and running federated searches differs depending on whether your federated providers use standard mode or transparent mode.

For an overview of federated search and federated search terminology, see About Federated Search for Splunk.

General requirements for federated search

Before you can run a federated search, your administrator must designate one or more remote deployments as federated providers. See Define a Splunk platform federated provider. Find out whether the providers you are searching over use standard mode or transparent mode.

Federated provider names added to search results

When you run federated searches, the names of the federated providers you search are added to your search results as the values of a field named splunk_federated_provider. You can use the splunk_federated_provider field to group or filter results by the federated providers that produced them.

For example, if you run transparent mode federated searches over multiple federated providers, you can use the splunk_federated_provider field to discern how many results are coming from each provider. Without this field it can be hard to tell where your results are coming from. The following transparent mode federated search breaks out result counts by provider name and then again by product name.

sourcetype=access_* | stats count(eval(splunk_federated_provider="ProviderWest")) AS Provider_West, count(eval(splunk_federated_provider="ProviderEast")) AS Provider_East BY productName

This feature applies to all federated searches, whether they are in standard mode or transparent mode.

This feature requires that your local Splunk platform deployment and all remote deployments set up as federated providers be upgraded to the following Splunk platform versions:

• Splunk Cloud Platform 9.0.2305 or higher.
• Splunk Enterprise 9.0.6 or higher Splunk Enterprise 9.0.x versions.
• Splunk Enterprise 9.1.1 or higher.

Search over a standard mode federated provider

Federated searches that run over a standard mode federated provider must use a special search syntax to invoke the federated search. When you work with standard mode federated providers and you run a search that does not include this syntax, the search runs only over your local deployment.

Standard mode federated searches are subject to specific requirements and restrictions.

Additional requirements for standard mode federated search

• Your administrator must create one or more federated indexes on your local federated search head. Each federated index maps to a specific remote dataset on a federated provider. See Define a Splunk platform federated provider.
• Your role must have permissions for the federated indexes on your local federated search head that you intend to search. See the section on federated indexes in Service accounts and security for Federated Search for Splunk.
• Any custom knowledge objects in your search such as lookups, calculated fields, or event types must be present on the local search head and the remote search head. If this duplication of knowledge objects is not present or is incorrectly applied, searches may fail or return incorrect results. See Custom knowledge object coordination for standard mode federated providers.

Write standard mode federated searches

The basic syntax for a standard mode federated search differs depending on the type of remote dataset that you are referencing in the search. A federated search of an events index dataset requires different syntax than a federated search of a metrics index, saved search, last job, or data model dataset.

However, all standard mode federated searches require a reference to at least one federated index that you have defined on your federated search head. This federated index maps to a remote dataset on the federated provider such as an events index, a metrics index, a saved search, a data model, or the last job run by a scheduled search. See Map a federated index to a remote Splunk dataset.

You can use Boolean operators such as AND and OR to reference multiple federated indexes in a subsearch.

If your role has the admin_all_objects and indexes_edit capabilities, you can view the federated indexes to which you have access and the remote datasets that those federated indexes map to on the Federated Indexes listing page at Settings > Federated Search > Federated Indexes. If your role does not have this capability, get the names of the federated indexes that you can search from your administrator.

Syntax for standard mode federated searches of remote datasets

The following table provides the search commands and syntax required to search various dataset types on a standard mode federated provider. All of the examples use the federated: prefix to invoke a federated index on the federated search head that maps to a dataset on the remote search head.

When you run a standard mode federated search, you must follow the syntax for the dataset type to which your federated index is mapped. For example, you cannot use the from command to reference an events index, and you cannot search a metrics index without using mstats or mcatalog and referencing the federated index to which the federated index is mapped with a WHERE clause. When your search syntax does not match the dataset type, Splunk software returns an error message.

Remote dataset type	Required syntax	More information
events index	Use search to search events index datasets. | search index=federated:<federated_index_for_events_index> or index=federated:<federated_index_for_events_index>	You can use a wildcard symbol (*) to reference all federated indexes that map to events indexes: | search index=federated:*
metrics index	Use mstats or mcatalog in conjunction with a WHERE clause to search metrics index datasets. | mstats <stats-func> WHERE index=federated:<federated_index_for_metrics_index>	You cannot use a wildcard symbol (*) to reference all federated indexes that map to metrics indexes.
saved search	Use from to reference federated indexes that map to saved search datasets. | from federated:<federated_index_for_saved_search>	A saved search dataset is composed of the results of an ad-hoc run of a saved search on the remote search head. For a comparison of saved search and last job datasets, see Map a federated index to a remote Splunk dataset.
last job	Use from to reference federated indexes that map to saved search datasets. | from federated:<federated_index_for_last_job_dataset>	A last job dataset is composed of the results of the last job run by a scheduled search on the remote search head. For a comparison of saved search and last job datasets, see Map a federated index to a remote Splunk dataset.
data model	Use from to search an unaccelerated remote data model dataset. | from federated:<federated_index_for_data_model> Use tstats in conjunction with a FROM clause to search an accelerated remote data model dataset. | tstats <stats-func> FROM datamodel=federated:<federated_index_for_data_model>

Run federated searches over lookups

If you are running a standard mode federated search that uses the lookup command to enrich your results you need to consider whether you want the lookup to be processed on the remote search heads of the federated providers invoked in your search or the federated search head of your local Splunk platform deployment. Federated searches that process the lookup remotely have better overall search performance, but there may be reasons that you would prefer to have the lookup be processed on your local Splunk platform deployment.

In a standard mode federated search, you can force a lookup command to be processed locally by applying local=true to it. If you do not set local=true, under specific conditions the lookup command will be processed remotely.

If the lookup definition and lookup tables expected by the lookup are not present on the search heads on which it is processed, the lookup search will not return results.

For details, see the lookup reference topic in the Search Reference.

Restrictions for standard mode federated search

Standard mode federated search does not support the following things:

• Generating commands other than search, eventcount, from, loadjob, mcatalog, mstats, or tstats. For example, standard mode federated searches cannot include the datamodel or inputlookup commands. You can find a list of generating commands in Command types, in the Search Reference.
• Using from to reference events index or metrics index datasets.
• Real-time search.
• Metrics-specific search commands other than mstats and mcatalog, such as mpreview. If you must use these commands in a federated search, consider mapping a federated index to a saved search or last job dataset that runs them. See Map a federated index to a remote Splunk dataset.

Standard mode federated search examples

Here are examples of federated searches of remote datasets on standard mode federated providers.

Simple search of a large remote events index dataset

Say you have a federated index on your local search head named airline_flights_SF. This federated index is mapped to an events index on a remote deployment that is defined as a federated provider for your local deployment. The remote events index contains 2,000,000 events relating to airline flight departures and arrivals at a San Francisco airport. To return the first 100 events of this large remote dataset, run this search:

index = federated:airline_flights_SF | head 100

As a best practice, run a simple search like this on a federated index that you're unfamiliar with. This practice helps you determine whether the federated index name is valid, since you won't see data if the federated index is invalid. It also helps you understand whether the dataset that the federated index is mapped to holds data that is worth searching or correlating with information on your local deployment.

Simple search of a remote metrics dataset

Following on from the previous example, you might also have a federated index on your local search head named airline_flights_SF_metrics. This federated index is mapped to a metrics index on a remote deployment that is defined as a federated provider for your local deployment. This remote metrics index contains a significantly large number of metric data points that also relate to San Francisco airline flight arrivals and departures.

The following search returns a count of a specific airline departure metric.

| mstats count(SF.airline.splunkpony.departure_delay_time) where index=federated:airline_flights_SF_metrics

For more information, see mstats in the Search Reference and Overview of metrics in Metrics.

Simple search of a last job dataset

The following search runs over the last job of a scheduled search. The scheduled search runs at the start of each hour and returns information about security badge errors at gates for the preceding hour.

| from federated:badge_errors_last_hour | timechart span=1m count by gate usenull=f

Combine local and standard mode federated search results for comparative analysis

Say you want to investigate the performance of specific airline carriers at competing airports. The following search uses the union command to combine a remote saved search dataset of San Francisco flight data and a local events index dataset of New York City flight data. Then it finds the average arrival delay between airports in the combined dataset, broken out by airline.

| union [search index=federated:search_of_airline_flights_SF] [search index = airlinedata_NYC] | stats avg (ArrDelay) by UniqueCarrier

If you want to search a local events index and a remote events index, you can use an OR operator to combine the results:

index = airlinedata_NYC OR index = federated:airline_flights_SF | stats avg (ArrDelay) by UniqueCarrier

Aggregate results with join for a count of unique values

The following standard mode federated search uses the join command to aggregate results from the remote and local deployments. It returns the total count of airline carriers running flights out of San Francisco and New York City.

index = federated:airline_flights_SF | join left = L right = R where L.UniqueCarrier = R.UniqueCarrier [search index = airlinedata_NYC] | stats count

Filter a remote data model dataset search by one or more index names

When you run a tstats search of a remote data model dataset on a standard mode federated provider, it is often helpful to filter the results of the search by one or more indexes. Indexes referenced by the WHERE keyword must be present on the remote search head.

| tstats count from datamodel = federated:remote_dm_1 WHERE index=index_fs_1 OR index=index_fs_2

You can also run tstats searches that mix local and federated indexes.

|tstats count where index=local_index_1 OR index=federated:remote_dm_1

Filter a remote data model dataset search on child data model datasets

When you run a tstats search of a remote data model dataset on a standard mode federated provider, you can filter the results on one or more of the child datasets within the data model by referencing the nodename of the child dataset.

| tstats count from datamodel = federated:remote_dm_1 WHERE nodename = DM_1_DS.DM_1_DS_CHILD_1

See tstats in the Search Reference for more information about using nodename to filter tstats searches on child data model datasets.

Search over a transparent mode federated provider

When you run searches over a transparent mode federated provider, you can search as if you were searching over your local deployment. No special syntax is required. The search permissions associated with your role govern what you can search on the federated provider.

Searching accelerated data models in transparent mode

After you set up transparent mode federated search, accelerated data models on your local Splunk platform deployment create data model acceleration summaries on indexers of your local deployment and on indexers of your transparent mode federated provider. You do not need to set up accelerated data models on your transparent mode federated provider.

In your federated search, reference a local accelerated data model to return both local and remote results.

The ability to run transparent mode federated searches over accelerated data models requires that both your local and remote Splunk platform deployments be at either Splunk Cloud Platform 9.0.2303 or higher, or Splunk Enterprise 9.1.0 or higher.

See About data models and Accelerate data models in the Knowledge Manager Manual.

Restrictions for transparent mode federated search

Transparent mode federated search does not support the following things:

• Real-time search.
• Using the meventcollect command in searches.
• Using the datamodel command to search remote data models.
• Using from to search saved search datasets on the federated provider. You can use from to search saved search datasets on your local deployment.
• Using the federated: syntax to refer to specific federated indexes. Transparent mode searches do not use federated indexes.

Blocked commands for transparent mode federated search

Certain commands are blocked in transparent mode federated searches. When you use a blocked command in a transparent mode federated search, the search returns a warning message and returns results only from your local search head.

The following commands are entirely blocked in transparent mode federated searches:

• delete
• dump
• map
• rest
• run
• runshellscript
• script
• sendalert
• sendemail.

The commands in this list are classified as risky commands. See SPL safeguards for risky commands in Securing Splunk.

In addition, in certain situations, the makeresults and tstats commands are blocked or restricted.

Transparent mode federated searches that use the makeresults command are blocked unless you use the splunk_server or splunk_server_group argument to reference a search head, search head cluster, indexer, or indexer cluster on either your local Splunk platform deployment or the remote federated provider involved in the search. Splunk software processes the makeresults search on the local or remote server that you indicate with either of these arguments. For more information see makeresults in the Search Reference.

tstats searches that include a FROM clause are blocked for transparent mode federated searches over federated providers with Splunk Cloud Platform versions lower than 9.0.2303 or Splunk Enterprise versions lower than 9.1.0. If you use multiple transparent mode federated providers, the tstats search is processed only on federated providers with qualifying versions. For more information see tstats in the Search Reference.

Troubleshooting federated searches

Federated searches can fail to return events for a variety of reasons. The following table covers the most common error messages and conditions, and gives you some ways to resolve them.

Error message or condition	Resolution	Link to more information
Invalid federated index specified	Provide the name of a federated index that exists on your local deployment.	Create a federated index
Invalid federated provider specified.	Your search references a federated index which specifies a nonexistent federated provider in its definition. Contact your administrator and have them correct the federated index definition.	Create a federated index
Unauthorized.	The service account credentials that have been set up for a federated provider in this search may be incorrect. Verify that the Service Account Username and Service Account Password that have been designated for the federated provider are correct.	Service accounts and security for Federated Search for Splunk
	If the federated provider uses transparent mode and the service account user for the provider has a role that does not have the fsh_manage capability, searches might not run.	Service accounts and security for Federated Search for Splunk
	If the federated provider uses standard mode, the service account user for the federated provider should have permission to search the index, saved search, and data model datasets on the federated provider that are involved in the search. For example, if the search references an index dataset on the federated provider, the service account for that federated provider should have the ability to search that index dataset. This ability is associated with the service account role.	Manage knowledge object permissions in the Knowledge Manager Manual
Zero results returned.	If the federated provider uses standard mode, verify that your role's permissions give you access to the federated indexes referenced in the search.	Service accounts and security for Federated Search for Splunk
	If the federated provider uses standard mode, verify that the federated index is mapped to a valid remote dataset.	Create a federated index
	If the remote dataset you are trying to search is an events index, verify that the events index is not disabled.	If you need to verify whether an index is disabled: And you use Splunk Cloud Platform, see Manage Splunk Cloud Platform indexes in the Splunk Cloud Platform Admin Manual. And you use Splunk Enterprise, see Remove indexes and indexed data in Managing Indexers and Clusters of Indexers.
	If you are using a standard mode federated provider and the remote dataset you are trying to search is a knowledge object such as a saved search or data model, verify that the knowledge object's permissions enable you to search it.	Manage knowledge object permissions in the Knowledge Manager Manual
	If possible, run a search job directly on the remote search head to verify that the search head is working correctly.
	If you are using a standard mode federated provider, verify that the remote dataset that the federated index maps to contains more than zero events.
Search fails to complete, possibly with the error "Socket error during transaction. Socket error: Success"	Splunk software can terminate federated searches when their search result preview generation duration exceeds a timeout set by another component in your network, such as an elastic load balancer (ELB). When your search is terminated in this manner you can change the max_preview_generation_duration setting in federated.conf to a number of seconds lower than the timeout. For example, if you have an ELB that times out at 60 seconds, you might set the max_preview_generation_duration to 55.	You can change this value with a REST API call. See Federated search endpoint descriptions in the REST API Reference Manual.
Knowledge object issues for standard mode federated providers	The knowledge objects aren't where the federated provider definition indicates they will be. Verify that the knowledge objects belong to the application identified by Application Short Name, and that the application is installed on the federated provider.	Set the app context for standard mode federated providers
	Your search includes custom knowledge objects that are not duplicated on the remote search head of the federated provider.	Custom knowledge object coordination for standard mode federated providers
	Verify that the permissions of knowledge objects on the remote search head are set so that users running federated searches from the local search head can access them.	Manage knowledge object permissions in the Knowledge Manager Manual.
Knowledge object issues for transparent mode federated providers	Did you run a search within minutes of setting up a transparent mode federated provider? The process by which knowledge objects are transferred from your local deployment to a transparent mode federated provider takes a few minutes. You might be encountering issues because the knowledge objects upon which your search depends aren't yet present on the remote search head. This latency can vary depending on your network bandwidth and the size of the knowledge object bundle. Wait and try your search again.