Map a federated index to a remote Splunk dataset

In Federated Search for Splunk, after you set up one or more remote Splunk platform deployments as standard mode federated providers for your local Splunk platform deployment, you need to create federated indexes for use in federated searches. Each federated index you create maps to one remote dataset on a standard mode federated provider.

Federated indexes do not ingest or store data or events. Federated indexes cannot be targets for data inputs. The function of a federated index is to route your federated search to a specific dataset on a standard mode federated provider.

The Splunk platform creates federated indexes on the federated search head of your local deployment.

In this task, you:

• Provide the name of the federated index.
• Select a standard mode federated provider. The federated provider must contain the remote dataset that you are mapping the federated index to.
• Select the remote dataset that you are mapping the federated index to.

You can map a federated index to only one remote dataset at a time. If a federated provider contains several remote datasets over which you want to run federated searches, define a separate federated index for each dataset.

Transparent mode federated providers do not use federated indexes. If you are running all of your federated searches in transparent mode, you can skip this topic.

See About Federated Search for Splunk for an overview of the standard and transparent modes.

Specifying remote datasets

When you create a federated index, you map the index to a specific remote dataset on a standard mode federated provider. Remote datasets can be events indexes, metrics indexes, saved searches, scheduled search jobs, or data models.

Remote dataset type	Definition
Index	Index datasets are events indexes. Each events index on a federated provider is a searchable dataset.
Metric index	Each metrics index on a federated provider is a searchable dataset.
Saved search	The result set produced by an ad-hoc run of a saved search on a federated provider is a searchable dataset.
Last job	The results for the last job run for a scheduled search on a federated provider is a searchable dataset.
Data model	The set of events defined by a data model on a federated provider is a searchable dataset.

You can map a federated index to an accelerated data model and then search it with the tstats command. See Run federated searches over remote Splunk platform deployments.

Use cases for saved search and last job dataset types

When you determine whether to set up federated indexes that map to saved search datasets or last job datasets, answer the following questions:

• Are you concerned about the amount of federated search processing that might take place on the remote search head?
• Do you require that the dataset contain fresh results, or can the dataset contain results from a search that was run in the recent past?

When you run a federated search that invokes a federated index which maps to a saved search dataset, the remote search head runs the saved search to get a result set, and then your federated search runs over that dataset to get the final results of the search.

If people are running a large number of searches on the remote search head, you might prefer to use last job datasets. Federated searches for Splunk that invoke load job datasets do not need to run a search on the remote search head to get the result set. Such searches use an existing result set from the last job run by a scheduled search.

The last job dataset approach can drastically reduce the amount of search processing overhead that federated searches might add to a remote search head. If your users run multiple federated searches around the same time, and these searches each invoke the same last job federated index, those searches can all run over the same result set without requiring additional search jobs to be run on the remote search head.

This table summarizes the tradeoff between the saved search and last job dataset types.

Dataset type	Amount of search processing required on the remote search head	Recency of data in dataset
Saved search	Requires the remote search head to run an ad-hoc saved search job to get a result set. This result set is then sent to the federated search head for federated search processing.	Current. When you launch your federated search, Splunk software runs a saved search job and then runs your federated search over the data returned by that saved search job.
Last job	Does not require additional search jobs. The result set from a previously run scheduled search job is sent to the federated search head for federated search processing.	Depends on the interval of the scheduled search. For example, if the scheduled search runs on the hour, the result set can be up to an hour out of date.

By default, all scheduled search jobs expire after a period of time that is two times the interval of the scheduled search, which means there is always a scheduled search job available for federated searches. See Extending job lifetimes in the Search Manual.

Use saved search or last job datasets to route around federated search limitations

You can use saved search datasets to get around certain limitations of federated searches over a standard mode federated provider. For example, standard mode federated searches cannot belong to the following search categories:

• Searches that use metrics search commands other than mstats or mcatalog to search data in metrics indexes, such as mpreview, or mcollect
• Searches that use any generating commands other than search, from, loadjob, mstats, mcatalog, or tstats.

However, you can create federated indexes that map to saved search or last job datasets which use commands that federated search does not support. Then you can write federated searches that reference those federated indexes. See Run federated searches over remote Splunk platform deployments.

Remote dataset restrictions

The following kinds of indexes, searches, and data models cannot be used as remote datasets for federated searches. Do not map federated indexes to them.

• Federated indexes
• Saved and scheduled searches with federated index references in their search strings
• Data models with constraint searches that refer to federated indexes

The saved search and data model limitations relate to the fact that federated search does not support federated index chaining.

Remote dataset permissions

Review the permission settings on saved searches, scheduled searches, and data models that you want to use as federated search datasets. These knowledge objects must either be shared globally, or they must have the same app context as the federated provider that the federated index is associated with. In either case they must be shared with read permissions enabled.

For example, if you are creating a federated index for a federated provider that is associated with the Search app, any saved search dataset for that index must be shared with the Search app as well, or shared globally.

See Manage knowledge object permissions in the Knowledge Manager Manual.

Ensure federated index replication to search head cluster members in your local Splunk Enterprise deployment

If your local deployment uses Splunk Cloud Platform, you can ignore this section.

If your local deployment uses Splunk Enterprise with a search head cluster as its search tier and you are going to run federated search with a standard mode federated provider, you must use the deployer to distribute an additional configuration to the server.conf files on your search head cluster members. This configuration enables your federated index definitions to replicate to each member of the search head cluster.

Do not create federated indexes until you have pushed this configuration to your search head cluster members. Federated index definitions that you create before you push this configuration are not replicated to search head cluster members after you push the configuration. Searches using federated indexes that have not been properly replicated can fail or return incorrect search results.

Use the deployer to push a configuration bundle that adds conf_replication_include.indexes = true to the [shclustering] stanza of server.conf on each member of your search head cluster. To use the deployer you must have an admin or similar role with the admin_all_objects capability.

For more information about using the deployer to push updates to search head cluster members, see Use the deployer to distribute apps and configuration updates in the Distributed Search manual.

Prerequisites for federated index creation

• Read About Federated Search for Splunk to familiarize yourself with federated search concepts and terminology.
• To create federated indexes, you must have a role with the admin_all_objects and indexes_edit capabilities.
  • If you use Splunk Cloud Platform, the sc_admin role has these capabilities by default. See Define roles on the Splunk platform with capabilities in the Securing Splunk Cloud Platform manual.
  • If you use Splunk Enterprise, the admin role has these capabilities by default. See Define roles on the Splunk platform with capabilities in the Securing Splunk Enterprise manual.
• You must define one or more federated providers. See Define a Splunk platform federated provider.
• If your local deployment uses Splunk Enterprise and has a search head cluster as its search tier, you must use the deployer to distribute an additional configuration to the server.conf files on your search head cluster members. This configuration enables your federated index definitions to replicate to each member of the search head cluster. See Ensure federated index replication to search head cluster members in your local Splunk Enterprise deployment.
• Know the names of the remote datasets to which you want your federated indexes to map.

Steps

1. On the local deployment, in Splunk Web, select Settings, then Federation.
2. On the Federated Indexes tab, select Add federated index.
3. Using the following table, specify the settings for your federated index.
   

   Setting	Description	Default value
   Federated Index Name	Specify the name of the federated index you're creating. The name must reference the remote dataset it maps to. Federated index names have the following restrictions: They can contain only lower-case letters, numbers, underscores, and hyphens. They must begin with a letter or number. They cannot be more than 2048 characters in length. They cannot contain the string "kvstore".	No default
   Federated Provider	Select the standard mode federated provider that contains the dataset to which this federated index will map.	No default
   Remote Dataset	Specify the remote Dataset Type that this federated index maps to and provide the Dataset Name. For Dataset Name, provide the name of a dataset of the selected Dataset Type that currently exists on the selected federated provider. For last job dataset types, Dataset Name values will be names of scheduled searches.	Dataset Type defaults to Index. Dataset Name has no default.

4. Select Save to save the federated index configuration.

The index is created on the federated search head of your local deployment.

In Splunk Web, you can view the federated indexes that you create for your deployment by selecting Settings > Federated Search > Federated Indexes.

Do not designate federated indexes as default indexes for roles or data inputs.

Give your users access to federated indexes

After you create a federated index, give your federated search users access to the index. If you do not do this, your users cannot search the remote dataset that the federated index maps to.

Just as with normal Splunk indexes, you grant access to federated indexes at the role level. This lets you grant federated index access to certain groups of users while disallowing access to other user groups.

To learn how to add a federated index to the set of searchable indexes for a role, see the section on federated indexes in Service accounts and security for Federated Search for Splunk.

Ensure service account roles can access remote datasets

Service account roles for standard mode federated providers must also have read permissions for any remote datasets that you expect your federated search users to access through federated indexes. For example, if you are going to set up a federated index that maps to a data model on a federated provider, make sure that the service account role for that federated provider has read permissions for that data model.

For more information about setting permissions for knowledge objects like saved searches and data models, see Manage knowledge object permissions in the Knowledge Manager Manual.

Reference federated indexes in federated searches

After you create your federated indexes, you can reference them in federated searches. When you reference a federated index in a search, you are searching over the remote dataset to which the federated index maps. See Run federated searches over remote Splunk platform deployments.