Splunk Cloud Platform

Search Manual

Acrobat logo Download manual as PDF


This documentation does not apply to the most recent version of Splunk Cloud Platform. For documentation on the most recent version, go to the latest release.
Acrobat logo Download topic as PDF

Create a federated index

After you set up one or more remote Splunk platform deployments as standard mode federated providers for your local Splunk platform deployment, you need to create federated indexes for use in federated searches. Each federated index you create maps to one remote dataset on a standard mode federated provider.

Transparent mode federated providers do not use federated indexes. If you are running all of your federated searches in transparent mode, you can skip this topic.

The Splunk platform creates federated indexes on the federated search head of your local deployment.

In this task, you:

  • Provide the name of the federated index.
  • Select a standard mode federated provider. The federated provider must contain the remote dataset that you are mapping the federated index to.
  • Select the remote dataset that you are mapping the federated index to.

You can map a federated index to only one remote dataset at a time. If a federated provider contains several remote datasets over which you want to run federated searches, define a separate federated index for each dataset.

See About federated search for an overview of the standard and transparent modes of federated search.

Specifying remote datasets

When you create a federated index, you map the index to a specific remote dataset on a standard mode federated provider. Remote datasets can be events indexes, metrics indexes, saved searches, scheduled search jobs, or data models.

Remote dataset type Definition
Index Each events index and metrics index on a federated provider is a searchable dataset.
Saved search The result set produced by an ad-hoc run of a saved search on a federated provider is a searchable dataset.
Last job The results for the last job run for a scheduled search on a federated provider is a searchable dataset.
Data model The set of events defined by a data model on a federated provider is a searchable dataset.

You can map a federated index to an accelerated data model and then search it with the tstats command. See Run federated searches.

Use cases for saved search and last job dataset types

When you determine whether to set up federated indexes that map to saved search datasets or last job datasets, answer the following questions:

  • Are you concerned about the amount of federated search processing that might take place on the remote search head?
  • Do you require that the dataset contain fresh results, or can the dataset contain results from a search that was run in the recent past?

When you run a federated search that invokes a federated index which maps to a saved search dataset, the remote search head runs the saved search to get a result set, and then your federated search runs over that dataset to get the final results of the search.

If people are running a large number of searches on the remote search head, you might prefer to use last job datasets. Federated searches which invoke load job datasets do not need to run a search on the remote search head to get the result set. Such searches use an existing result set from the last job run by a scheduled search.

The last job dataset approach can drastically reduce the amount of search processing overhead that federated searches might add to a remote search head. If your users run multiple federated searches around the same time, and these searches each invoke the same last job federated index, those searches can all run over the same result set without requiring additional search jobs to be run on the remote search head.

This table summarizes the tradeoff between the saved search and last job dataset types.

Dataset type Amount of search processing required on the remote search head Recency of data in dataset
Saved search Requires the remote search head to run an ad-hoc saved search job to get a result set. This result set is then sent to the federated search head for federated search processing. Current. When you launch your federated search, Splunk software runs a saved search job and then runs your federated search over the data returned by that saved search job.
Last job Does not require additional search jobs. The result set from a previously run scheduled search job is sent to the federated search head for federated search processing. Depends on the interval of the scheduled search. For example, if the scheduled search runs on the hour, the result set can be up to an hour out of date.

By default, all scheduled search jobs expire after a period of time that is two times the interval of the scheduled search, which means there is always a scheduled search job available for federated searches. See Extending job lifetimes in the Search Manual.

Use saved search or last job datasets to route around federated search limitations

You can use saved search datasets to get around certain limitations of federated searches over a standard mode federated provider. For example, standard mode federated searches cannot belong to the following search categories:

  • Searches that use metrics search commands other than mstats to search data in metrics indexes, such as mpreview, or mcatalog
  • Searches that use any generating commands other than search, from, loadjob, mstats, or tstats.

However, you can create federated indexes that map to saved search or last job datasets which use commands that federated search does not support. Then you can write federated searches that reference those federated indexes. See Run federated searches.

Remote dataset restrictions

The following kinds of indexes, searches, and data models cannot be used as remote datasets for federated searches. Do not map federated indexes to them.

  • Federated indexes
  • Saved and scheduled searches with federated index references in their search strings
  • Data models with constraint searches that refer to federated indexes

The saved search and data model limitations relate to the fact that federated search does not support federated index chaining.

Remote dataset permissions

Review the permission settings on saved searches, scheduled searches, and data models that you want to use as federated search datasets. These knowledge objects must either be shared globally, or they must have the same app context as the federated provider that the federated index is associated with. In either case they must be shared with read permissions enabled.

For example, if you are creating a federated index for a federated provider that is associated with the Search app, any saved search dataset for that index must be shared with the Search app as well, or shared globally.

See Manage knowledge object permissions in the Knowledge Manager Manual.

Ensure federated index replication to search head cluster members in your local Splunk Enterprise deployment

If your local deployment uses Splunk Cloud Platform, you can ignore this section.

If your local deployment uses Splunk Enterprise with a search head cluster as its search tier and you are going to run federated search with a standard mode federated provider, you must use the deployer to distribute an additional configuration to the server.conf files on your search head cluster members. This configuration enables your federated index definitions to replicate to each member of the search head cluster.

Do not create federated indexes until you have pushed this configuration to your search head cluster members. Federated index definitions that you create before you push this configuration are not replicated to search head cluster members after you push the configuration. Searches using federated indexes that have not been properly replicated can fail or return incorrect search results.

Use the deployer to push a configuration bundle that adds conf_replication_include.indexes = true to the [shclustering] stanza of server.conf on each member of your search head cluster. To use the deployer you must have an admin or similar role with the admin_all_objects capability.

For more information about using the deployer to push updates to search head cluster members, see Use the deployer to distribute apps and configuration updates in the Distributed Search manual.

Prerequisites for federated index creation

Steps

  1. On the local deployment, in Splunk Web, select Settings, then Federated Search.
  2. On the Federated Indexes tab, select Add Federated Index.
  3. Using the following table, specify the settings for your federated index.
    Setting Description Default value
    Federated Index Name Specify the name of the federated index you're creating. The name must reference the remote dataset it maps to.

    Federated index names have the following restrictions:
    • They can contain only lower-case letters, numbers, underscores, and hyphens.
    • They must begin with a letter or number.
    • They cannot be more than 2048 characters in length.
    • They cannot contain the string "kvstore".
    No default
    Federated Provider Select the standard mode federated provider that contains the dataset to which this federated index will map. No default
    Remote Dataset Specify the remote Dataset Type that this federated index maps to and provide the Dataset Name.

    For Dataset Name, provide the name of a dataset of the selected Dataset Type that currently exists on the selected federated provider.

    For last job dataset types, Dataset Name values will be names of scheduled searches.

    Dataset Type defaults to Index.

    Dataset Name has no default.
  4. Select Save to save the federated index configuration.

The index is created on the federated search head of your local deployment.

In Splunk Web, you can view the federated indexes that you create for your deployment by selecting Settings > Federated Search > Federated Indexes.

Do not designate federated indexes as default indexes for roles or data inputs.

Give your users access to federated indexes

After you create a federated index, give your federated search users access to the index. If you do not do this, your users cannot search the remote dataset that the federated index maps to.

Just as with normal Splunk indexes, you grant access to federated indexes at the role level. This lets you grant federated index access to certain groups of users while disallowing access to other user groups.

To learn how to add a federated index to the set of searchable indexes for a role, see the section on federated indexes in Service accounts and federated search security.

Ensure service account roles can access remote datasets

Service account roles for standard mode federated providers must also have read permissions for any remote datasets that you expect your federated search users to access through federated indexes. For example, if you are going to set up a federated index that maps to a data model on a federated provider, make sure that the service account role for that federated provider has read permissions for that data model.

For more information about setting permissions for knowledge objects like saved searches and data models, see Manage knowledge object permissions.

Reference federated indexes in federated searches

After you create your federated indexes, you can reference them in federated searches. When you reference a federated index in a search, you are searching over the remote dataset to which the federated index maps. See Run federated searches.

Last modified on 21 March, 2024
PREVIOUS
Define a federated provider
  NEXT
Give your users role-based access control of federated indexes

This documentation applies to the following versions of Splunk Cloud Platform: 9.0.2303


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters