Splunk® Enterprise

Distributed Search

Download manual as PDF

Download topic as PDF

Best practice: Forward search head data to the indexer layer

It is considered a best practice to forward all search head internal data to the search peer (indexer) layer. This has several advantages:

  • It accumulates all data in one place. This simplifies the process of managing your data: You only need to manage your indexes and data at one level, the indexer level.
  • It enables diagnostics for the search head if it goes down. The data leading up to the failure is accumulated on the indexers, where another search head can later access it.
  • By forwarding the results of summary index searches to the indexer level, all search heads have access to them. Otherwise, they're only available to the search head that generates them.

Forward search head data

The preferred approach is to forward the data directly to the indexers, without indexing separately on the search head. You do this by configuring the search head as a forwarder. These are the main steps:

1. Make sure that all necessary indexes exist on the indexers. For example, the S.o.S app uses a scripted input that puts data into a custom index. If you install S.o.S on the search head, you need to also install the S.o.S Add-on on the indexers, to provide the indexers with the necessary index settings for the data the app generates. On the other hand, since _audit and _internal exist on indexers as well as search heads, you do not need to create separate versions of those indexes to hold the corresponding search head data.

2. Configure the search head as a forwarder. Create an outputs.conf file on the search head that configures the search head for load-balanced forwarding across the set of search peers (indexers). You must also turn off indexing on the search head, so that the search head does not both retain the data locally as well as forward it to the search peers.

Here is an example outputs.conf file:

# Turn off indexing on the search head
index = false
defaultGroup = my_search_peers 
forwardedindex.filter.disable = true  
indexAndForward = false 

This example assumes that each indexer's receiving port is set to 9997.

For details on configuring outputs.conf, read "Configure forwarders with outputs.conf" in the Forwarding Data manual.

Forward data from search head cluster members

You perform the same configuration steps to forward data from search head cluster members to their set of search peers. However, you must ensure that all members use the same outputs.conf file. To do so, do not edit the file on the individual search heads. Instead, use the deployer to propagate the file across the cluster. See "Use the deployer to distribute apps and configuration updates."

Add search peers to the search head
Manage distributed server names

This documentation applies to the following versions of Splunk® Enterprise: 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.2.14, 6.2.15, 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.4.9, 6.4.10, 6.4.11, 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.5.6, 6.5.7, 6.5.8, 6.5.9, 6.5.10, 6.6.0, 6.6.1, 6.6.2, 6.6.3, 6.6.4, 6.6.5, 6.6.6, 6.6.7, 6.6.8, 6.6.9, 6.6.10, 6.6.11, 6.6.12, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.3.0, 7.3.1, 7.3.2, 8.0.0


Realsplunk - The search head shouldn't be putting any data into the internal indexes if you're using those settings. Are you finding otherwise?

Sgoodman, Splunker
December 6, 2017

Hello, is it 100% sure that data won't be logged on search heads? Thanks.

FYI we use

[indexAndForward] index = false

December 5, 2017

Jagadeeshm - Internal data doesn't count against your license.

Sgoodman, Splunker
July 15, 2016

Does forwarding the data from Search Head to the Indexer takes up license usage?


July 6, 2016

After following this, I believe it is successful because the Search Head is now seen as an Heavy Forwarder in the DMC console.

I am however wondering how to make sure that the Search Head is not indexing internal data anymore.

When I look at internal db default location (ll /opt/splunk/var/lib/splunk/_internaldb/), it still shows recent modification.

Any hint ? Thanks in advance

January 14, 2016

Butzowj -

The optimal directory for placing your edited conf files varies and depends on several factors. All versions of a conf file, across all locations, are combined at runtime, according to a defined order of precedence. The one thing that is certain, you do not want to edit any conf files located in the default directory, as those files get overwritten with each upgrade.

Before doing any editing of conf files, it is a good idea to study conf files in detail. See this topic in the Admin Manual, as well as the topics that follow it in the same chapter: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Aboutconfigurationfiles

Sgoodman, Splunker
January 11, 2016

For this step: Create an outputs.conf file on the search head that configures the search head for load-balanced forwarding across the set of search peers (indexers).

Could you supply the default directory where this outputs.conf file would be saved? I believe this would remove some ambiguity and would make the documentation more complete.

January 11, 2016

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters