Splunk® Enterprise

Distributed Search

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Best practice: Forward search head data to the indexer layer

It is considered a best practice to forward all search head internal data to the search peer (indexer) layer. This has several advantages:

  • It accumulates all data in one place. This simplifies the process of managing your data: You only need to manage your indexes and data at one level, the indexer level.
  • It enables diagnostics for the search head if it goes down. The data leading up to the failure is accumulated on the indexers, where another search head can later access it.
  • By forwarding the results of summary index searches to the indexer level, all search heads have access to them. Otherwise, they're only available to the search head that generates them.

The preferred approach is to forward the data directly to the indexers, without indexing separately on the search head. You do this by configuring the search head as a forwarder. These are the main steps:

1. Make sure that all necessary indexes exist on the indexers. For example, the S.o.S app uses a scripted input that puts data into a custom index. If you install S.o.S on the search head, you need to also install the S.o.S Add-on on the indexers, to provide the indexers with the necessary index settings for the data the app generates. On the other hand, since _audit and _internal exist on indexers as well as search heads, you do not need to create separate versions of those indexes to hold the corresponding search head data.

2. Configure the search head as a forwarder. Create an outputs.conf file on the search head that configures the search head for load-balanced forwarding across the set of search peers (indexers). You must also turn off indexing on the search head, so that the search head does not both retain the data locally as well as forward it to the search peers.

Here is an example outputs.conf file:

# Turn off indexing on the search head
[indexAndForward]
index = false
 
[tcpout]
defaultGroup = my_search_peers 
forwardedindex.filter.disable = true  
indexAndForward = false 
 
[tcpout:my_search_peers]
server=10.10.10.1:9997,10.10.10.2:9997,10.10.10.3:9997
autoLB = true

This assumes that each indexer's receiving port is set to 9997.

For details on configuring outputs.conf, read "Configure forwarders with outputs.conf" in the Forwarding Data manual.

PREVIOUS
Manage distributed server names
  NEXT
About mounted bundles

This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.0.15, 6.1, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.7, 6.1.8, 6.1.9, 6.1.10, 6.1.11, 6.1.12, 6.1.13, 6.1.14


Comments

Santharam - Thank you for catching that typo. It's fixed now.

Sgoodman
March 31, 2014

indexandforward in the tcp stanza should be like 'indexAndForward' not all lowercase.<br /><br />You will get the following error during splunk service re-start when you use indexandforward with all lowercase.<br /><br />Example message:<br />=================<br />Possible typo in stanza [tcpout] in /opt/splunk/etc/system/local/outputs.conf, line 21: indexandforward = false

Santharam
March 31, 2014

Daniel333 - As with most Splunk conf files, you can locate outputs.conf in any of several places. The link provided at the end of the current topic takes you to another topic that provides further details on file locations for outputs.conf specifically: <br /><br />http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Configureforwarderswithoutputs.confd<br /><br />In addition, you might find this background topic on Splunk configuration file locations to be helpful: <br /><br />http://docs.splunk.com/Documentation/Splunk/latest/Admin/Configurationfiledirectories

Sgoodman
November 26, 2013

What is the path for this file ? <br /><br />"2. Configure the search head as a forwarder. Create an outputs.conf file on the search head that configures the search head for load-balanced forwarding across the set of search peers (indexers). For example:<br />"

Daniel333
November 25, 2013

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters