Best practice: Forward search head data to the indexer layer
It is considered a best practice to forward all search head internal data to the search peer (indexer) layer. This has several advantages:
- It accumulates all data in one place. This simplifies the process of managing your data: You only need to manage your indexes and data at one level, the indexer level.
- It enables diagnostics for the search head if it goes down. The data leading up to the failure is accumulated on the indexers, where another search head can later access it.
- By forwarding the results of summary index searches to the indexer level, all search heads have access to them. Otherwise, they're only available to the search head that generates them.
Forward search head data
The preferred approach is to forward the data directly to the indexers, without indexing separately on the search head. You do this by configuring the search head as a forwarder. These are the main steps:
1. Make sure that all necessary indexes exist on the indexers. For example, the S.o.S app uses a scripted input that puts data into a custom index. If you install S.o.S on the search head, you need to also install the S.o.S Add-on on the indexers, to provide the indexers with the necessary index settings for the data the app generates. On the other hand, since
_internal exist on indexers as well as search heads, you do not need to create separate versions of those indexes to hold the corresponding search head data.
2. Configure the search head as a forwarder. Create an
outputs.conf file on the search head that configures the search head for load-balanced forwarding across the set of search peers (indexers). You must also turn off indexing on the search head, so that the search head does not both retain the data locally as well as forward it to the search peers.
Here is an example
# Turn off indexing on the search head [indexAndForward] index = false [tcpout] defaultGroup = my_search_peers forwardedindex.filter.disable = true indexAndForward = false [tcpout:my_search_peers] server=10.10.10.1:9997,10.10.10.2:9997,10.10.10.3:9997
This example assumes that each indexer's receiving port is set to 9997.
For details on configuring
outputs.conf, read "Configure forwarders with outputs.conf" in the Forwarding Data manual.
Forward data from search head cluster members
You perform the same configuration steps to forward data from search head cluster members to their set of search peers. However, you must ensure that all members use the same
outputs.conf file. To do so, do not edit the file on the individual search heads. Instead, use the deployer to propagate the file across the cluster. See "Use the deployer to distribute apps and configuration updates."
Add search peers to the search head
Modify the knowledge bundle
This documentation applies to the following versions of Splunk® Enterprise: 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.2.14, 6.2.15, 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.4.9, 6.4.10, 6.4.11, 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.5.6, 6.5.7, 6.5.8, 6.5.9, 6.5.10, 6.6.0, 6.6.1, 6.6.2, 6.6.3, 6.6.4, 6.6.5, 6.6.6, 6.6.7, 6.6.8, 6.6.9, 6.6.10, 6.6.11, 6.6.12, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.2.0, 7.2.1, 7.2.2, 7.2.3