Configure parallel reduce search processing

To enable parallel reduce search processing for your deployment, you need to configure your indexers to work as intermediate reducers and determine how your deployment should distribute the parallel reduction workload across your indexers.

If this is your first time reading about this feature, see Overview of parallel reduce search processing for an overview of parallel reduce search processing and a list of prerequisites.

Configure your indexers to work as intermediate reducers

To gain the benefits of parallel reduce search processing, you must configure all of your indexers so that they have the potential to work as intermediate reducers. You accomplish this configuration by giving each of your indexers an identical pass4SymmKey security key. This security key enables secure communication between indexers and intermediate reducers.

To update your indexer configurations, you must have access to the server.conf file for your Splunk deployment, located in $SPLUNK_HOME/etc/system/local/. See About configuration files and the topics that follow it in the Admin Manual for more information about making configuration file updates.

Parallel reduce search processing is not site-aware. Do not add this configuration to your indexers if they are in a multisite indexer cluster or if they are non-clustered and spread across several sites.

Set a security key for your intermediate reducers

Place a pass4SymmKey security key in a [parallelreduce] stanza for each indexer configuration in server.conf. The security key value must be identical for each indexer. It secures communication between the indexers and the intermediate reducers in your deployment.

Your indexer configurations might already have pass4SymmKey values under their [general] and [clustering] stanzas. Do not change those pass4SymmKey settings. Do not use the same security key values as those pass4SymmKey settings.

Save a copy of the key. After you set the key for an indexer and reboot the indexer, the security key changes from clear text to encrypted form, and it is no longer recoverable from server.conf. If you add a new intermediate reducer later, you must use the clear text version of the key to set it.

Prerequisites

The following prerequisite topics are useful if you run an indexer cluster.

• Secure your clusters with pass4SymmKey, in Securing Splunk Enterprise. Learn how pass4SymmKey is also used to authenticate communications between members of indexer clusters and search head clusters.
• Configure the indexer cluster with server.conf and Configure peer nodes with server.conf, in Managing Indexers and Clusters of Indexers. Learn how to update configurations for individual indexers.

Steps

1. Open server.conf and locate the settings for an indexer. Indexers are identified with a [<hostname>:<port>] stanza.
2. Add the following stanza and security configuration to the settings for the indexer:
   

   [parallelreduce]
   pass4SymmKey=<password>
   

3. Save your server.conf changes.
4. Restart the indexer with the CLI restart command:

   $SPLUNK_HOME/bin/splunk restart
   

Repeat these steps for each indexer in your deployment. Use the same <password> for each indexer in your deployment.

Determine how your parallel reduction workload is distributed

Settings in the [parallelreduce] stanza of limits.conf determine the number of intermediate reducers that are selected from your indexers for a parallel reduce search process. They also determine how parallel reduce search processing work is distributed across your indexers.

For example, if you keep the default parallel reduce settings in limits.conf, the Splunk platform randomly selects a certain number of intermediate reducers each time you run a parallel reduce search. If all of your indexers are in a single-site indexer cluster, the random selection aids in distributing the parallel reduction workload across the cluster.

However, if your indexers are not clustered, and some of your indexers have large indexing loads on average while others do not, you can use the reducers setting to configure the low-load indexers to be dedicated intermediate reducers. Dedicated intermediate reducers are always used when you run a parallel reduce search process.

These two methods are mutually exclusive. When you set up dedicated intermediate reducers, the Splunk platform cannot randomly select intermediate reducers.

To configure parallel reduce search processing, you must have access to the limits.conf file for your Splunk deployment, located in $SPLUNK_HOME/etc/system/local/. See About configuration files and the topics that follow it in the Admin Manual for more information about making configuration file updates.

Enable random selection of intermediate reducers

Random selection of indexers for intermediate reduction service is ideal if you are running a single-site indexer cluster. If you run several parallel reduce searches concurrently, the random selection ensures that the intermediate reduction work is evenly distributed across the cluster.

The default parallel reduce search processing settings enable the Splunk platform to randomly select intermediate reducers from the larger set of indexers when you run parallel reduce searches. The default number of indexers that the Splunk platform repurposes as intermediate reducers during the intermediate reduce phase of the parallel reduce search process is 50% of the total number of indexers in your indexer pool, up to a maximum of 4 indexers.

Random intermediate reducer selection is determined by the maxReducersPerPhase and winningRate settings. They belong to the [parallelreduce] stanza of limits.conf.

Setting name	Definition	Default value
maxReducersPerPhase	The maximum number of indexers that can be used as intermediate reducers in the intermediate reduce phase of a parallel reduce search.	4
winningRate	The percentage of indexers that can be selected from the total pool of indexers and used as intermediate reducers in a parallel reduce search process. This setting applies only when the reducers setting is not configured in limits.conf. See Enable dedicated intermediate reducers.	50

Enable dedicated intermediate reducers

To configure a set of non-clustered indexers as dedicated intermediate reducers, add the reducers setting to the [parallelreduce] stanza in limits.conf.

The value of reducers is a comma-separated list of indexers that you have configured as search peers. Identify each indexer by specifying its host and port using the following format: <host>:<port>. For example:

reducers=docteam-unix-4:8089, docteam-unix-5:8089, docteam-unix-6:8089

Do not include clustered indexers on the reducers list.

All indexers in the reducers list are used as intermediate reducers when you run a parallel reduce search. If the number of indexers in the reducers list exceeds the value of the maxReducersPerPhase setting, the Splunk platform randomly selects the intermediate reducers from the reducers list. For example, if the reducers setting lists five reducers and maxReducersPerPhase=4, the Splunk platform randomly selects four intermediate reducers from the list.

If all of the indexers in the reducers list are down or are otherwise invalid, searches with the redistribute command run without parallel reduction. All reduce operations are processed on the search head.

When you configure the reducers setting for your deployment, the Splunk platform ceases to apply the winningRate setting.

Override the number of reducers for a specific search

When you run a parallel reduce search with the redistribute command, you can use the num_of_reducers argument to override the number of reducers determined by the parallel reduce search settings in the limits.conf file.

For example, say your limits.conf settings determine that seven intermediate reducers are used by default in all parallel reduce searches. You can design a parallel reduce search where num_of_reducers = 5. Every time that search runs, only five intermediate reducers are used in its intermediate reduce phase.

If you provide a value for the num_of_reducers setting that exceeds the limit set by the maxReducersPerPhase setting in the limits.conf file, the Splunk platform sets the number of reducers to the maxReducersPerPhase value.

Next steps

Use the redistribute command to apply parallel reduce search processing to your high-cardinality searches. See Apply parallel reduce processing to searches.