Splunk® Enterprise

Distributed Search

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

How configuration changes propagate across the search head cluster

Read this first

Before reading this topic, see:

The importance of configuration files in a search head cluster

Settings in configuration files control the functionality of a search head, including the set of knowledge objects. For example, there are configuration files for saved searches, event types, and workflow actions. Other configuration files provide the settings for non-search functionality, such as data inputs and indexing. See "List of configuration files" in the Admin Manual.

Besides the configuration files, other files are important to search-time functionality. For example, static lookup tables, dashboards, and data models use various files as part of their definition.

For a search head cluster to function properly, its members must all use the same set of search-related configurations. For example, all search heads in the cluster need access to the same set of saved searches. They must therefore use the same savedsearches.conf settings.

Members should also use the same set of user-related settings. See "Add users to the search head cluster."

Apps must also be identical across all search heads in a cluster. An app is essentially just a set of configurations.

How configuration changes propagate in a search head cluster

A search head cluster uses two means to ensure that configurations are identical across its members: automatic replication and the deployer.

Replicated changes

The cluster automatically replicates any runtime knowledge object changes on one cluster member to all other members. This includes, for example, changes or additions to saved searches, lookup tables, and dashboards. For example, when a user in Splunk Web defines a field extraction, the cluster replicates that field extraction to all other search heads in the cluster.

See "Configuration updates that the cluster replicates."

Deployed changes

The cluster does not replicate all configuration changes, but only runtime changes made to knowledge objects through Splunk Web, the CLI, or the REST API. For other configuration changes and additions, you must explicitly push the changes to all cluster members. You do this through a special Splunk Enterprise instance called the deployer.

Examples of changes that require use of the deployer include any configuration files that you edit directly. For example, if you make a change in limits.conf, you must push the change through the deployer. Similarly, if you directly edit a knowledge object configuration file, like savedsearches.conf, you must use the deployer to distribute it to cluster members. In addition, you must use the deployer to push new or upgraded apps to the cluster members.

See "Use the deployer to distribute apps and configuration updates."

Add non-clustered search peers to a search head cluster

Adding non-clustered search peers (that is, indexers that are not part of an indexer cluster) to the search head cluster is an example of the type of configuration change that the cluster does not replicate automatically. At the same time, however, it might not be convenient to add search peers by using the deployer to push an updated distsearch.conf, because the deployer will then initiate a rolling restart of all cluster members.

To avoid a restart of cluster members, you can use the CLI splunk add search-server command to add peers to each cluster member individually. For details, see "Connect the search heads in clusters to search peers."

Caution: Complete this operation across all cluster members quickly, so that all members maintain the same set of search peers.

The Settings menu

The Settings menu in Splunk Web organizes settings into several groups, including one called Knowledge, which contains the knowledge object settings. Search head clustering hides all non-Knowledge groups in each member's Settings menu by default. For example, it hides settings for data inputs and the distributed environment. You can unhide the hidden categories, if necessary.

The reason for hiding non-Knowledge settings is that the cluster only replicates changes made to settings in the Knowledge category. If you make a change on one member to a setting in a non-Knowledge category, the cluster does not automatically replicate that change to the other members.

If you need to access a non-Knowledge setting on a member, you can unhide the hidden settings:

1. Click Settings in the upper right corner of Splunk Web. A list of settings, limited to the Knowledge group, appears.

2. Click the Show All Settings button at the end of the list. A dialog box reminds you that hidden settings will not be replicated.

3. To continue, click Show in the dialog box. The full list of settings, dependent on your role permissions, appears.

The settings are now unhidden for all users with permission to view them; typically, all admin users. To rehide the settings, you must restart the instance.

Important: If you make a change to a non-Knowledge setting, the changed configuration will exist only on the cluster member where you made the change. If you want other members to get that change as well, you must use the deployer to push the underlying configuration file for that setting.

PREVIOUS
Set a security key for the search head cluster
  NEXT
Configuration updates that the cluster replicates

This documentation applies to the following versions of Splunk® Enterprise: 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.2.14, 6.2.15


Comments

Jkat54 - Any change that you can make in Settings on Splunk Web without first needing to unhide the hidden settings will get automatically replicated to all the cluster members.

However, if you make a change directly in a conf file, such as alert_action.conf, it will not get replicated automatically. To propagate such settings, you must put the conf file on the deployer and then push it to the cluster members. For details on how to distinguish between replicated and deployed changes, see the subsection above entitled, "How configuration changes propagate in a search head cluster."

Sgoodman, Splunker
November 6, 2015

Can anyone explain why email server configuration settings are made in the advance settings menu of any search head already joined to the search head cluster and then replicated auto-magically to the other search heads? Are alert_action.conf or email settings considered knowledge objects?

Important: If you make a change to a non-Knowledge setting, the changed configuration will exist only on the cluster member where you made the change. If you want other members to get that change as well, you must use the deployer to push the underlying configuration file for that setting.

Jkat54
November 6, 2015

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters