How configuration changes propagate across the search head cluster

Read this first

Before reading this topic, see:

• "Administer Splunk Enterprise with configuration files" in the Admin Manual. The topics in that chapter provide important background information on configuration files.

The importance of configuration files in a search head cluster

Settings in configuration files control the functionality of a search head, including the set of knowledge objects. For example, there are configuration files for saved searches, event types, and workflow actions. Other configuration files provide the settings for non-search functionality, such as data inputs and indexing. See "List of configuration files" in the Admin Manual.

Besides the configuration files, other files are important to search-time functionality. For example, static lookup tables, dashboards, and data models use various files as part of their definition.

For a search head cluster to function properly, its members must all use the same set of search-related configurations. For example, all search heads in the cluster need access to the same set of saved searches. They must therefore use the same savedsearches.conf settings.

Members should also use the same set of user-related settings. See "Add users to the search head cluster."

Apps must also be identical across all search heads in a cluster. An app is essentially just a set of configurations.

How configuration changes propagate in a search head cluster

A search head cluster uses two means to ensure that configurations are identical across its members: automatic replication and the deployer.

Replicated changes

The cluster automatically replicates any runtime knowledge object changes on one cluster member to all other members. This includes, for example, changes or additions to saved searches, lookup tables, and dashboards. For example, when a user in Splunk Web defines a field extraction, the cluster replicates that field extraction to all other search heads in the cluster.

In addition, the cluster replicates a few other runtime changes as well, such as changes to users and roles.

See "Configuration updates that the cluster replicates."

Deployed changes

The cluster does not replicate all configuration changes, but rather only certain changes, primarily to knowledge objects, made at runtime through Splunk Web, the CLI, or the REST API. For other configuration changes and additions, you must explicitly push the changes to all cluster members. You do this through a special Splunk Enterprise instance called the deployer.

Examples of changes that require use of the deployer include any configuration files that you edit directly. For example, if you make a change in limits.conf, you must push the change through the deployer. Similarly, if you directly edit a knowledge object configuration file, like savedsearches.conf, you must use the deployer to distribute it to cluster members. In addition, you must use the deployer to push new or upgraded apps to the cluster members.

You also use the deployer to migrate app and user settings from an existing search head pool or standalone search head to the search head cluster.

See "Use the deployer to distribute apps and configuration updates."

Add non-clustered search peers to a search head cluster

Adding non-clustered search peers (that is, indexers that are not part of an indexer cluster) to the search head cluster is an example of the type of configuration change that the cluster does not replicate automatically. At the same time, however, it might not be convenient to add search peers by using the deployer to push an updated distsearch.conf, because the deployer will then initiate a rolling restart of all cluster members.

To avoid a restart of cluster members, you can use the CLI splunk add search-server command to add peers to each cluster member individually. For details, see "Connect the search heads in clusters to search peers."

Caution: Complete this operation across all cluster members quickly, so that all members maintain the same set of search peers.

The Settings menu

The Settings menu in Splunk Web organizes settings into several groups, including one called Knowledge, which contains the knowledge object settings. Search head clustering hides most non-Knowledge groups in each member's Settings menu by default. For example, it hides settings for data inputs and the distributed environment. You can unhide the hidden groups, if necessary.

The reason for hiding non-Knowledge settings is that the cluster only replicates certain setting changes, mainly those in the Knowledge category. If you make a change on one member to a setting in a non-Knowledge category, the cluster, with a few exceptions, does not automatically replicate that change to the other members. This can lead to the members being out of sync with each other.

If you need to access a hidden setting on a member, you can unhide those settings:

1. Click Settings in the upper right corner of Splunk Web. A list of settings, mainly limited to the Knowledge group, appears.

2. Click the Show All Settings button at the end of the list. A dialog box reminds you that hidden settings will not be replicated.

3. To continue, click Show in the dialog box. The full list of settings, dependent on your role permissions, appears.

The settings are now unhidden for all users with permission to view them; typically, all admin users. To rehide the settings, you must restart the instance.

Important: If you make a change to a hidden setting, the changed configuration will exist only on the cluster member where you made the change. If you want other members to get that change as well, you must use the deployer to push the underlying configuration file for that setting.

CLI commands and cluster members

Most general and search-related CLI commands are available for use on cluster members. If you run the command on one member, the cluster replicates the resulting configuration changes to the other members.

However, do not run the splunk clean command, in any of its variants, on an active cluster member. For example, the splunk clean all command should only be run after a member is removed from the cluster, as that command deletes the _raft folder, /etc/passwd, and so on. Similarly, if you run splunk clean userdata on one member, the user data will be cleaned on that member only. The change will not replicate to the other members, causing user/role information to differ between members.

For more information on replicated changes, see "Configuration updates that the cluster replicates."