Search Head Clustering

Implementing the Splunk App for Enterprise Security on clustered search heads changes the interaction with specific features of the Enterprise Security app. This topic details clustered search head requirements specific to the ES app, and does not replace the full documentation review and testing required to implement the search head clustering feature.

For an overview of search head clustering, see "Search head clustering architecture" in the Splunk Enterprise Distributed Search Manual.

System Requirements

The Splunk App for Enterprise Security requires the key value store feature for implementation on a search head cluster. A search head cluster cannot be deployed on Microsoft Windows operating systems. Additionally, the key value store feature is limited to 64-bit OS support. For the list of requirements, see "System requirements and other deployment considerations for search head clusters" in the Splunk Enterprise Distributed Search Manual.

Migrate your existing deployment

An Enterprise Security search head or search head pool member cannot be added directly to a search head cluster. A new search cluster must be created and deployed with the latest Enterprise Security app. The customized configurations from an existing ES installation must be reviewed and migrated to the deployer manually for replication to the cluster members. For more information, see the topic "Migrate from a standalone search head to a search head cluster" in the Splunk Enterprise Distributed Search Manual.

For assistance in planning a Splunk App for Enterprise Security deployment migration, contact the "Splunk Professional Services" team.

Using the Splunk App for Enterprise Security with other apps

Install only ES or CIM compatible apps or add-ons alongside the Enterprise Security app when deployed in a search head cluster.

Splunk Stream is not compatible with search head clustering. See "Distributed deployment" in the Splunk Stream User Manual.

Forward search head data to indexers

The search head cluster members must send all locally generated data to the indexers. See the topic "Forward data from search head cluster members" in the Splunk Enterprise Distributed Search Manual.

Deploying configuration changes

Using the search head clustering feature changes the method used to deploy apps and configuration files to the search head cluster members. The deployment server is not supported as a means to distribute configurations or apps to a search head cluster. To distribute configurations across the set of search head cluster members, you must use the search head cluster deployer. See "Use the deployer to distribute apps and configuration updates" in the Splunk Enterprise Distributed Search Manual.

To facilitate using the deployer to manage configuration files with hashed passwords, synchronizing the splunk.secret file across cluster members is recommended. See "Deploy secure passwords across multiple servers" in the Securing Splunk Enterprise Manual.

FIPS Support

To enable FIPS support on a Search Head Cluster, the server.conf file on the cluster members must reference the full path to the local certificates for the KV store feature to function.

[kvstore]

caCertPath = /opt/splunk/etc/auth/cacert.pem

sslKeysPassword = password

sslKeysPath = /opt/splunk/etc/auth/server.pem

Dashboard changes

There are two categories of configuration changes made on a search head: UI and search-related configurations, and system configurations.

Any member of a search head cluster can create or update UI and search configurations. The changes replicate to the other search cluster members automatically without using the deployer.

System configurations, such as updating inputs for threat sources must be managed centrally and require the use of the deployer. To review which configuration files are replicated between cluster members and which ones must be deployed, see "How configuration changes propagate across the search head cluster" in the Splunk Enterprise Distributed Search Manual.

Identity Management

Adding or disabling an identities list from Identity Management cannot be done from a search head cluster member or captain. When reviewing the dashboard Configure > Identity Management from any cluster member, the page states: "Current instance is running in SHC mode and is not able to add new inputs.”

New workflow: Configure the new or changed identities list on your Enterprise Security testing or staging environment. After testing the configuration, use the search head cluster deployer to distribute updated configurations and a new lookup file across the search head cluster.

Threat Source Management

Adding or disabling a threat source from Threat Intelligence Manager or Threat Intelligence Downloads cannot be configured from a search head cluster member or captain. When reviewing the dashboard Configure > Data Enrichment > Threat Intelligence Downloads from any cluster member, the page states: "Current instance is running in SHC mode and is not able to add new inputs.”

New workflow: Configure the new or changed threat source on your Enterprise Security testing or staging environment. After testing the configuration, migrate the configuration to the search head cluster deployer and distribute the updated inputs.conf configurations across the search head cluster.

Upgrading ES on a search head cluster

This process discusses the upgrade of an existing search head cluster running the Splunk App for Enterprise Security. Review all procedures and the order of operations before proceeding with the upgrade.

Prerequisites

1. Upgrade Splunk Enterprise on all search head instances as required. For more information, see "Upgrade a search head cluster" in the Splunk Enterprise Distributed Search Manual.
2. Download the latest version of the Enterprise Security app.

Prepare a staging instance

The staging instance is used to compare the deployer's copy of the Enterprise Security app with the latest release. If you have a clean testing or QA instance in your environment for the ES app, you may use that instance for staging the upgrade if no other apps are installed.

1. Prepare a single instance of Splunk Enterprise. This instance is for staging only, and should not be configured as a search head.
2. Copy the Enterprise Security installation from the deployer's $SPLUNK_HOME/etc/shcluster/apps into $SPLUNK_HOME/etc/apps on staging. The deployer's copy of Enterprise Security hosts the configuration files that have been updated and deployed to the cluster.

Upgrade staging to the latest version of ES

Upgrade the staging instance by following steps 1 - 5 of the topic "Upgrade Splunk App for Enterprise Security" in this manual.

Note: The upgrade process will not evaluate or inform on conflicts with customized knowledge objects and configurations that are managed by the search head cluster captain.

Caution: Deprecated apps and add-on changes must be handled manually by the customer after upgrading the staging instance. Retain a copy of the ES upgrade report for a list of any warnings, deprecated apps, and changes to configuration files associated with the upgrade. Printing the report is recommended.

Migrate the upgraded ES install to the deployer

The upgraded contents of the staging instance will be migrated back to the deployer and deployed to the search head cluster members.

1. On the staging instance, copy $SPLUNK_HOME/etc/apps to $SPLUNK_HOME/etc/shcluster/apps on the deployer.
2. On the deployer, remove any deprecated apps or add-ons in $SPLUNK_HOME/etc/shcluster/apps that were removed during the upgrade on staging. Confirm by reviewing the ES upgrade report generated on staging, or by examining the apps moved into $SPLUNK_HOME/etc/disabled-apps on staging.

Deploy the changes to the cluster members

On the deployer, use the preserve-lookups true switch to deploy the ES app while retaining all lookup file content generated on the cluster members. For more information, see "Maintain lookup files across app upgrades" in the Splunk Enterprise Distributed Search Manual.