Search Head Clustering

Implementing the Splunk App for Enterprise Security on clustered search heads changes the interaction with specific features of the Enterprise Security app. This topic details clustered search head requirements specific to the ES app, and does not replace the full documentation review and testing required to implement the search head clustering feature.

For an overview of search head clustering, see "Search head clustering architecture" in the Splunk Enterprise Distributed Search Manual.

System Requirements

The Splunk App for Enterprise Security requires the key value store feature for implementation on a search head cluster. A search head cluster cannot be deployed on Microsoft Windows operating systems. Additionally, the key value store feature is limited to 64-bit OS support. For the list of requirements, see "System requirements and other deployment considerations for search head clusters" in the Splunk Enterprise Distributed Search Manual.

Migrate your existing deployment

An Enterprise Security search head or search head pool member cannot be added directly to a search head cluster. A new search cluster must be created and deployed with the latest Enterprise Security app. The customized configurations from an existing ES installation must be reviewed and migrated to the deployer manually for replication to the cluster members. For more information, see the topic "Migrate from a standalone search head to a search head cluster" in the Splunk Enterprise Distributed Search Manual.

For assistance in planning a Splunk App for Enterprise Security deployment migration, contact the "Splunk Professional Services" team.

Using the Splunk App for Enterprise Security with other apps

Install only ES or CIM compatible apps or add-ons alongside the Enterprise Security app when deployed in a search head cluster.

Splunk Stream is not compatible with search head clustering. To initiate a Stream capture through the Enterprise Security app, the job must be created on the one search cluster node chosen to host Splunk Stream. See "About Search Head Clustering" in the Splunk Stream User Manual.

Forward search head data to indexers

The search head cluster members must send all locally generated data to the indexers. See the topic "Forward data from search head cluster members" in the Splunk Enterprise Distributed Search Manual.

Deploying configuration changes

Using the search head clustering feature changes the method used to deploy apps and configuration files to the search head cluster nodes. The deployment server is not supported as a means to distribute configurations or apps to cluster nodes. To distribute configurations across the set of search head cluster nodes, you must use the search head cluster deployer. See "Use the deployer to distribute apps and configuration updates" in the Splunk Enterprise Distributed Search Manual.

To facilitate using the deployer to manage configuration files with hashed passwords, synchronizing the splunk.secret file across cluster members is recommended. See "Deploy secure passwords across multiple servers" in the Securing Splunk Enterprise Manual.

FIPS Support

To enable FIPS support on a Search Head Cluster, the server.conf file on the cluster members must reference the full path to the local certificates for the KV store feature to function.

[kvstore]

caCertPath = /opt/splunk/etc/auth/cacert.pem

sslKeysPassword = password

sslKeysPath = /opt/splunk/etc/auth/server.pem

Dashboard changes

There are two categories of configuration changes made on a search head: UI and search-related configurations, and system configurations.

Any member of a search head cluster can create or update UI and search configurations. The changes replicate to the other search cluster nodes automatically without using the deployer.

System configurations, such as updating shared credentials, or creating a new lookup file must be managed centrally and require the use of the deployer node. To review which configuration files are replicated between cluster members and which ones must be deployed, see "How configuration changes propagate across the search head cluster" in the Splunk Enterprise Distributed Search Manual.

Incident Review

The Incident Review status updates for notable events are stored and replicated with the KV Store feature of search head clustering. For more information about the KV Store feature, see the topic "About the app key value store" in the Splunk Enterprise Admin Manual.

Notable Event Statuses

Adding, enabling, or disabling a review status to the Notable Event workflow cannot be done from a search head cluster member or captain. When reviewing the dashboard Configure > Incident Management > Notable Event Statuses from any cluster member, the current configuration is displayed but no configuration change option is available.

New workflow: Configure the Notable Event Status changes on your Enterprise Security testing or staging environment. After testing the configuration, use the search head cluster deployer to distribute the new or updated authorize.conf and review_statuses.conf configurations across the search head cluster nodes.

Credential Manager

Adding, enabling, or disabling a credential stored in Credential Management cannot be done from a search head cluster member or captain. When reviewing the dashboard Configure > General > Credential Management from any cluster member, the page states: "Credentials cannot be edited via the graphical user interface because search head clustering is enabled.”

New workflow: Configure the credential changes on your Enterprise Security testing or staging environment. After testing the configuration, use the search head cluster deployer to distribute the new or updated app.conf configurations across the search head cluster nodes.

Identity Management

Adding or disabling an identities list from Identity Management cannot be done from a search head cluster member or captain. When reviewing the dashboard Configure > Identity Management from any cluster member, the page states: "Current instance is running in SHC mode and is not able to add new inputs.”

New workflow: Configure the new or changed identities list on your Enterprise Security testing or staging environment. After testing the configuration, use the search head cluster deployer to distribute updated configurations and a new lookup file across the search head cluster nodes.

Threat List Management

Adding or disabling a threat list input from Threat lists cannot be done from a search head cluster member or captain. When reviewing the dashboard Configure > Data Enrichment > Threat Lists from any cluster member, the page states: "Current instance is running in SHC mode and is not able to add new inputs.”

New workflow: Configure the new or changed threat list on your Enterprise Security testing or staging environment. After testing the configuration, use the search head cluster deployer to distribute the updated inputs.conf configurations across the search head cluster nodes.

Upgrading ES on a search head cluster

Review all procedures and the order of operations before proceeding with the upgrade.

Prerequisites

1. Upgrade Splunk Enterprise as required. For more information, see "Upgrade a search head cluster" in the Splunk Enterprise Distributed Search Manual.
2. Download the latest version of the Enterprise Security app.

Prepare a staging instance

The staging instance is used to merge the search head cluster lookup files with the latest version of the Splunk App for Enterprise Security. If you have a clean testing or QA instance in your environment for the ES app, you may use that instance for staging the upgrade if no other apps are installed.

1. Prepare a single instance of Splunk Enterprise. This instance is for staging only, and should not be configured as a search head.
2. Copy the Enterprise Security installation from the deployer's $SPLUNK_HOME/etc/shcluster/apps into $SPLUNK_HOME/etc/apps on staging. The deployer's copy of Enterprise Security hosts the configuration files changed and deployed to the cluster. Upon migration to staging, the configurations are used for the comparison and notification of changes to default apps and add-ons during the upgrade process.

Copy the lookup files to staging

On a search head cluster member, copy the latest snapshot bundle to the staging instance.

1. Copy the snapshot bundle from $SPLUNK_HOME/var/run/splunk/snapshot from any cluster member to the staging instance.
2. Extract the ES app lookup files from the snapshot bundle to the staging instance. Example:

ssh into staging machine.

cd $SPLUNK_HOME/etc

tar xvf snapshot.bundle '*/lookups/*.csv

Upgrade staging to the latest version of ES

Upgrade the staging instance by following steps 1 - 5 of the topic "Upgrade Splunk App for Enterprise Security" in this manual.

Note: The upgrade process will not evaluate or inform on conflicts with customized knowledge objects and configurations that are managed by the search head cluster captain.

Caution: Deprecated apps and add-on changes will be handled manually by the customer after upgrading the staging instance. Retain copy of the upgrade report for a list of any warnings, deprecated apps, and changes to configuration files associated with the upgrade. Printing the report is recommended.

Migrate the upgraded ES install to the deployer

The contents of the staging instance along with the latest lookup files will be migrated to the deployer and deployed to the search head cluster members.

1. On the staging instance, copy $SPLUNK_HOME/etc/apps to $SPLUNK_HOME/etc/shcluster/apps on the deployer.
2. On the deployer, remove any deprecated apps or add-ons in $SPLUNK_HOME/etc/shcluster/apps that were removed during the upgrade on staging. Confirm by reviewing the ES upgrade report generated on staging, or by examining the apps moved into $SPLUNK_HOME/etc/disabled-apps on staging.

Deploy the changes to the cluster members

On the deployer, deploy the Enterprise Security app to search head cluster nodes.