Install Splunk Enterprise Security in a search head cluster environment
Splunk Enterprise Security has specific requirements and processes for implementing search head clustering.
- For an overview of search head clustering, see Search head clustering architecture in the Distributed Search Manual.
- For a complete list of search head clustering requirements, see System requirements and other deployment considerations for search head clusters in the Distributed Search Manual.
If you are installing Enterprise Security on an existing search head cluster environment which might have other apps deployed already, all of the steps in this section apply. Be careful to not delete or remove any existing content in the
Prerequisites for installing Enterprise Security in a search head cluster environment
Before installing Enterprise Server in a search head cluster environment, verify that you have:
- One deployer and one staging instance (where Enterprise Security is installed first)
- The same version of Splunk Enterprise on the deployer, staging instance, and search head cluster nodes
- The same app (not including Enterprise Security) versions on the deployer and search head cluster nodes
- The same version of Enterprise Security on the deployer, staging instance, and SHC nodes
- The backup of
etc/shcluster/appson the deployer before installing Enterprise Security
- The backup of
etc/appsfrom one of search head cluster nodes
- The backup of the KVstore from one of search head cluster nodes
Installing Enterprise Security in a search head cluster environment
To install Enterprise Security on a search head cluster:
- Prepare a staging instance. Use a staging instance to prepare Enterprise Security for the deployer. If you do not have a staging instance available, you can use a testing or QA Splunk Enterprise instance that does not have any other apps installed. A staging instance cannot be connected to production indexers or search peers. Use a staging instance for configuration changes and upgrades.
- Install Enterprise Security on the staging instance.
- Migrate the Enterprise Security installation to the deployer. Copy the apps, SAs, DAs, and TAs associated with the Splunk Enterprise Security Suite from
$SPLUNK_HOME/etc/appson the staging instance to
$SPLUNK_HOME/etc/shcluster/appson the deployer. Do not copy the entire folder because you do not want to include default apps, such as the search app.
- Use the deployer to deploy Enterprise Security to the cluster members. From the deployer, run this command:
splunk apply shcluster-bundle
Managing configuration changes in a search head cluster
Some system configuration changes must be deployed using the deployer.
- Instead of making the changes on a search head cluster member, make the changes on a staging instance.
- Test the configuration changes on the staging instance.
- Migrate the necessary files to the search head cluster deployer.
- Deploy the updated configuration to the search head cluster.
Configuration changes that must be deployed using the deployer:
|Configuration change||File modified|
|Enable or disable indexed real-time searches on the General Settings page.|
|Modify the indexed real-time disk sync delay on the General Settings page.|
|Send notable events to Splunk UBA on the UBA Setup page.|
Most configuration changes that you make in a search head cluster replicate automatically to other search head cluster members. For example:
- Add, modify, and disable threat intelligence sources
- Add, modify, and disable asset and identity source lists
- Changes to the user interface
- Changes to searches
See How configuration changes propagate across the search head cluster in the Distributed Search Manual.
Migrate an existing search head to a search head cluster
An Enterprise Security standalone search head or search head pool member cannot be added to a search head cluster. To migrate ES configurations to a search head cluster:
- Identify any custom configurations and modifications in the prior ES installation. Check to make sure there is no local copy of
ess_setup.confthat could conflict with the default one when you deploy Enterprise Security to the cluster.
- Implement a new search head cluster.
- Deploy the latest version of Enterprise Security on the search head cluster.
- Review and migrate the customized configurations to the search head cluster deployer for replication to the cluster members.
- Shut down the old ES search head.
For more information, see the topic Migrate from a standalone search head to a search head cluster in the Splunk Enterprise Distributed Search Manual.
For assistance in planning a Splunk Enterprise Security deployment migration, contact Splunk Professional Services.
Back up and restore Splunk Enterprise Security in a search head cluster environment
Back up and restore a Splunk Enterprise Security search head cluster (SHC) environment with at least three SHC nodes. All of the nodes in the SHC must be running the same version of Splunk Enterprise Security. Restoring an SHC environment might be necessary in the event of a disaster. You might also need to restore to a single search head if you add a member node to the SHC.
Take regular backups from the SHC, so that you have a backup from a time when the environment is healthy. For example, you could automate taking backups every hour. Choose a frequency of backups based on recovery point objectives.
To check if your environment is healthy, you can use one of the following methods:
- CLI command:
./splunk show shcluster-status –verbose
In the output, look for the following fields:
|dynamic_captain||Whether the cluster has a dynamically elected captain.|
|stable_captain||Whether the cluster captain is in a stable state.|
|service_ready_flag||Whether the cluster has enough members to support replication factor.|
|splunk_version||Whether all members, including the cluster master, are running Splunk version 7.1.0.|
|out_of_sync||Whether all nodes are currently in-sync.|
Back up a search head cluster environment
Back up an SHC environment by backing up the KV store, the deployer, and the SHC nodes. The backup procedure does not require shutting down the SHC cluster or any node in the cluster.
Back up the KV store
To back up the KV store, run the following command from the CLI from the SHC node with the most recent data:
splunk backup kvstore -auth "admin:<password>"
This command creates an archive file in the
$SPLUNK_HOME/var/lib/splunk/kvstorebackup directory. For example, the file might be named
Back up the deployer and search head cluster nodes
- On the deployer, back up the files in the
- On the SHC node with the most recent data, note the GUID from the
shclusteringstanza from the
$SPLUNK_HOME/etc/system/local/server.conffile. This information is necessary during the restore process.
- On the SHC node with the most recent data, back up the files in the
- Create a
tar.gzfile from these backups.
Restore from a backup of a search head cluster environment
You need the following information to restore from a backup:
- The GUID from the
server.conffile from one of the SHC members from before you begin restoring.
After your restore the cluster, the restored cluster will have the same GUID as the cluster that was backed up.
- A backup of the deployer.
- A backup of the SHC node with the most recent data.
- A backup of the KV store from the SHC node with the most recent data.
The restore procedure requires the SHC to be shut down, but not the KV store.
Restore the deployer
- On the deployer, extract the deployer backup file to the
- Apply the bundle with the following command:
splunk apply shcluster-bundle
Restore the search head cluster nodes
Complete the following steps on each SHC node that you want to restore:
- Run the following command to stop Splunk Enterprise Security:
- Create a temporary folder and name it
- Extract the SHC node backup file to the
- Move the
config.bundlefile from the
tempdirectory to the
- Extract the
config.bundlefile to the
Restore the KV store
- On an SHC node, check the
$SPLUNK_HOME/var/lib/splunk/kvstorebackupdirectory to make sure that the
kvdump_example.tar.gzbackup file that you want to use to restore is there. If it is not in that directory, manually copy your tar.gz file to that location. Note that the KV store backup file is not automatically replicated across each SHC member.
- Ensure that Splunk Enterprise Security is installed. The
collections.conffile is necessary to complete the restore.
- Run the following command to restore the KV store:
splunk restore kvstore -archiveName kvdump_example.tar.gz
- Restore the snapshot bundle by extracting the backup tar file from the
$SPLUNK_HOME/var/run/splunk/snapshotdirectory to the
- Repeat these steps on each SHC node that you want to restore. Restoring the KV store on one SHC node does not cause the KV store to automatically replicate across each SHC member.
Entries that are present in both the current KV store and in the backup are updated and replaced by the entry in the backup. Entries that are in the current KV store but not in the backup are not deleted or affected in any way.
Complete restoring the search head cluster environment
Finish restoring Splunk Enterprise Security from backup in an SHC environment.
- In the
$SPLUNK_HOME/etc/system/local/server.conffile, locate the
- Update the field ID in this stanza with the GUID copied from the
server.conffile during backup.
- Run the following command to restart Splunk Enterprise:
Install Splunk Enterprise Security
Deploy add-ons included with Splunk Enterprise Security
This documentation applies to the following versions of Splunk® Enterprise Security: 5.1.0, 5.1.1