Install Splunk Enterprise Security in a search head cluster environment

Splunk Enterprise Security has specific requirements and processes for implementing search head clustering.

• For an overview of search head clustering, see Search head clustering architecture in the Distributed Search Manual.
• For a complete list of search head clustering requirements, see System requirements and other deployment considerations for search head clusters in the Distributed Search Manual.

Do not install Splunk Enterprise Security on a stretched search head cluster in a multi-site indexer cluster deployment. Splunk Enterprise Security must be installed on a single dedicated search head cluster contained within a site since the app requires a consistent set of runtime artifacts. This cannot be guaranteed in a stretched search head cluster when a site outage occurs. Third party technology can be used to help recover a Splunk Enterprise Security search head from a site failure. You can failover the search head instances or provision a warm standby of the Splunk Enterprise Security search head to keep it in sync with the primary Splunk Enterprise Security environment. Contact Splunk Professional Services when deploying Splunk Enterprise Security in a high-availability or a disaster recovery scenario.

For more information on deploying a search head cluster in a multi-site environment, see Deploy a search head cluster in a multisite environment in the Splunk® Enterprise Distributed Search manual.

If you are installing Enterprise Security on an existing search head cluster environment which might have other apps deployed already, all of the steps in this section apply. Be careful to not delete or remove any existing content in the $SPLUNK_HOME/etc/shcluster/apps folder.

Differences between deploying on a search head and a search head cluster environment

The following table outlines the primary differences between deploying Splunk Enterprise Security on a search head and a search head cluster environment:

Deployment on a search head	Deployment on a search head cluster
Splunk Enterprise Security is deployed on etc/apps.	Splunk Enterprise Security is deployed on etc/shcluster/apps.
The value of deployment_type setting in the searchbnf.conf configuration file is search_head.	The value of the deployment_type setting in the searchbnf.conf configuration file is shc_deployer.
You can automatically enable SSL to update the web.conf configuration file. However, you also have the option to configure the ssl_enablement setting to strict, ignore, or auto in the web.conf configuration file.	You cannot automatically enable SSL to deploy Splunk Enterprise Security in a search head cluster environment. In a search head cluster environment, the ssl_enablement setting must be set to the default value of strict in the web.conf configuration file for security purposes.

Prerequisites for installing Enterprise Security in a search head cluster environment

Splunk Enterprise Security supports installation on Linux-based search head clusters only. At this time, Windows search head clusters are not supported by Splunk Enterprise Security.

Before installing Enterprise Security in a search head cluster environment, verify that you have:

• One deployer
• The same version of Splunk Enterprise on the deployer and search head cluster nodes
• The same app versions of any other apps on the deployer and search head cluster nodes (not yet including Enterprise Security)
• The backup of etc/shcluster/apps on the deployer before installing Enterprise Security
• The backup of etc/apps from one of search head cluster nodes
• The backup of the KVstore from one of search head cluster nodes

Installing Splunk Enterprise Security in a search head cluster environment

The installer dynamically detects if you're installing in a single search head environment or search head cluster environment. The installer is also bigger than the default upload limit for Splunk Web. Follow these steps to install Splunk Enterprise Security on a search head cluster:

1. Prepare the deployer using the instructions in Prerequisites for installing Splunk Enterprise Security in a search head cluster environment.
2. Increase the Splunk Web upload limit to 2 GB by creating a file called $SPLUNK_HOME/etc/system/local/web.conf with the following stanza.
   
   [settings]
   max_upload_size = 2048
3. Verify that the splunkdConnectionTimeout setting in the [settings] stanza of the web.conf configuration file is set to 300.
   
   [settings]
   splunkdConnectionTimeout = 300
   
   
4. Increase the limit of the max_content_length setting in the [httpServer] stanza of the server.conf configuration file to 5000000000 .
   
   [httpServer]
   max_content_length = 5000000000
   
   Changing the configuration setting to 5000000000 increases the size limit of downloadable apps to 5 GB from a default value of approximately 2 GB and enables the app to be deployed on all search head cluster members.

   You must change the configuration setting max_content_length to 5 GB and expand the size capacity for both the search head cluster deployer and the search head cluster members.

5. Install Enterprise Security on the deployer.
6. On the Splunk toolbar, select Apps.
7. Select Manage apps and then select Install app from file.
8. Select Choose File and select the Splunk Enterprise Security product file.
9. Select Upload to begin the installation.
10. Select Continue to app setup page
    

    Note the message that Splunk Enterprise Security is being installed on the deployer of a search head cluster environment and that technology add-ons will not be installed as part of the post-install configuration.

11. Select Start Configuration Process.
12. If you are not using Secure Sockets Layer (SSL) in your environment, do one of the following steps when you see the SSL Warning message:
    1. Select Enable SSL to turn on SSL and start using https:// for encrypted data transfer.
    2. Select Do Not Enable SSL to keep SSL turned off and continue using http:// for data transfer.
13. Wait for the process to complete.
14. Verify that the DA-ESS and SA apps exist in the /installdirectory. The DA-ESS and SA apps in SplunkEnterpriseSecuritySuite are automatically extracted and deployed throughout the search head cluster. However, if there is a delay in the synchronization, and if the synchronization isn't automatically completed within a couple of minutes, you can move the SplunkEnterpriseSecuritySuite from $SPLUNK_HOME/etc/apps to $SPLUNK_HOME/etc/shcluster/apps.
15. Use the deployer to deploy Splunk Enterprise Security to the cluster members. From the deployer, run this command:
    splunk apply shcluster-bundle --answer-yes -target <URI>:<management_port> -auth <username>:<password>
    
    For more information on using the deployer to distribute apps and configuration updates to search head cluster members, see Use the deployer to distribute apps and configuration updates.

To verify that Splunk Enterprise Security is deployed to the cluster members:

• From the GUI of a cluster member, you can check the Help > About menu to check the version number.
• From the CLI of a cluster member, you can check the /etc/apps directory to verify the supporting add-ons and domain add-ons for Enterprise Security: DA-ESS-AccessProtection, DA-ESS-EndpointProtection, DA-ESS-IdentityManagement, DA-ESS-NetworkProtection, DA-ESS-ThreatIntelligence, SA-AccessProtection, SA-AuditAndDataProtection, SA-EndpointProtection, SA-IdentityManagement, SA-NetworkProtection, SA-ThreatIntelligence, SA-UEBA, SA-Utils, Splunk_DA-ESS_PCICompliance, SplunkEnterpriseSecuritySuite, Splunk_SA_CIM, Splunk_ML_Toolkit, and Splunk_SA_Scientific_Python_linux_x86_64 (or Splunk_SA_Scientific_Python_windows_x86_64 for windows)
• From the CLI of a cluster member, you can check the $SPLUNK_HOME/etc/apps/SplunkEnterpriseSecuritySuite/local/inputs.conf file to see that the data model accelerations settings are enabled.

Although technology add-ons are bundled in the installer, they are not deployed as part of the installation process. You must deploy them manually if you want to use them. See Deploy add-ons included with Splunk Enterprise Security.

Installing Splunk Enterprise Security from the command line in a search head cluster environment

Follow these steps to install Splunk Enterprise Security using the Splunk software command line. See About the CLI for more information about the Splunk software command line.

1. Follow the instructions in Download Splunk Enterprise Security to download Splunk Enterprise Security and place it on the deployer.
2. Install Splunk Enterprise Security on the deployer using the ./splunk install app <filename> command. Alternatively, you can perform a REST call to start the installation from the server command line.
   For example:
   curl -k -u admin:password https://localhost:8089/services/apps/local -d filename="true" -d name="<file name and directory>" -d update="true" -v
3. On the deployer, use the Splunk software command line to run the following command:
   
   splunk search '| essinstall --deployment_type shc_deployer' -auth admin:password
   
   On the command line, the installer doesn't auto detect if it is being launched from a deployer. Therefore, it is necessary to add a command line option: '--deployment_type', default='search_head', choices=['search_head', 'shc_deployer'], help='select deployment type'.
4. The preferred setting for ssl_enablement is strict, which is the default value for security reasons, especially in FedRamp deployed environments. However, you can use the following table to identify the optimal value for ssl_enablement during your installation:
   The web.conf configuration file is the following location in a search head cluster environment: etc/shcluster/apps. The web.conf configuration file is the following location in a search head environment: etc/system/local/web.conf.

   SSL mode	Description
   strict	Default mode Ensure that SSL is enabled in the web.conf configuration file to use this mode. Otherwise, the installer exits with an error.
   auto	Enables SSL in the etc/system/local/web.conf configuration file. The auto mode does not apply to search head cluster environments and causes the essinstall command to fail.
   ignore	Ignores whether SSL is enabled or disabled.

5. Restart with ./splunk restart only if SSL is changed from disabled to enabled or vice versa.
6. Use the deployer to deploy Enterprise Security to the cluster members. From the deployer, run this command:
   
   splunk apply shcluster-bundle --answer-yes -target <URI>:<management_port> -auth <username>:<password>

If you run the search command to install Enterprise Security in Splunk Web, you can review the progress of the installation as search results. If you run the search command from the command line, you can review the installation log in: $SPLUNK_HOME/var/log/splunk/essinstaller2.log.

Managing configuration changes in a search head cluster

Some system configuration changes must be deployed using the deployer.

Follow these steps to manage configuration changes in a search head cluster:

1. Instead of making the changes on a search head cluster member, make the changes on a deployer.
2. Migrate the necessary files to the search head cluster deployer.
3. Deploy the updated configuration to the search head cluster.

Use the following table to make configuration changes using the deployer:

Configuration change	File modified
Activate or deactivate indexed real-time searches on the General Settings page.	inputs.conf
Modify the indexed real-time disk sync delay on the General Settings page.	inputs.conf

Most configuration changes that you make in a search head cluster replicate automatically to other search head cluster members. For example:

• Add, modify, and disable threat intelligence sources
• Add, modify, and disable asset and identity source lists
• Changes to the user interface
• Changes to searches

See How configuration changes propagate across the search head cluster in the Distributed Search Manual.

Migrate an existing search head to a search head cluster

An Enterprise Security standalone search head or search head pool member cannot be added to a search head cluster. To migrate ES configurations to a search head cluster:

1. Identify any custom configurations and modifications in the prior Splunk Enterprise Security installation. Check to make sure there is no local copy of ess_setup.conf that could conflict with the default one when you deploy Enterprise Security to the cluster.
2. Implement a new search head cluster.
3. Deploy the latest version of Enterprise Security on the search head cluster.
4. Review and migrate the customized configurations to the search head cluster deployer for replication to the cluster members.
5. Shut down the old Splunk Enterprise Security search head.

For more information, see the topic Migrate from a standalone search head to a search head cluster in the Splunk Enterprise Distributed Search Manual.

For assistance in planning a Splunk Enterprise Security deployment migration, contact Splunk Professional Services.

Back up and restore Splunk Enterprise Security in a search head cluster environment

Back up and restore a Splunk Enterprise Security search head cluster (SHC) environment with at least three SHC nodes. All of the nodes in the SHC must be running the same version of Splunk Enterprise Security. Restoring a SHC environment might be necessary in the event of a disaster.

Take regular backups from the SHC, so that you have a backup from a time when the environment is healthy. For example, you could automate taking backups every hour. Choose a frequency of backups based on recovery point objectives.

To check if your environment is healthy, you can use one of the following methods:

• CLI command: ./splunk show shcluster-status –verbose
• API: /services/shcluster/status?advanced=1

In the output, look for the following fields:

Field	Description
dynamic_captain	Whether the cluster has a dynamically elected captain.
stable_captain	Whether the cluster captain is in a stable state.
service_ready_flag	Whether the cluster has enough members to support replication factor.
splunk_version	Whether all members, including the cluster master, are running the same Splunk version.
out_of_sync	Whether all nodes are currently in-sync.

Back up a search head cluster environment

Back up an SHC environment by backing up the KV store, the deployer, and the SHC nodes. The backup procedure does not require shutting down the SHC cluster or any node in the cluster.

Back up the KV store

To back up the KV store, run the following command from the CLI from the SHC node with the most recent data:

splunk backup kvstore -auth "admin:<password>"

This command creates an archive file in the $SPLUNK_HOME/var/lib/splunk/kvstorebackup directory. For example, the file might be named kvdump_example.tar.gz.

Back up the deployer and search head cluster nodes

1. On the deployer, back up the files in the $SPLUNK_HOME/etc/shcluster directory.
2. On the SHC node with the most recent data, note the GUID from the shclustering stanza from the $SPLUNK_HOME/etc/system/local/server.conf file. This information is necessary during the restore process.
3. On the SHC node with the most recent data, back up the files in the $SPLUNK_HOME/var/run/splunk/snapshot/$LATEST_TIME-$CHECKSUM.bundle bundle.
4. Create a tar.gz file from these backups.

Restore from a backup of a search head cluster environment

You need the following information to restore from a backup:

• The GUID from the server.conf file from one of the SHC members from before you begin restoring.

  After your restore the cluster, the restored cluster will have the same GUID as the cluster that was backed up.

• A backup of the deployer.
• A backup of the SHC node with the most recent data.
• A backup of the KV store from the SHC node with the most recent data.

The restore procedure requires the SHC to be shut down, but not the KV store.

Restore the deployer

1. On the deployer, extract the deployer backup file to the $SPLUNK_HOME/etc/shcluster directory.
2. Apply the bundle with the following command:

   splunk apply shcluster-bundle

Restore the search head cluster nodes

Complete the following steps initially on the search head captain and then, on each SHC node that you want to restore:

1. Run the following command to stop Splunk Enterprise Security:

    splunk stop

2. Create a temporary folder and name it temp.
3. Extract the SHC node backup file to the temp directory.
4. Move the config.bundle file from the temp directory to the $SPLUNK_HOME/etc directory.
5. Extract the config.bundle file to the $SPLUNK_HOME/etc directory.

For more information, on deploying a search head cluster, see Deploy a search head cluster in the Splunk Enterprise Distributed Search manual.

Restore the KV store

1. On an SHC node, check the $SPLUNK_HOME/var/lib/splunk/kvstorebackup directory to make sure that the kvdump_example.tar.gz backup file that you want to use to restore is there. If it is not in that directory, manually copy your tar.gz file to that location. Note that the KV store backup file is not automatically replicated across each SHC member.
2. Ensure that Splunk Enterprise Security is installed. The collections.conf file is necessary to complete the restore.
3. Run the following command to restore the KV store:

   splunk restore kvstore -archiveName kvdump_example.tar.gz

4. Restore the snapshot bundle by extracting the backup tar file from the $SPLUNK_HOME/var/run/splunk/snapshot directory to the $SPLUNK_HOME/etc directory.
5. Repeat these steps on each SHC node that you want to restore. Restoring the KV store on one SHC node does not cause the KV store to automatically replicate across each SHC member.

Entries that are present in both the current KV store and in the backup are updated and replaced by the entry in the backup. Collections that are in the current KV store but not in the backup are preserved, but not necessarily the documents inside of the collection.

Complete restoring the search head cluster environment

Finish restoring Splunk Enterprise Security from backup in an SHC environment.

1. In the $SPLUNK_HOME/etc/system/local/server.conf file, locate the shclustering stanza.
2. Update the field ID in this stanza with the GUID copied from the server.conf file during backup.
3. Run the following command to restart Splunk:

   splunk restart

Restore the history of the analyst queue from internal audit logs

In the event that the backup process was not established prior to a data loss event with the KVStore, some of the information pertaining to analyst queue history can still be recovered using internal logs, as dictated by index _audit retention settings.

earliest=-30d index=_audit sourcetype=incident_review | rex "@@\w+,(?<rule_name>[^,]+),(?<status>[^,]*),(?<owner>[^,]*),(?<urgency>[^,]*),(?<comment>.*),(?<user>[^,]+), (?<something>[^,]+)" | eval time=_time | table comment owner rule_id rule_name status time urgency user | outputlookup append=t incident_review_lookup_REMOVE_FOR_SAFETY

The regex command might not work as expected if one of the fields has a comma (,) character because the regex searches for that character as the delimiter.

See also

For more information on installing Splunk Enterprise Security on a single instance search head, see the product documentation:

Install Splunk Enterprise Security on an on-prem search head