Upgrade Splunk Enterprise Security in a search head cluster environment

Splunk Enterprise Security supports installation on Linux-based search head clusters (SHC) only. At this time, Windows search head clusters are not supported by Splunk Enterprise Security.

Upgrading Enterprise Security in a search head cluster environment

The installer dynamically detects if you're upgrading in a single search head environment or search head cluster environment. The installer is also bigger than the default upload limit for Splunk Web.

To upgrade Enterprise Security on a search head cluster deployer:

1. Prepare the deployer. See Prerequisites for installing Enterprise Security in a search head cluster environment.
2. Verify that you have the same version of Enterprise Security on the deployer and SHC nodes.
3. Increase the Splunk Web upload limit to 1GB by creating a file called $SPLUNK_HOME/etc/system/local/web.conf with the following stanza.
   
   [settings]
   max_upload_size = 1024
4. To restart Splunk from the Splunk toolbar, select Settings > Server controls and click Restart Splunk.
5. Install Enterprise Security on the deployer (this method is via the UI).
   1. On the Splunk toolbar, select Apps > Manage Apps and click Install App from File.
   2. Click Choose File and select the Splunk Enterprise Security product file.
   3. Check the checkbox for Upgrade App.
   4. Click Upload.
6. Click Restart Now.
7. Click the Enterprise Security app.
8. Click Continue to app setup page.
   

   Note the message that Enterprise Security is being installed on the deployer of a search head cluster environment and that technology add-ons will not be installed as part of the post-install configuration.

9. Click Start Configuration Process.

For more information on installing Splunk Enterprise Security in a search head cluster environment, see Install Splunk Enterprise in a search head cluster environment.

Deploy the changes to the cluster members

As of 7.3.0, Splunk Enterprise has four deployer modes for pushing application configuration changes to search head cluster members.

The previous behavior for pushing the app bundle from the deployer to the members was to merge the $SPLUNK_HOME/shcluster/apps/<appname>/default and $SPLUNK_HOME/shcluster/apps/<appname>/local folders of the deployer to overwrite the $SPLUNK_HOME/etc/apps/<appname>/default folder of each SHC member.

Although that merge behavior is still available as one of the configuration options, the default behavior is to duplicate $SPLUNK_HOME/shcluster/apps/<appname>/default along with $SPLUNK_HOME/shcluster/apps/<appname>/local on the SHC members. See the "Mode_merge_to_default" section of the Choose a deployer push mode in the Splunk Enterprise Distributed Search Manual.

In addition, lookups were previously preserved for all apps or for no apps. As of Splunk Enterprise 7.3.0, you're able to select the specific apps where you want to preserve lookups. See Preserve lookup files across app upgrades in the Splunk Enterprise Distributed Search Manual.

Splunk Enterprise 7.3.0 is not a requirement for upgrading, but you need Splunk Enterprise 7.3.0 or later if you want to take advantage of the deployer modes and the per-app lookup preservation.

To deploy the app to cluster members for Splunk Enterprise Security:

1. Choose a deployer push mode, such as full to configure system wide for the first time or merge_to_default to configure on a per-app basis. See the Choose a deployer push mode in the Splunk Enterprise Distributed Search Manual.
2. Use the deployer to deploy Enterprise Security to the cluster members. From the deployer, run this command:
   splunk apply shcluster-bundle

As of Enterprise Security 6.2.0, the default for the deployer's apply shcluster-bundle -preserve-lookups option is true to retain lookup file content generated on the search head cluster members. The [shclustering] stanza is now also included in the app.conf file of each bundled domain add-on (DA) and supporting add-on (SA) in Splunk Enterprise Security. The -preserve-lookups true argument, combined with deployer_lookups_push_mode in the app's app.conf file indicates how csv lookup files in the app are deployed. See shclustering in the Splunk Enterprise Admin Manual.

If you do not want to retain the lookup file content on cluster members for a particular app, you can comment out deployer_lookups_push_mode of always_preserve in the [shclustering] stanza of $SPLUNK_HOME/shcluster/apps/<appname>/local and it persists as your local setting from now on.

Validate the configuration on the search cluster

After you distribute the copy of Enterprise Security on the deployer to the search head cluster members, use the ES Configuration Health dashboard to compare the cluster-replicated knowledge objects to the latest installation of Enterprise Security.

1. Log in to Splunk Web on a search head cluster member.
2. Open Enterprise Security.
3. From the Enterprise Security menu bar, select Audit > ES Configuration Health.
4. Review potential conflicts and changes to the default settings.

See ES Configuration Health in Use Splunk Enterprise Security.