Upgrade Splunk Enterprise Security in a search head cluster environment

Splunk Enterprise Security supports installation on Linux-based search head clusters (SHC) only. At this time, Windows search head clusters are not supported by Splunk Enterprise Security.

Upgrading Enterprise Security in a search head cluster environment

The installer dynamically detects if you're upgrading in a single search head environment or search head cluster environment. The installer is also bigger than the default upload limit for Splunk Web.

During an upgrade, large apps such as the Scientific for Python might not be pushed by the deployer to all the search head cluster members. If this occurs, you can increase the limit of the max_content_length setting in the [httpServer] stanza of the server.conf configuration file as outlined in the following steps:

To upgrade Enterprise Security on a search head cluster deployer:

1. Prepare the deployer. See Prerequisites for installing Enterprise Security in a search head cluster environment.
2. Verify that you have the same version of Enterprise Security on the deployer and SHC nodes.
3. Increase the Splunk Web upload limit to 2 GB by creating a file called $SPLUNK_HOME/etc/system/local/web.conf with the following stanza.
   
   [settings]
   max_upload_size = 2048
4. Verify that the splunkdConnectionTimeout setting in the [settings] stanza of the web.conf configuration file is set to 300.
   
   [settings]
   splunkdConnectionTimeout = 300
   
   
5. Increase the limit of the max_content_length setting in the [httpServer] stanza of the server.conf configuration file to 5000000000 .
   
   [httpServer]
   max_content_length = 5000000000
   
   Changing the configuration setting to 5000000000 increases the size limit of downloadable apps to 5 GB from a default value of approximately 2 GB and enables the app to be deployed on all search head cluster members.

   You must change the configuration setting max_content_length to 5 GB and expand the size capacity for both the search head cluster deployer and the search head cluster members.

6. To restart Splunk from the Splunk toolbar, select Settings > Server controls and click Restart Splunk.
7. Install Enterprise Security on the deployer (this method is via the UI).
   1. On the Splunk toolbar, select Apps > Manage Apps and click Install App from File.
   2. Click Choose File and select the Splunk Enterprise Security product file.
   3. Check the checkbox for Upgrade App.
   4. Click Upload.
8. Click Restart Now.
9. Click the Enterprise Security app.
10. Click Continue to app setup page.
    

    Note the message that Enterprise Security is being installed on the deployer of a search head cluster environment and that technology add-ons will not be installed as part of the post-install configuration.

11. Click Start Configuration Process.

Upgrading Splunk Enterprise Security from the command line in a search head cluster environment

Follow these steps to upgrade Splunk Enterprise Security using the Splunk software command line. See About the CLI for more information about the Splunk software command line.

1. Install Splunk Enterprise Security on the deployer using the ./splunk install app <filename> command. Alternatively, you can perform a REST call to start the installation from the server command line.
   For example:
   curl -k -u admin:password https://localhost:8089/services/apps/local -d filename="true" -d name="<file name and directory>" -d update="true" -v
2. On the deployer, use the Splunk software command line to run the following command:
   
   splunk search '| essinstall --deployment_type shc_deployer' -auth admin:password
   
   On the command line, the installer doesn't auto detect if it is being launched from a deployer. Therefore, it is necessary to add a command line option: '--deployment_type', default='search_head', choices=['search_head', 'shc_deployer'], help='select deployment type'.
3. The preferred setting for ssl_enablement is strict, which is the default value for security reasons, especially in FedRamp deployed environments. However, you can use the following table to identify the optimal value for ssl_enablement during your installation:
   The web.conf configuration file is the following location in a search head cluster environment: etc/shccluster/apps. The web.conf configuration file is the following location in a search head environment: etc/system/local/web.conf.

   SSL mode	Description
   strict	Default mode Ensure that SSL is enabled in the web.conf configuration file to use this mode. Otherwise, the installer exits with an error.
   auto	Enables SSL in the etc/system/local/web.conf configuration file. The auto mode does not apply to search head cluster environments and causes the essinstall command to fail.
   ignore	Ignores whether SSL is enabled or disabled.

4. Restart with ./splunk restart only if SSL is changed from disabled to enabled or vice versa.
5. Use the deployer to deploy Enterprise Security to the cluster members. From the deployer, run this command:
   
   splunk apply shcluster-bundle --answer-yes -target <URI>:<management_port> -auth <username>:<password>

If you run the search command to install Enterprise Security in Splunk Web, you can review the progress of the installation as search results. If you run the search command from the command line, you can review the installation log in: $SPLUNK_HOME/var/log/splunk/essinstaller2.log.

For more information on installing Splunk Enterprise Security in a search head cluster environment, see Install Splunk Enterprise in a search head cluster environment.

Deploy the changes to the cluster members

As of 7.3.0, Splunk Enterprise has four deployer modes for pushing application configuration changes to search head cluster members.

The default configuration mode is merge_to_default. The merge_to_default option pushes the app bundle from the deployer to the members and merges the $SPLUNK_HOME/shcluster/apps/<appname>/default and $SPLUNK_HOME/shcluster/apps/<appname>/local folders of the deployer to overwrite the $SPLUNK_HOME/etc/apps/<appname>/default folder of each search head cluster member.

See the merge_to_default section of the Choose a deployer push mode in the Splunk Enterprise Distributed Search Manual.

In addition, lookups were previously preserved for all apps or for no apps. As of Splunk Enterprise 7.3.0, you're able to select the specific apps where you want to preserve lookups. See Preserve lookup files across app upgrades in the Splunk Enterprise Distributed Search Manual.

Splunk Enterprise 7.3.0 is not a requirement for upgrading, but you need Splunk Enterprise 7.3.0 or later if you want to take advantage of the deployer modes and the per-app lookup preservation.

To deploy the app to cluster members for Splunk Enterprise Security:

1. Choose a deployer push mode, such as full to configure system wide for the first time or merge_to_default to configure on a per-app basis. See the Choose a deployer push mode in the Splunk Enterprise Distributed Search Manual.
2. Use the deployer to deploy Enterprise Security to the cluster members. From the deployer, run this command:
   splunk apply shcluster-bundle

As of Enterprise Security 6.2.0, the default for the deployer's apply shcluster-bundle -preserve-lookups option is true to retain lookup file content generated on the search head cluster members. The [shclustering] stanza is now also included in the app.conf file of each bundled domain add-on (DA) and supporting add-on (SA) in Splunk Enterprise Security. The -preserve-lookups true argument, combined with deployer_lookups_push_mode in the app's app.conf file indicates how csv lookup files in the app are deployed. See shclustering in the Splunk Enterprise Admin Manual.

If you do not want to retain the lookup file content on cluster members for a particular app, you can comment out deployer_lookups_push_mode of always_preserve in the [shclustering] stanza of $SPLUNK_HOME/shcluster/apps/<appname>/local and it persists as your local setting from now on.

Validate the configuration on the search cluster

After you distribute the copy of Enterprise Security on the deployer to the search head cluster members, use the ES Configuration Health dashboard to compare the cluster-replicated knowledge objects to the latest installation of Enterprise Security.

1. Log in to Splunk Web on a search head cluster member.
2. Open Enterprise Security.
3. From the Enterprise Security menu bar, select Audit > ES Configuration Health.
4. Review potential conflicts and changes to the default settings.

See ES Configuration Health in Use Splunk Enterprise Security.