Upgrade Splunk Enterprise Security in a search head cluster environment
Splunk Enterprise Security supports installation on Linux-based search head clusters (SHC) only. At this time, Windows search head clusters are not supported by Splunk Enterprise Security.
Upgrading Enterprise Security in a search head cluster environment
The installer dynamically detects if you're upgrading in a single search head environment or search head cluster environment. The installer is also bigger than the default upload limit for Splunk Web.
During an upgrade, large apps such as the Scientific for Python might not be pushed by the deployer to all the search head cluster members. If this occurs, you can increase the limit of the
max_content_length setting in the
[httpServer] stanza of the
server.conf configuration file as outlined in the following steps:
To upgrade Enterprise Security on a search head cluster deployer:
- Prepare the deployer. See Prerequisites for installing Enterprise Security in a search head cluster environment.
- Verify that you have the same version of Enterprise Security on the deployer and SHC nodes.
- Increase the Splunk Web upload limit to 2 GB by creating a file called
$SPLUNK_HOME/etc/system/local/web.confwith the following stanza.[settings]
max_upload_size = 2048
- Verify that the
splunkdConnectionTimeoutsetting in the
[settings]stanza of the
web.confconfiguration file is set to
splunkdConnectionTimeout = 300
- Increase the limit of the
max_content_lengthsetting in the
[httpServer]stanza of the
server.confconfiguration file to
max_content_length = 5000000000
Changing the configuration setting to
5000000000increases the size limit of downloadable apps to 5 GB from a default value of approximately 2 GB and enables the app to be deployed on all search head cluster members.
You can change the configuration setting
max_content_lengthto 5 GB and expand the size capacity for both the search head cluster deployer and the search head cluster members.
- To restart Splunk from the Splunk toolbar, select Settings > Server controls and click Restart Splunk.
- Install Enterprise Security on the deployer (this method is via the UI).
- On the Splunk toolbar, select Apps > Manage Apps and click Install App from File.
- Click Choose File and select the Splunk Enterprise Security product file.
- Check the checkbox for Upgrade App.
- Click Upload.
- Click Restart Now.
- Click the Enterprise Security app.
- Click Continue to app setup page.
Note the message that Enterprise Security is being installed on the deployer of a search head cluster environment and that technology add-ons will not be installed as part of the post-install configuration.
- Click Start Configuration Process.
Upgrading Splunk Enterprise Security from the command line in a search head cluster environment
Follow these steps to upgrade Splunk Enterprise Security using the Splunk software command line. See About the CLI for more information about the Splunk software command line.
- Install Splunk Enterprise Security on the deployer using the
./splunk install app <filename>command. Alternatively, you can perform a REST call to start the installation from the server command line.
For example:curl -k -u admin:password https://localhost:8089/services/apps/local -d filename="true" -d name="<file name and directory>" -d update="true" -v
- On the deployer, use the Splunk software command line to run the following command: splunk search '| essinstall --deployment_type shc_deployer' -auth admin:password
On the command line, the installer doesn't auto detect if it is being launched from a deployer. Therefore, it is necessary to add a command line option:
'--deployment_type', default='search_head', choices=['search_head', 'shc_deployer'], help='select deployment type'.
- The preferred setting for
strict, which is the default value for security reasons, especially in FedRamp deployed environments. However, you can use the following table to identify the optimal value for
ssl_enablementduring your installation:
web.confconfiguration file is the following location in a search head cluster environment:
web.confconfiguration file is the following location in a search head environment:
SSL mode Description strict Default mode
Ensure that SSL is enabled in the
web.confconfiguration file to use this mode. Otherwise, the installer exits with an error.
auto Enables SSL in the
The auto mode does not apply to search head cluster environments and causes the
essinstallcommand to fail.
ignore Ignores whether SSL is enabled or disabled.
- Restart with ./splunk restart only if SSL is changed from disabled to enabled or vice versa.
Use the deployer to deploy Enterprise Security to the cluster members. From the deployer, run this command:
splunk apply shcluster-bundle --answer-yes -target <URI>:<management_port> -auth <username>:<password>
If you run the search command to install Enterprise Security in Splunk Web, you can review the progress of the installation as search results. If you run the search command from the command line, you can review the installation log in:
For more information on installing Splunk Enterprise Security in a search head cluster environment, see Install Splunk Enterprise in a search head cluster environment.
Deploy the changes to the cluster members
As of 7.3.0, Splunk Enterprise has four deployer modes for pushing application configuration changes to search head cluster members.
The default configuration mode is
merge_to_default option pushes the app bundle from the deployer to the members and merges the
$SPLUNK_HOME/shcluster/apps/<appname>/local folders of the deployer to overwrite the
$SPLUNK_HOME/etc/apps/<appname>/default folder of each search head cluster member.
merge_to_default section of the Choose a deployer push mode in the Splunk Enterprise Distributed Search Manual.
In addition, lookups were previously preserved for all apps or for no apps. As of Splunk Enterprise 7.3.0, you're able to select the specific apps where you want to preserve lookups. See Preserve lookup files across app upgrades in the Splunk Enterprise Distributed Search Manual.
Splunk Enterprise 7.3.0 is not a requirement for upgrading, but you need Splunk Enterprise 7.3.0 or later if you want to take advantage of the deployer modes and the per-app lookup preservation.
To deploy the app to cluster members for Splunk Enterprise Security:
- Choose a deployer push mode, such as
fullto configure system wide for the first time or
merge_to_defaultto configure on a per-app basis. See the Choose a deployer push mode in the Splunk Enterprise Distributed Search Manual.
- Use the deployer to deploy Enterprise Security to the cluster members. From the deployer, run this command:
splunk apply shcluster-bundle
As of Enterprise Security 6.2.0, the default for the deployer's apply shcluster-bundle
-preserve-lookups option is
true to retain lookup file content generated on the search head cluster members. The
[shclustering] stanza is now also included in the app.conf file of each bundled domain add-on (DA) and supporting add-on (SA) in Splunk Enterprise Security. The
-preserve-lookups true argument, combined with
deployer_lookups_push_mode in the app's app.conf file indicates how csv lookup files in the app are deployed. See shclustering in the Splunk Enterprise Admin Manual.
If you do not want to retain the lookup file content on cluster members for a particular app, you can comment out
always_preserve in the
[shclustering] stanza of
$SPLUNK_HOME/shcluster/apps/<appname>/local and it persists as your local setting from now on.
Validate the configuration on the search cluster
After you distribute the copy of Enterprise Security on the deployer to the search head cluster members, use the ES Configuration Health dashboard to compare the cluster-replicated knowledge objects to the latest installation of Enterprise Security.
- Log in to Splunk Web on a search head cluster member.
- Open Enterprise Security.
- From the Enterprise Security menu bar, select Audit > ES Configuration Health.
- Review potential conflicts and changes to the default settings.
See ES Configuration Health in Use Splunk Enterprise Security.
Upgrade Splunk Enterprise Security
This documentation applies to the following versions of Splunk® Enterprise Security: 7.0.1, 7.0.2, 7.1.0, 7.1.1
Feedback submitted, thanks!