Upgrade Splunk Enterprise Security in a search head cluster environment
Splunk Enterprise Security supports installation on Linux-based search head clusters (SHC) only. At this time, Windows search head clusters are not supported by Splunk Enterprise Security.
Upgrading Enterprise Security in a search head cluster environment
The installer dynamically detects if you're upgrading in a single search head environment or search head cluster environment. The installer is also bigger than the default upload limit for Splunk Web.
During an upgrade, large apps such as the Scientific for Python might not be pushed by the deployer to all the search head cluster members. If this occurs, you can increase the limit of the max_content_length
setting in the [httpServer]
stanza of the server.conf
configuration file as outlined in the following steps:
To upgrade Enterprise Security on a search head cluster deployer:
- Prepare the deployer. See Prerequisites for installing Enterprise Security in a search head cluster environment.
- Verify that you have the same version of Enterprise Security on the deployer and SHC nodes.
- Increase the Splunk Web upload limit to 2 GB by creating a file called
$SPLUNK_HOME/etc/system/local/web.conf
with the following stanza.[settings]
max_upload_size = 2048 - Verify that the
splunkdConnectionTimeout
setting in the[settings]
stanza of theweb.conf
configuration file is set to300
.[settings]
splunkdConnectionTimeout = 300 - Increase the limit of the
max_content_length
setting in the[httpServer]
stanza of theserver.conf
configuration file to5000000000
.[httpServer]
max_content_length = 5000000000
Changing the configuration setting to5000000000
increases the size limit of downloadable apps to 5 GB from a default value of approximately 2 GB and enables the app to be deployed on all search head cluster members.You must change the configuration setting
max_content_length
to 5 GB and expand the size capacity for both the search head cluster deployer and the search head cluster members. - To restart Splunk from the Splunk toolbar, select Settings > Server controls and click Restart Splunk.
- Install Enterprise Security on the deployer (this method is via the UI).
- On the Splunk toolbar, select Apps > Manage Apps and click Install App from File.
- Click Choose File and select the Splunk Enterprise Security product file.
- Check the checkbox for Upgrade App.
- Click Upload.
- Click Restart Now.
- Click the Enterprise Security app.
- Click Continue to app setup page.
Note the message that Enterprise Security is being installed on the deployer of a search head cluster environment and that technology add-ons will not be installed as part of the post-install configuration.
- Click Start Configuration Process.
Upgrading Splunk Enterprise Security from the command line in a search head cluster environment
Follow these steps to upgrade Splunk Enterprise Security using the Splunk software command line. See About the CLI for more information about the Splunk software command line.
- Install Splunk Enterprise Security on the deployer using the
./splunk install app <filename>
command. Alternatively, you can perform a REST call to start the installation from the server command line.
For example:curl -k -u admin:password https://localhost:8089/services/apps/local -d filename="true" -d name="<file name and directory>" -d update="true" -v - On the deployer, use the Splunk software command line to run the following command: splunk search '| essinstall --deployment_type shc_deployer' -auth admin:password
On the command line, the installer doesn't auto detect if it is being launched from a deployer. Therefore, it is necessary to add a command line option:'--deployment_type', default='search_head', choices=['search_head', 'shc_deployer'], help='select deployment type'
. - The preferred setting for
ssl_enablement
isstrict
, which is the default value for security reasons, especially in FedRamp deployed environments. However, you can use the following table to identify the optimal value forssl_enablement
during your installation:
Theweb.conf
configuration file is the following location in a search head cluster environment:etc/shccluster/apps
. Theweb.conf
configuration file is the following location in a search head environment:etc/system/local/web.conf
.SSL mode Description strict Default mode
Ensure that SSL is enabled in theweb.conf
configuration file to use this mode. Otherwise, the installer exits with an error.auto Enables SSL in the etc/system/local/web.conf
configuration file.The auto mode does not apply to search head cluster environments and causes the
essinstall
command to fail.ignore Ignores whether SSL is enabled or disabled. - Restart with ./splunk restart only if SSL is changed from disabled to enabled or vice versa.
-
Use the deployer to deploy Enterprise Security to the cluster members. From the deployer, run this command:
splunk apply shcluster-bundle --answer-yes -target <URI>:<management_port> -auth <username>:<password>
If you run the search command to install Enterprise Security in Splunk Web, you can review the progress of the installation as search results. If you run the search command from the command line, you can review the installation log in: $SPLUNK_HOME/var/log/splunk/essinstaller2.log
.
For more information on installing Splunk Enterprise Security in a search head cluster environment, see Install Splunk Enterprise in a search head cluster environment.
Deploy the changes to the cluster members
As of 7.3.0, Splunk Enterprise has four deployer modes for pushing application configuration changes to search head cluster members.
The default configuration mode is merge_to_default
. The merge_to_default
option pushes the app bundle from the deployer to the members and merges the $SPLUNK_HOME/shcluster/apps/<appname>/default
and $SPLUNK_HOME/shcluster/apps/<appname>/local
folders of the deployer to overwrite the $SPLUNK_HOME/etc/apps/<appname>/default
folder of each search head cluster member.
See the merge_to_default
section of the Choose a deployer push mode in the Splunk Enterprise Distributed Search Manual.
In addition, lookups were previously preserved for all apps or for no apps. As of Splunk Enterprise 7.3.0, you're able to select the specific apps where you want to preserve lookups. See Preserve lookup files across app upgrades in the Splunk Enterprise Distributed Search Manual.
Splunk Enterprise 7.3.0 is not a requirement for upgrading, but you need Splunk Enterprise 7.3.0 or later if you want to take advantage of the deployer modes and the per-app lookup preservation.
To deploy the app to cluster members for Splunk Enterprise Security:
- Choose a deployer push mode, such as
full
to configure system wide for the first time ormerge_to_default
to configure on a per-app basis. See the Choose a deployer push mode in the Splunk Enterprise Distributed Search Manual. - Use the deployer to deploy Enterprise Security to the cluster members. From the deployer, run this command:
splunk apply shcluster-bundle
As of Enterprise Security 6.2.0, the default for the deployer's apply shcluster-bundle -preserve-lookups
option is true
to retain lookup file content generated on the search head cluster members. The [shclustering]
stanza is now also included in the app.conf file of each bundled domain add-on (DA) and supporting add-on (SA) in Splunk Enterprise Security. The -preserve-lookups true
argument, combined with deployer_lookups_push_mode
in the app's app.conf file indicates how csv lookup files in the app are deployed. See shclustering in the Splunk Enterprise Admin Manual.
If you do not want to retain the lookup file content on cluster members for a particular app, you can comment out deployer_lookups_push_mode
of always_preserve
in the [shclustering]
stanza of $SPLUNK_HOME/shcluster/apps/<appname>/local
and it persists as your local setting from now on.
Validate the configuration on the search cluster
After you distribute the copy of Enterprise Security on the deployer to the search head cluster members, use the ES Configuration Health dashboard to compare the cluster-replicated knowledge objects to the latest installation of Enterprise Security.
- Log in to Splunk Web on a search head cluster member.
- Open Enterprise Security.
- From the Enterprise Security menu bar, select Audit > ES Configuration Health.
- Review potential conflicts and changes to the default settings.
See ES Configuration Health in Use Splunk Enterprise Security.
Upgrade Splunk Enterprise Security |
This documentation applies to the following versions of Splunk® Enterprise Security: 7.0.1, 7.0.2, 7.1.0, 7.1.1, 7.1.2, 7.2.0, 7.3.0, 7.3.1, 7.3.2
Feedback submitted, thanks!