Upgrade Splunk Enterprise Security

This topic describes how to upgrade Splunk Enterprise Security on an on-premises search head from version 6.0.2 and higher. Splunk Cloud Platform customers work with Splunk Support to coordinate upgrades to Enterprise Security.

Step 1. Review the planning topic

1. For an overview of the upgrade process and prerequisites, see Planning an upgrade in this manual.
2. Perform a full backup of the search head, including the KV Store, before upgrading. The upgrade process does not back up the existing installation before upgrading. See Back up KV Store for instructions on how to back up the KV Store on the search head.

To back out of the upgrade, you must restore the prior version of Splunk Enterprise Security from backup.

Step 2. Download Splunk Enterprise Security

1. Open splunk.com and log in with your Splunk.com ID. You must be a licensed Enterprise Security customer to download the product.
2. Download the latest Splunk Enterprise Security product.
3. Choose Download and save the Splunk Enterprise Security product file to your desktop.
4. Log in to the Enterprise Security search head as an administrator.

Step 3. Install the latest Splunk Enterprise Security

The installer dynamically detects if you're installing in a single search head environment or search head cluster environment. The installer is also bigger than the default upload limit for Splunk Web. For more information on installing Splunk Enterprise Security, see Install Splunk Enterprise Security.

1. Increase the Splunk Web upload limit to 2 GB by creating a file called $SPLUNK_HOME/etc/system/local/web.conf with the following stanza.
   
   [settings]
   max_upload_size = 2048
2. To restart Splunk from the Splunk toolbar, select Settings > Server controls and click Restart Splunk.
3. On the Splunk Enterprise search page, select Apps > Manage Apps and choose Install App from File.

Upgrade Splunk Enterprise Security using the UI

1. Select the Splunk Enterprise Security product file.
2. Click Choose File and select the Splunk Enterprise Security product file.
3. Click Upgrade app to overwrite the existing Splunk Enterprise Security installation.
4. Click Upload to begin the installation.
5. Select one of two options: Set up now or Set up later.
6. Select Set up now > Start Configuration Process to configure Splunk Enterprise Security
7. Restart Splunk when you see messages about restarting Splunk.

For more information about restarting Splunk, see Restart your Splunk instance.

Upgrade Splunk Enterprise Security using the CLI

Follow either step 1 OR step 2 to upgrade Splunk Enterprise Security using the CLI:

1. On the search head, use the following command to start the installation process from the server command line.
   ./splunk install app <path to app> -update 1 -auth <username>:<password>
   OR
2. Use the following curl command to upgrade Splunk Enterprise Security.
   curl -k -u admin:password https://localhost:8089/services/apps/local -d filename="true" -d name="<file name and directory>" -d update="true" -v

If you do not run the setup procedure promptly after the file upload completes, Enterprise Security displays errors.

Step 4. Set up Splunk Enterprise Security

After Splunk Web returns after the restart, set up Splunk Enterprise Security.

1. Click Continue to app setup page to start the ES setup.
2. Click Start.
   # The Splunk Enterprise Security Post-Install Configuration page indicates the upgrade status as it moves through the stages of installation. When the setup is complete, the page may prompt you to restart Splunk Platform services if you opted to enable SSL before the setup.
3. Click Restart Splunk to finish the installation. For Enterprise Security version 6.4.1, you may see an error on the dashboard if you do not restart Splunk and navigate to the Incident Review page or the Investigation Dashboard.

Step 5. Validate the upgrade

The Splunk Enterprise Security upgrade process is now complete. Objects disabled during the upgrade process will automatically be enabled.

1. On the Enterprise Security menu bar, select Audit > ES Configuration Health.
2. Review potential conflicts and changes to the default settings. See ES Configuration Health in the User Manual.
3. Clear the browser cache of the browser you use to access Splunk Web to make sure that you access a fresh version of Splunk Web after upgrading. If you do not clear the browser cache, some pages might fail to load.

Splunk logs the upgrade in $SPLUNKHOME$/var/log/splunk/essinstaller2.log

Version-specific upgrade notes

Things to do after upgrading to specific versions of Enterprise Security.

After upgrading to version 7.2.0

When you upgrade to Splunk Enterprise Security version 7.2.0, the modular input confcheck_es_bias_language_cleanup displays the following error after upgrade even the modular input is disabled.

Unable to initialize modular input "confcheck_es_bias_language_cleanup" defined in the app "SplunkEnterpriseSecuritySuite"

To prevent the display of the error messages, follow these workaround steps:

1. Modify following file:
   On the search head cluster: /opt/splunk/etc/shcluster/apps/SplunkEnterpriseSecuritySuite/README/inputs.conf.spec On a standalone ES instance this file: /opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/README/inputs.conf.spec
2. Add the following comment at the end of the file:
   ###### Conf File Check for Bias Language ######
   #[confcheck_es_bias_language_cleanup://default]
   #debug = <boolean>
   
3. On the search head cluster: Push the changes to the search head cluster by pushing the bundle apps.
4. On standalone search head: follow these additional steps:
   1. (optional) Clean the messages from the top of the page so that they do not display again.
   2. Restart the Splunk process.

After upgrading to version 7.1.0

When you upgrade to Splunk Enterprise Security version 7.1.0, contributing risk events for risk notables might not be visible in the Risk Event Timeline if the risk notables are created before the upgrade and any one of the following conditions are met:

• Entity zones are enabled
• Changes are made to the entity zones that apply to existing risk notables
• Asset and identity framework is disabled

However, risk notables that are created after upgrading to Splunk Enterprise Security version 7.1.0 are displayed in the Risk Event Timeline.

Example 1:
Say, the asset and identity framework was enabled and risk notables were created. However, upon upgrading to Splunk Enterprise Security version 7.1, the asset and identity framework was disabled using Configure > Data Enrichment>Asset and Identity Management > Correlation Setup > Disable for all sourcetypes.

Now, if you click on the Risk Event Timeline to search for a risk event, you might see the following error message "Risk event search did not return any results. Please verify notable drill down search."

Example 2:
Say, entity zones were enabled and risk notables were created. However, upon upgrading to Splunk Enterprise Security version 7.1, the entity zones were reconfigured or disabled using Configure > Data Enrichment>Asset and Identity Management > Global Settings > Enable zones for assets and identities.

Now, if you click on the Risk Event Timeline to search for a risk event, you might see an error message "Risk event search did not return any results. Please verify notable drilldown search."

Workaround: Run a search on the Risk index to view the contributing risk events for a risk notable. For example: Use the following search to identify the normalized risk objects (user) "pratik" without using the reference to entity zone.
.

| from datamodel Risk.All_Risk | search risk_object="pratik"

instead of:

| from datamodel Risk.All_Risk | search normalized_risk_object="pratik_sanfrancisco"

Using the search without any reference to the entity zone provides the list of normalized risk objects. However, this list of normalized risk objects that contribute to a risk notable is not rendered in the Risk Event Timeline.

For more information on entity zones, see Enable entity zones for assets and identities in Splunk Enterprise Security

After upgrading to version 7.0.0

When you upgrade the Splunk Enterprise Security app to versions 7.0.0 or higher, the short IDs for notables that were created prior to the upgrade are not displayed on the Incident Review page. As a workaround, you can recreate all the short IDs that were available prior to the upgrade.

After upgrading to version 6.6.0

When you upgrade the Splunk Enterprise Security app to versions 6.6.0 or higher, you will no longer have support for Glass Tables. Do not upgrade to ES 6.6.0 or higher if you need to continue using Glass Tables. For more information on this deprecated feature, see Deprecated features.

Additionally, if "Incident Review - Table Attributes" is changed before upgrading to ES 6.6.0, the Incident Review page does not display fields such as Disposition. As a workaround, check if the "table_attributes" in the /etc/apps/SA-ThreatIntelligence/local/log_review.conf configuration file exists. If the "table_attributes" in the /etc/apps/SA-ThreatIntelligence/local/log_review.conf configuration file exists, remove the "table_attributes" and revert to the default configuration values for ES 6.6.0. Manually add the following default fields to the "table attributes". For more information, see Change Incident Review Columns.

You can customize the configuration file later, if required.

table_attributes = [\
{"field": "rule_title", "label": "Title"}
,\
{"field": "risk_object", "label": "Risk Object"}
,\
{"field": "risk_score", "label": "Aggregated Risk Score"}
,\
{"field": "risk_event_count", "label": "Risk Events"}
,\
{"field": "notable_type", "label": "Type"}
,\
{"field": "_time", "label": "Time"}
,\
{"field": "disposition_label", "label": "Disposition"}
,\
{"field": "security_domain", "label": "Security Domain"}
\
]

Also, If the "event_attributes" in the /etc/apps/SA-ThreatIntelligence/local/log_review.conf configuration file exists, remove the "event_attributes" and revert to the default configuration values for ES 6.6.0. Manually add the following default fields to the "event_attributes":

table_attributes = [\
{"field": "risk_event_count",  "label": "Risk Events"}
\
]

After upgrading to version 6.4.1

When you upgrade the Splunk Enterprise Security app to versions 6.0 or higher, you might see the issues mentioned in After upgrading to version 6.0. For complete details, see Manage asset and identity upon upgrade.

Upgrading to Enterprise Security 6.4.1 or higher does not allow you to remove existing fields using the Risk Factors Editor but only allows you to add fields to the risk data model. This prevents the mismatch between the default and local configuration of the risk data model. For more information on upgrade issues with the risk data model, see After upgrading from version 6.2.0 or lower to a version 6.3.0 or higher

After upgrading to version 6.4

When you upgrade the Splunk Enterprise Security app to versions 6.0 or higher, you might see the issues mentioned in After upgrading to version 6.0. For complete details, see Manage asset and identity upon upgrade.

Upgrading the Splunk Enterprise Security app to versions 6.4.0 or higher may cause the following issues:

• Threat intelligence manager is no longer available from the Splunk Enterprise menu bar at Configure > Settings > Data inputs > Threat Intelligence Manager.
• Threat intelligence uploads are no longer available from the Enterprise Security menu bar at Configure > Data Enrichment > Threat Intelligence Uploads.
• Health check warnings appear.

See Manage UI issues impacting threat intelligence after upgrading Splunk Enterprise Security. in the Administer Splunk Enterprise Security manual.

When you upgrade to Enterprise Security 6.4.0, notable actions for some correlation searches are disabled. If you want these correlation searches to generate notables, you must re-enable the notable actions for the correlation searches, see Enable notables for correlation searches in the Administer Splunk Enterprise Security manual.

When you upgrade to Enterprise Security 6.4.0 or higher, ensure that the email conventions that the identity lookups use to create a common unique key between different identity sources are configured as required. Note that only the Email convention is enabled by default. If you use the Email Short convention, make sure that the checkbox for Email Short is selected, otherwise you may lose data related to the Email Short convention during an upgrade. For more information on adding an identity input stanza for the lookup source, see Add an identity input stanza for the lookup source.

After upgrading to version 6.3

When you upgrade the Splunk Enterprise Security app to versions 6.0 or higher, you might see the issues mentioned in After upgrading to version 6.0. For complete details, see Manage asset and identity upon upgrade in the Administer Splunk Enterprise Security manual.

MLTK app version 5.2.0 is included in the ES installer. The previously generated models from MLTK 5.0 are compatible as-is. The previously generated models MLTK 4.x are not compatible and have to be regenerated. See Machine Learning Toolkit Overview in Splunk Enterprise Security for general information about models in MLTK 5.2.0.

After upgrading to version 6.2

When you upgrade the Splunk Enterprise Security app to versions 6.0 or higher, you might see the issues mentioned in After upgrading to version 6.0. For complete details, see Manage asset and identity upon upgrade.

After upgrading to Enterprise Security 6.2.0, you need to enable Enforce props in the Global Settings of Asset and Identity Management if you want the identity manager to automatically enforce configuration file settings. On a fresh installation, Enterprise Security 6.2.0 has Enforce props set to enabled by default and the setting is enforced continuously. However, prior versions only enforce once and then switch the setting to false right away. If you're already using a previous version of ES with assets and identities, the /local/inputs.conf file already has enforce_props=false and it needs to be set back to true after you upgrade, if you want to ensure that settings are managed for you. The majority of users who configure settings through the Splunk Web UI will benefit from enabling the setting. See Revise the enforcements used by the identity manager framework.

After upgrading to version 6.1.0

When you upgrade the Splunk Enterprise Security app to versions 6.0 or higher, you might see the issues mentioned in After upgrading to version 6.0. For complete details, see Manage asset and identity upon upgrade.

MLTK app version 5 is now included in the ES installer. The previously generated models from MLTK 4.x are not compatible and have to be regenerated. See Machine Learning Toolkit Overview in Splunk Enterprise Security for general information about models in MLTK 5. See Update Splunk MLTK models for Python 3 in the Splunk Enterprise Python 3 Migration manual for information about rebuilding models.

After upgrading to version 6.0

When you upgrade the Splunk Enterprise Security app to versions 6.0 or higher, you might see the following issues in Assets and Identities:

• The Asset and Identity Management navigation bar and page does not display if you have customized the menu bar in Splunk Enterprise Security. See Restore the default navigation or Recover the new view of Assets and Identities Navigation page.
• The Asset and Identity page merges the data for your assets and identities after the upgrade. For more information on how to avoid merged rows to display, see Avoid merged assets and identities data.
• The asset and identity collections, search previews, and search results may display differently from before the upgrade. To restore the previous view that you had prior to the upgrade, see Recover the new view of Assets and Identities Navigation page.
• The Asset and Identity page does not enable access to some of your previously saved macros. You may no longer be able to access saved macros if they were not documented for public use.

For complete details, see Manage asset and identity upon upgrade.

After upgrading to version 5.3.x

If you followed the default behavior and installed the technology add-ons that shipped with Enterprise Security in previous versions, after upgrading to version 5.3.x you will notice messages that state, "TA-<name> version <number> is lower than required." This is expected behavior with the recent installer enhancements. After upgrading, do one of the following:

• Update the technology add-ons manually. See Deploy add-ons included with Splunk Enterprise Security.
• Disable the modular input that manages these messages by going to Settings > Data Inputs > Configuration Checker, and disabling the confcheck_es_app_version input.

After upgrading to version 5.1.x

Manually delete the file $SPLUNK_HOME/etc/apps/splunk_instrumentation/default/data/ui/views/search.xml. After deleting the file, do one of the following:

• Refresh Splunk Web: http(s)://yoursplunkurl.com:8000/en-US/debug/refresh?entity=data/ui/views
• Refresh splunkd: http(s)://yoursplunkurl.com:8089/services/data/ui/views/_reload

After upgrading to version 5.0.x

• Select the Edit Lookups permission checkbox again. Because the Edit Lookups permission now includes an additional capability, the permission is not checked by default. Roles still have the edit_lookups capability. See Configure users and roles in the Installation and Upgrade Manual
• Enable the Access - Geographically Improbable Access - Summary Gen search to see data on the Geographically Improbable Access panel of the Access Anomalies dashboard or notable events produced by the Geographically Improbable Access Detected correlation search.

After upgrading from a version prior to 4.1.x

• Correlation search editor configurations might be inconsistent with pre-upgrade settings if the search migration process is still running. Search the internal index to look for successfully migrated searches and review the status of the migration operation.
  

  index=_internal sourcetype=configuration_check file="confcheck_es_modactions*" migrated

• Enabled correlation searches that are not configured to create notable events revert to creating notable events.
  For example, a correlation search that by default created a notable event and a risk modifier that you configured to create only a risk modifier will, after upgrade, create both a risk modifier and a notable event.

1. Before upgrading, note enabled correlation searches that do not create notable events using the following search.
   

   | rest splunk_server=local count=0 /services/saved/searches search="name=\"*-Rule\"" | where disabled=0 AND 'action.summary_index'=0 | table 'eai:acl.app',title

2. After the upgrade is complete, update the affected correlation searches so that the searches no longer create notable events.

After upgrading from version 6.2.0 or lower to a version 6.3.0 or higher

Modifying risk factors using the Risk Factor Editor automatically creates corresponding field entries in the risk datamodel Risk.json. However, upgrading from enterprise Security version 6.2.0 or lower to a version 6.3.0 or higher may cause a mismatch if there is a discrepancy in the field entries of the local and the default configuration files of the risk data model and result in an error. Upgrading to Enterprise Security 6.4.1 or higher does not allow you to remove existing fields but only allows you to add fields to the risk data model. This prevents the mismatch between the local and the default configuration of the risk data model.

Test upgrade and setup of Splunk Enterprise Security

You can test the upgrade and setup of Splunk Enterprise Security before you perform the full upgrade. You must complete Step 1. Review the planning topic, Step 2. Download Splunk Enterprise Security, and the first two sub-steps of Step 3. Install the latest Splunk Enterprise Security before you can follow these steps.

1. From Splunk Web, open the Search and Reporting app.
2. Type the following search to perform a dry run of the upgrade and setup.

   |essinstall --dry-run