Upgrade IT Service Intelligence in a search head cluster environment

After you deploy a new ITSI version to cluster members or in a distributed environment, you must also upgrade your indexers and ITSI license components.

Before upgrading

• Before you upgrade ITSI, you must perform all prerequisite steps specified in Before you upgrade IT Service Intelligence.
• Ensure stable and optimal connectivity between the deployer and search head cluster members, such as minimal network latency, no dropped packets, and so on. Problems with network connectivity during an shcluster bundle push of a new version of ITSI may leave ITSI in an inconsistent state and require further steps to resolve.
• If upgrading to a Python 3 release of Splunk (version 8.x), you must upgrade IT Service Intelligence and all other apps before upgrading Splunk Enterprise. For more information, see Python 3 migration with ITSI.
• ITSI supports upgrades from up to three versions prior.

Step 1: Confirm the cluster is in a healthy state

Confirm that the cluster is in a healthy state before you begin the upgrade:

splunk show shcluster-status

Check the following criteria:

1. Locate the current search head captain and use it as the target when running the splunk apply shcluster-bundle.
2. Make sure the search head cluster is fully functional and that there are no pending replication updates.

For information on health check criteria, see Health check output details.

Step 2. Deploy the new version to the cluster members

Use the deployer to distribute the new version of ITSI to search head cluster members, the same way you initially deployed ITSI on the search head cluster. A migration script runs on the captain after upgrading. The upgrade then propagates to all other cluster members.

1. Log in to splunk.com with your Splunk.com ID.
2. Download the latest Splunk IT Service Intelligence product.
3. If you're upgrading from a pre-4.6.0 ITSI version to version 4.6.0 or higher, you need to stop the Rules Engine before upgrading so it can pick up the fields added to the KV store for migration:
   1. Within Splunk Web, go to Settings > Searches, reports, and alerts.
   2. In the App dropdown, select All.
   3. Use the filter to locate the itsi_event_grouping search.
   4. Click Actions > Disable.
4. Untar the ITSI installation package into $SPLUNK_HOME/etc/shcluster/apps/. For example:

   tar -xvf splunk-it-service-intelligence_<latest_version>.spl -C $SPLUNK_HOME/etc/shcluster/apps/
   

   On Windows, rename the file extension from .spl to .tgz first and use a third-party utility like 7-Zip to perform the extraction.

5. Run the splunk apply shcluster-bundle command on the deployer:

   splunk apply shcluster-bundle -target <URI>:<management_port> -auth <username>:<password>
   

   At the end of the bundle push, a rolling restart occurs if necessary. During a rolling restart, approximately 10% of the members restart at a time, until all have restarted. Note that if you're using options to disable rolling restarts, you need to trigger one manually. For instructions and more information, see Use the deployer to distribute apps and configuration updates.
6. A migration screen steps you through the upgrade process. For Skip over localized failures, choose whether to skip over the following types of failures:
   • Missing dependencies in service KPIs, such as a missing macro
   • Multiple entity split or filter fields in KPI base searches
   • Missing dependencies in KPI base searches
   • Missing dependencies in correlation searches
   • Duplicate services

   Skipping over these failures means the problematic objects aren't migrated. You'll receive a list of skipped objects when the upgrade completes.

7. Click Start Upgrade. The migration script runs to migrate existing ITSI knowledge objects to the new version.
8. When the upgrade completes, open the ITSI homepage.
9. Re-enable the itsi_event_grouping search.

To check migration related logs, run the following Splunk search:

index=_internal sourcetype=itsi_internal_log "[itsi.migration]"

Step 3. Upgrade indexers

For non-clustered distributed environments, copy the SA-IndexCreation file to $SPLUNK_HOME/etc/apps on each indexer in your deployment.

If you have an indexer cluster, use the configuration bundle method to replicate SA-IndexCreation across all peer nodes. On the master node, place a copy of SA-IndexCreation in $SPLUNK_HOME/etc/master-apps/. For information about updating peers in an indexer cluster, and for CLI instructions, see Manage app deployment across all peers in the Splunk Enterprise Managing Indexers and Clusters of Indexers manual.

Step 4: (Optional) Enable the Metrics Backfill Process Queue modular input

A new metrics-based summary index was introduced in ITSI version 4.6.0. To provide a more continuous experience, a modular input for backfill functionality was added to migrate data from the itsi_summary index to the new metrics-based index. For more information about the metrics index, see ITSI metrics summary index reference in the Administration Manual.

In version 4.6.1, the modular input for backfill functionality is disabled by default as opposed to running automatically. If you upgraded to version 4.6.1 or higher and you need to use the Service Analyzer to inspect service or KPI data from before the upgrade, enable the backfill modular input. If you choose not to enable the backfill modular input, sparklines on the Service Analyzer might appear flat at first due to lack of data.

To enable to backfill modular input, modify the inputs.conf file in etc/shcluster/apps/SA-ITOA/local.

1. Change disabled setting to disabled=0.
2. Optionally, you can modify the default configurations to backfill more or less data. If you do modify the defaults, first determine if your environment can backfill data at a higher rate than set by the default throttle and concurrent search settings.
1. (Optional) Customize the metrics_backfill_length setting. This is the length of time in days that the modular input (using search) backfills at one time. You can change this to a lower number if needed.
2. (Optional) Customize the metrics_backfill_throttle setting. This is the wait time in seconds between the searches used to backfill the metrics summary index. You can increase or decrease this as necessary for your environment.

[itsi_summary_metrics_backfill://metrics_backfiller]
disabled = 0
interval = 5
log_level = INFO
metrics_backfill_concurrent_searches = 1
metrics_backfill_length = 3
metrics_backfill_throttle = 10
python.version = python3

3. Use the deployer to distribute these changes to search head cluster members.

Step 5. Upgrade ITSI license components

When you upgrade Splunk IT Service Intelligence, you must also upgrade SA-ITSI-Licensechecker and SA-UserAccess on any license master in a distributed or search head cluster environment.

If one of the search heads in your environment is also a license master, the license master components are upgraded when you upgrade ITSI on the search heads.

Step 6: Validate the upgrade

The Splunk IT Service Intelligence upgrade process is now complete. Objects disabled during the upgrade process are automatically reenabled. The ITSI shows the following message: IT Service Intelligence upgrade has completed successfully.

1. In Splunk Web, click Help > About to verify that the upgrade was successful.
2. Clear the browser cache of the browser you use to access Splunk Web. If you do not clear the browser cache, some pages might fail to load.

You can also check the installed version, latest version, and previous version by running the following search:

| rest splunk_server=local /services/apps/local/itsi | stats values(version) as itsi_installed_version | join [|inputlookup itsi_migration_check]

After upgrading

Perform the following steps after upgrading IT Service Intelligence.

1. If there's a problem with the new version, see Troubleshoot an upgrade of IT Service Intelligence.
2. If the upgrade fails, see Roll back an upgrade of ITSI.
3. See the Version-specific upgrade notes for ITSI for the version you upgraded to.