Before you upgrade IT Service Intelligence

Perform the steps in this topic before you upgrade IT Service Intelligence (ITSI) to the latest release. Splunk Cloud customers must work with Splunk Support to coordinate upgrades to ITSI. This version of ITSI supports upgrading from version 4.1.x or later. To upgrade from earlier versions, perform intermediary upgrades.

Copy any changes to itsi_rules_engine.properties

As of version 4.4.0, you can make changes to a local copy of the itsi_rules_engine.properties file at $SPLUNK_HOME/etc/apps/SA-ITOA/local/ and these changes will take precedence over the default file. Previously, this file was not treated like a regular Splunk configuration file, so changes to a local copy of the file had no impact.

If you've made changes to the default file in the past, make a copy of these changes before upgrading to version 4.4.0 or higher from a pre-4.4.0 version. After you upgrade, create a blank itsi_rules_engine.properties file at $SPLUNK_HOME/etc/apps/SA-ITOA/local/ and add these changed settings to the local file. This step ensures that your changes to the file will persist through future upgrades.

Make all future changes to itsi_rules_engine.properties in the local file rather than the default file. For the contents of the file, see Rules Engine properties reference in ITSI in the Event Analytics Manual.

For more information about how Splunk handles changes to configuration files, see For more information, see Configuration file precedence in the Splunk Enterprise Admin Manual.

Trim down episode KV store collections

Upgrades to version 4.4.x might be slower than usual for the following reasons:

• The mod_time field is added to all existing objects in the itsi_notable_group_user KV store collection.
• Episode comments are migrated from the itsi_notable_event_comment collection to the itsi_grouped_alerts index.

To prevent slow upgrades, trim down the Event Analytics KV store collections to less than 50,000 objects before upgrading to version 4.4.x. After trimming these collections, the deleted episodes will no longer appear in Episode Review.

Performing these steps on versions prior to 4.4.x will delete the objects from the KV store. These deleted entries are not archived.

Prerequisites

You must have the itoa_admin role to delete objects from these KV store collections. For more information, see KV store collection permissions in ITSI.

Steps

1. Check the number of objects in the itsi_notable_group_system KV store collection. If there are more than 50,000 objects, trim the collection to less than 50,000.
   1. Open or create a local copy of itsi_notable_event_retention.conf at $SPLUNK_HOME/etc/apps/SA-ITOA/local/.
   2. Add the following stanza:

      [itsi_notable_group_system]
      # 30 days
      retentionTimeInSec = 2592000
      

   3. Reduce the retentionTimeInSec setting based on approximately how long it takes for your system to generate 50,000 episodes. For example, if it takes 15 days to generate 50,000 episodes, the retention time in seconds would be 1296000.
2. Set the data type of the mod_time field as time for the following KV store collections.
   1. Open or create a local copy of collections.conf at $SPLUNK_HOME/etc/apps/SA-ITOA/local/.
   2. Add the following stanzas:

      [itsi_notable_group_user]
      field.mod_time = time
      
      [itsi_notable_group_system]
      field.mod_time = time
      

3. Include additional fields for the itsi_notable_group_user command to support.
   1. Open or create a local copy of transforms.conf at $SPLUNK_HOME/etc/apps/SA-ITOA/local/.
   2. Add the following stanza:

      [itsi_notable_group_user_lookup]
      fields_list = _key, status, severity, owner, event_identifier_hash, object_type, mod_time
      

4. Either wait an hour for the modular input to run, or restart your Splunk software to run it immediately.
5. Run the following SPL search to remove objects from itsi_notable_group_user that don't exist in itsi_notable_group_system:

   | inputlookup itsi_notable_group_system_lookup | fields _key | rename _key as id | lookup itsi_notable_group_user_lookup _key as id OUTPUT owner severity status event_identifier_hash object_type mod_time | rename id as _key | outputlookup itsi_notable_group_user_lookup



6. Check the number of comments in the itsi_notable_event_comment collection. If it's more than 1 million, trim down the collection. The following search trims the comments to the last 90 days:

   | inputlookup itsi_notable_event_comment_lookup | where mod_time > now() - 3*30*24*3600 | eval object_type="notable_event_comment" | outputlookup itsi_notable_event_comment_lookup

Check entity configurations

The strict entity association change introduced in version 4.2.0 and the removal of entity alias filtering can affect certain entities in your environment. Before upgrading, unzip and run the following script on any search head: Check_kpi_entity_configs.zip.

The script outputs a list of entities that might break as a result of the strict entity association change described in Removed features in IT Service Intelligence.

Make sure no service templates are syncing

If any service templates are syncing when you upgrade ITSI, the upgrade fails. Check the sync status of service templates by clicking Configure > Service Templates from the ITSI main menu.

Back up the search head

Take a full backup of the search head. For instructions, see Back up and restore ITSI KV store data. To back out of the upgrade, you must restore the prior version of Splunk IT Service Intelligence from a backup.

Check admin role inheritance

Make sure the Splunk admin role inherits from the itoa_admin role. The default settings for admin role inheritance for ITSI are contained in authorize.conf. Problems can occur when these settings have been modified in a local version of the file.

Check KV store size limits

The limit of a single batch save to a KV store collection is 500 MB. Check the total amount of data that your services contain, and, if necessary, increase the KV store size limit in $SPLUNK_HOME/etc/apps/SA-ITOA/local/limits.conf. This setting controls the maximum size, in megabytes (MB), of the results that are returned for a single query to a collection.

Prerequisites

• Only users with file system access, such as system administrators, can increase the KV store size limit.
• Review the steps in How to edit a configuration file in the Splunk Enterprise Admin Manual.

Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location.

Steps

1. Open or create a local limits.conf file in $SPLUNK_HOME/etc/apps/SA-ITOA/local/.
2. Increase the max_size_per_result_mb value in the [kvstore] stanza:

   [kvstore]
   max_size_per_result_mb = [new value]
   

Review known issues and changes

Review the following topics before you upgrade ITSI:

1. Compatible versions of the Splunk platform. See Splunk Enterprise system requirements.
2. Hardware requirements. See Planning your hardware requirements.
3. Known issues with the latest release of IT Service Intelligence. See Known issues in Splunk IT Service Intelligence in the Release Notes.
4. Removed features in the latest release of IT Service Intelligence. See Removed features in the Release Notes.

Recommendations for upgrading IT Service Intelligence

Upgrade both the Splunk platform and IT Service Intelligence in the same maintenance window. See the Splunk Enterprise system requirements to verify which versions of Splunk ITSI and Splunk Enterprise are supported with each other.

If you're upgrading to the Python 3 release of Splunk Enterprise (version 8.0.x), you must upgrade ITSI and all other apps before upgrading Splunk Enterprise. For more information, see Python 3 migration with ITSI.

1. Upgrade Splunk Enterprise to a compatible version. See How to upgrade a distributed Splunk Enterprise environment in the Splunk Enterprise Installation Manual.
2. Upgrade Splunk platform instances.
3. Upgrade Splunk IT Service Intelligence.
4. Review, upgrade, and deploy add-ons.
5. See Version-specific upgrade notes for post-installation tasks.

Upgrading ITSI deployed on a search head cluster is a multi-step process. The procedure is detailed in Upgrade IT Service Intelligence in a search head cluster environment in this manual.