Splunk® IT Service Intelligence

Install and Upgrade Manual

Acrobat logo Download manual as PDF

This documentation does not apply to the most recent version of ITSI. Click here for the latest version.
Acrobat logo Download topic as PDF

Before you upgrade IT Service Intelligence

Perform the steps in this topic before you upgrade IT Service Intelligence (ITSI) to version 4.6.x. Splunk Cloud customers must work with Splunk Support to coordinate upgrades to ITSI. Version 4.6.x supports upgrading from version 4.3.x or later. To upgrade from earlier versions, perform intermediary upgrades.

Remove episode lookup entries from transforms.conf

Version 4.6.0 updated the notable event system KV store collection in transforms.conf with the following fields:

  • parent_group_id
  • split_by_hash
  • first_event_id
  • group_template_id

Before upgrading to version 4.6.0 or later from a pre-4.6.0 version, you need to remove any entries for itsi_notable_group_system_lookup in your local transforms.conf so these fields can be populated in the system KV store collection.

In addition, the instructions field was added to the itsi_notable_group_user KV store collection in version 4.5.0. Before upgrading to version 4.5.0 or later from a pre-4.5.0 version, remove any entries for itsi_notable_group_user_lookup in your local transforms.conf so this value can be populated in the user KV store collection.

Copy any changes to itsi_rules_engine.properties

As of version 4.4.0, you can make changes to a local copy of the itsi_rules_engine.properties file at $SPLUNK_HOME/etc/apps/SA-ITOA/local/ and these changes will take precedence over the default file. Previously, this file was not treated like a regular Splunk configuration file, so changes to a local copy of the file had no impact.

If you've made changes to the default file in the past, make a copy of these changes before upgrading to version 4.4.0 or higher from a pre-4.4.0 version. After you upgrade, create a blank itsi_rules_engine.properties file at $SPLUNK_HOME/etc/apps/SA-ITOA/local/ and add these changed settings to the local file. This step ensures that your changes to the file will persist through future upgrades.

Make all future changes to itsi_rules_engine.properties in the local file rather than the default file. For the contents of the file, see Rules Engine properties reference in ITSI in the Event Analytics Manual.

For more information about how Splunk handles changes to configuration files, see For more information, see Configuration file precedence in the Splunk Enterprise Admin Manual.

Trim down episode KV store collections

Upgrades to version 4.4.2 and higher might be slower than usual for the following reasons:

  • The mod_time field is added to all existing objects in the itsi_notable_group_user KV store collection.
  • Episode comments are migrated from the itsi_notable_event_comment collection to the itsi_grouped_alerts index.

To prevent slow upgrades, trim down the Event Analytics KV store collections to less than 500,000 objects before upgrading to version 4.4.2. After trimming these collections, the deleted episodes will no longer appear in Episode Review.

Performing these steps on versions prior to 4.4.x will delete the objects from the KV store. These deleted entries are not archived.

Prerequisites

You must have the itoa_admin role to delete objects from these KV store collections. For more information, see KV store collection permissions in ITSI.

Steps

  1. Check the number of objects in the itsi_notable_group_system KV store collection. If there are more than 500,000 objects, trim the collection to less than 500,000.
    1. Open or create a local copy of itsi_notable_event_retention.conf at $SPLUNK_HOME/etc/apps/SA-ITOA/local/.
    2. Add the following stanza:
      [itsi_notable_group_system]
      # 30 days
      retentionTimeInSec = 2592000
      
    3. Reduce the retentionTimeInSec setting based on approximately how long it takes for your system to generate 500,000 episodes. For example, if it takes 15 days to generate 500,000 episodes, the retention time in seconds would be 1296000.
  2. Set the data type of the mod_time field as time for the following KV store collections.
    1. Open or create a local copy of collections.conf at $SPLUNK_HOME/etc/apps/SA-ITOA/local/.
    2. Add the following stanzas:
      [itsi_notable_group_user]
      field.mod_time = time
      accelerated_fields.mod_time = {"mod_time": 1}
      
      [itsi_notable_group_system]
      field.mod_time = time
      accelerated_fields.mod_time = {"mod_time": 1} 
      
  3. Either wait an hour for the modular input to run, or restart your Splunk software to run it immediately.
  4. Include additional fields for the itsi_notable_group_user command to support.
    1. Open or create a local copy of transforms.conf at $SPLUNK_HOME/etc/apps/SA-ITOA/local/.
    2. Add the following stanza:
      [itsi_notable_group_user_lookup]
      fields_list = _key, status, severity, owner, event_identifier_hash, object_type, mod_time
      
  5. After trimming the itsi_notable_group_system collection, run the following SPL search to remove objects from itsi_notable_group_user that don't exist in itsi_notable_group_system:

    | inputlookup itsi_notable_group_system_lookup | fields _key | rename _key as id | lookup itsi_notable_group_user_lookup _key as id OUTPUT owner severity status event_identifier_hash object_type mod_time | rename id as _key | outputlookup itsi_notable_group_user_lookup



  6. Check the number of comments in the itsi_notable_event_comment collection. If it's more than 1 million, trim down the collection. The following search trims the comments to the last 90 days:

    | inputlookup itsi_notable_event_comment_lookup | where mod_time > now() - 3*30*24*3600 | eval object_type="notable_event_comment" | outputlookup itsi_notable_event_comment_lookup

  7. After you trim all the collections, revert the changes to collections.conf and transforms.conf.

Make sure no service templates are syncing

If any service templates are syncing when you upgrade ITSI, the upgrade fails. Check the sync status of service templates by clicking Configuration > Service Templates from the ITSI main menu.

Check admin role inheritance

Make sure the Splunk admin role inherits from the itoa_admin role. The default settings for admin role inheritance for ITSI are contained in authorize.conf. Problems can occur when these settings have been modified in a local version of the file.

Check KV store size limits

The limit of a single batch save to a KV store collection is 500 MB. Check the total amount of data that your services contain, and, if necessary, increase the KV store size limit in $SPLUNK_HOME/etc/apps/SA-ITOA/local/limits.conf. This setting controls the maximum size, in megabytes (MB), of the results that are returned for a single query to a collection.

Prerequisites

  • Only users with file system access, such as system administrators, can increase the KV store size limit.
  • Review the steps in How to edit a configuration file in the Splunk Enterprise Admin Manual.

Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location.

Steps

  1. Open or create a local limits.conf file in $SPLUNK_HOME/etc/apps/SA-ITOA/local/.
  2. Increase the max_size_per_result_mb value in the [kvstore] stanza:
    [kvstore]
    max_size_per_result_mb = [new value]
    

Review known issues and changes

Review the following topics before you upgrade ITSI:

  1. Compatible versions of the Splunk platform. See Splunk Enterprise system requirements.
  2. Hardware requirements. See Planning your hardware requirements.
  3. Known issues with the latest release of IT Service Intelligence. See Known issues in Splunk IT Service Intelligence in the Release Notes.
  4. Removed features in the latest release of IT Service Intelligence. See Removed features in the Release Notes.

Recommendations for upgrading IT Service Intelligence

Upgrade both the Splunk platform and IT Service Intelligence in the same maintenance window. See the Splunk Enterprise system requirements to verify which versions of Splunk ITSI and Splunk Enterprise are supported with each other.

If you're upgrading to a Python 3 release of Splunk Enterprise (version 8.x), you must upgrade ITSI and all other apps before upgrading Splunk Enterprise. For more information, see Python 3 migration with ITSI.

  1. Upgrade Splunk Enterprise to a compatible version.
  2. Upgrade Splunk platform instances.
  3. Upgrade Splunk IT Service Intelligence.
  4. Review, upgrade, and deploy add-ons.
  5. See Version-specific upgrade notes for post-installation tasks.

Upgrading ITSI deployed on a search head cluster is a multi-step process. The procedure is detailed in Upgrade IT Service Intelligence in a search head cluster environment in this manual.

Last modified on 08 November, 2020
PREVIOUS
Uninstall Splunk IT Service Intelligence
  NEXT
Upgrade IT Service Intelligence on a single instance

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.6.0 Cloud only, 4.6.1 Cloud only, 4.6.2 Cloud only


Was this documentation topic helpful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters