Splunk® IT Service Intelligence

Install and Upgrade Splunk IT Service Intelligence

Download manual as PDF

This documentation does not apply to the most recent version of ITSI. Click here for the latest version.
Download topic as PDF

Upgrade IT Service Intelligence on a single instance

This topic describes how to upgrade Splunk IT Service Intelligence (ITSI) on an on-premises search head from version 4.0.x or later to the latest release. Splunk Cloud customers work with Splunk Support to coordinate upgrades to IT Service Intelligence.

Before upgrading

  1. Unzip and run the following script on any search head: Check_kpi_entity_configs.zip. The script outputs a list of entities that might break as a result of the strict entity association change described in Removed features in IT Service Intelligence.
  2. Make sure no service templates are syncing. Check the sync status of service templates by clicking Configure > Service Templates from the ITSI main menu.
  3. Perform a full backup of the search head. For information, see Back up and restore ITSI KV store data. To back out of the upgrade, you must restore the prior version of Splunk IT Service Intelligence from a backup.
  4. If you've made any changes to the itsi_rules_engine.properties file, make a copy of it before upgrading. Upgrading overwrites any changes you made. After the upgrade is complete, you must re-implement any changes you made to the file.
  5. Make sure the Splunk admin role inherits from the itoa_admin role. The default settings for admin role inheritance for ITSI are contained in authorize.conf. Problems can occur when these settings have been modified in a local version of the file.
  6. Review the compatible versions of the Splunk platform. See Splunk Enterprise system requirement.
  7. Review the hardware requirements to make sure that your server hardware supports ITSI. See Planning your hardware requirements.
  8. Review known issues with the latest release of IT Service Intelligence. See Known issues in Splunk IT Service Intelligence in the Release Notes.
  9. Review removed features in the latest release of IT Service Intelligence. See Removed features in the Release Notes.
  10. Check KV store size limits. The limit of a single batch save to a KV store collection is 50 MB. Check the total amount of data that your services contain, and, if necessary, increase the KV store size limit in limits.conf.

Step 1. Install the latest version of ITSI

On a single-instance deployment, a single Splunk Enterprise instance serves as both the search head and indexer.

You must upgrade ITSI by extracting the ITSI installation package. ITSI does not support installation using the app manager in Splunk Web or using the splunk install app command at the command line.

  1. Log in to splunk.com with your Splunk.com ID.
  2. Download the latest Splunk IT Service Intelligence product.
  3. Stop your Splunk platform instance:
    cd $SPLUNK_HOME/bin
    ./splunk stop
    
  4. Extract the ITSI installation package into $SPLUNK_HOME/etc/apps. For example:
    tar -xvf splunk-it-service-intelligence_<latest_version>.spl -C $SPLUNK_HOME/etc/apps
    

    On Windows, rename the file extension from .spl to .tgz first and use a third-party utility like 7-Zip to perform the extraction.

  5. Start your Splunk software.
    cd $SPLUNK_HOME/bin
    ./splunk start
    

The first time you start Splunk Enterprise after installing the new files, a migration script runs to migrate existing ITSI knowledge objects to the new version. To check migration related logs, run the following Splunk search:

index="_internal" source= "*itsi_upgrade.log*"

Step 2. Upgrade Indexers

You must place the SA-IndexCreation add-on on all indexers. For non-clustered distributed environments, copy SA-IndexCreation to $SPLUNK_HOME/etc/apps/ on individual indexers. Indexers must be running a compatible version of Splunk Enterprise. If you upgrade your indexers, verify whether you must also upgrade your search heads. For information, see Splunk Enterprise version compatibility.

If you have an indexer cluster, use the configuration bundle method to replicate SA-IndexCreation across all peer nodes. On the master node, place a copy of SA-IndexCreation in $SPLUNK_HOME/etc/master-apps/. For information about updating peers in an indexer cluster, see Manage app deployment across all peers in the Splunk Enterprise Managing Indexers and Clusters of Indexers manual.

Step 3. Validate the upgrade

The Splunk IT Service Intelligence upgrade process is now complete. Objects disabled during the upgrade process are automatically reenabled.

  1. In Splunk Web, click Help > About to verify that the upgrade was successful.
  2. Clear the browser cache of the browser you use to access Splunk Web. If you do not clear the browser cache, some pages might fail to load.

Why are things missing after I upgrade?

When objects are exported during a backup or migration, if the number of KPIs linked to a service is high, the instance can hit a KV store memory size limit. This can cause some objects to be dropped from the backup and lost upon upgrade. To fix this issue, increase the KV store bulk get limit in limits.conf.

Prerequisites

  • Only users with file system access, such as system administrators, can increase the KV store bulk get limit.
  • Review the steps in How to edit a configuration file in the Admin Manual.

Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location.

Steps

  1. Open or create a local limits.conf file at $SPLUNK_HOME/etc/apps/SA-ITOA/local/.
  2. Under the [kvstore] stanza, increase the max_size_per_result_mb value as necessary.
  3. Retry the backup or upgrade.

Why is the Global team gone after I upgrade?

The Teams feature introduced in version 3.0 requires that all services be assigned to a team. When upgrading to ITSI 3.0 or later from a version of ITSI prior to 3.0, the migration script creates the default team, Global, in the KV store and assigns all existing services to the Global team.

If this step in the migration fails, you receive the error "Failed to import Team settings" and ITSI is unusable. If you encounter this error when upgrading to version 3.0, download and install ITSI version 3.0.1. ITSI 3.0.1 contains a Python script called itsi_reset_default_team.py in the apps/SA-ITOA/bin directory. Running this script manually creates the Global team in the KV store which completes the migration.

If you encounter this error when upgrading to version 3.0.1 from a version prior to 3.0, manually run the Python script included in version 3.0.1.

To run the itsi_reset_default_team.py script in version 3.0.1:

  1. Run the following command on any search head in your ITSI deployment:
    cd $SPLUNK_HOME/etc/apps/SA-ITOA/bin
    $SPLUNK_HOME/bin/splunk cmd python itsi_reset_default_team.py
  2. Provide the splunkd port number and your Splunk username and password when prompted.
  3. After the script has successfully finished, the Global group is created in the KV store.
  4. Restart your Splunk software.

Install version 3.0.1 rather than downloading and running the itsi_reset_default_team.py file from the ITSI 3.0.1 installation package. ITSI version 3.0.1 has better error logging than version 3.0. In the event that the script does not work, version 3.0.1 provides more information on the reason for the failure.

Version-specific upgrade notes

Consider the following guidelines when upgrading to specific versions of IT Service Intelligence.

After upgrading to version 3.1.x

  1. If you have a dedicated license master, remove SA-ITOA from the license master since ITSI no longer requires the add-on as of version 3.1.x.
  2. When the objects in ITSI are exported during a backup or migration, if the number of KPIs linked to a service is high, the instance can hit a KV store memory size limit causing some objects to be dropped from the backup and lost after the upgrade.

    Workaround: Increase the KV store bulk get limit in $SPLUNK_HOME/etc/apps/SA-ITOA/local/limits.conf and retry the backup or upgrade. Increase the max_size_per_result_mb value as necessary.
    [kvstore]
    # The maximum size, in megabytes (MB), of the result that will be returned for a single query to a collection.
    # ITSI requires approximately 50MB per 1,000 KPIs. Override this value if necessary.
    # Default: 500 MB
    max_size_per_result_mb = 500
    

This action increases the memory used by the KV store during operations.

After upgrading to version 4.0.x

  1. Remove unnecessary XML files from the ITSI OS Module that were removed or renamed as of ITSI 4.0.0. Remove the following files from $SPLUNK_HOME/etc/apps/DA-ITSI-OS/default/data/ui/panels:
      • cpu_memory_usage.xml
      • memory_free_percent.xml
      • memory_disk_ops.xml
      • forecast_network.xml
      • storage_volumes_most_used.xml
      • storage_devices_iostats_chart.xml
  2. Version 4.0.x ships with an internal license stack called IT Service Intelligence Internals *DO NOT COPY* stack to ensure that you don't pay for notable events generated by ITSI. The sourcetypes used to track notable events and episodes are counted on this special stack with no impact on your Splunk Enterprise license. When calculating your daily license usage, disregard this stack. For more information, see ITSI license requirements.

After upgrading to version 4.0.4

To initiate the fix for ITSI-1868 concerning entity rules, you need to trigger the service-entity rule change handler. To trigger the handler, run the kvstore_to_json mode 4 option, which will regenerate your KPI search schedules.

After upgrading to version 4.2.x

The Entity Alias Filtering field used in KPI searches was removed in version 4.2.0. With the removal of entity alias filtering, ITSI now strictly matches entities against KPI search results using both the alias key and value, whereas before it only used the alias value. This strict association change can cause some entities to not be included in KPI results. If this is the case, a message appears in Splunk Web with a link to documentation on how to fix potentially broken entities. For information, see Removed features in Splunk IT Service Intelligence.

PREVIOUS
Plan an upgrade of IT Service Intelligence
  NEXT
Upgrade IT Service Intelligence in a search head cluster environment

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.3.0, 4.3.1


Comments

You're right, the script can run on any search head in your environment because the KV store is replicated. I updated the text above to read "...on any search head". Thanks for pointing that out.

Esnyder splunk, Splunker
October 24, 2019

with regards to the following above - what is required when ITSI is on a SHC - does this need to run on all SH's OR on just the captain OR just any SH ?
I understand this will check the kvstore which is replicated accross SH's but wanted to check ?

Before upgrading
Unzip and run the following script on the search head: Check_kpi_entity_configs.zip. The script outputs a list of entities that might break as a result of the strict entity association change described in Removed features in IT Service Intelligence.

Esky73
October 23, 2019

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters