Upgrade a single instance of IT Service Intelligence
This topic describes how to upgrade Splunk IT Service Intelligence (ITSI) on an on-premises search head from version 3.1.x or later to the latest release. Splunk Cloud customers work with Splunk Support to coordinate upgrades to IT Service Intelligence.
- Unzip and run the following script on the search head: Check_kpi_entity_configs.zip. The script outputs a list of entities that might break as a result of the strict entity association change described in Removed features in IT Service Intelligence.
- Make sure no service templates are syncing. Check the sync status of service templates by clicking Configure > Service Templates from the ITSI main menu.
- Perform a full backup of the search head. For information, see Back up and restore ITSI KV store data. To back out of the upgrade, you must restore the prior version of Splunk IT Service Intelligence from a backup.
- If you've made any changes to the
itsi_rules_engine.propertiesfile, make a copy of it before upgrading. Upgrading overwrites any changes you made. After the upgrade is complete, you must re-implement any changes you made to the file.
- Make sure the Splunk admin role inherits from the itoa_admin role. The default settings for admin role inheritance for ITSI are contained in authorize.conf. Problems can occur when these settings have been modified in a local version of the file.
- Review the compatible versions of the Splunk platform. See Splunk Enterprise system requirement.
- Review the hardware requirements to make sure that your server hardware supports ITSI. See Planning your hardware requirements.
- Review known issues with the latest release of IT Service Intelligence. See Known issues in Splunk IT Service Intelligence in the Release Notes.
- Review removed features in the latest release of IT Service Intelligence. See Removed features in the Release Notes.
- Check KV store size limits. The limit of a single batch save to a KV store collection is 50 MB. Check the total amount of data that your services contain, and, if necessary, increase the KV store size limit in limits.conf.
Step 1. Install the latest Splunk IT Service Intelligence
In a single-instance deployment, a single Splunk Enterprise instance serves as both search head and indexer.
You must upgrade ITSI by extracting the ITSI installation package. ITSI does not support installation using the app manager in Splunk Web or using the
splunk install app command at the command line.
- Log in to splunk.com with your Splunk.com ID.
- Download the latest Splunk IT Service Intelligence product.
- Stop Splunk. For example:
cd $SPLUNK_HOME/bin ./splunk stop
- Extract the ITSI installation package into
$SPLUNK_HOME/etc/apps. For example:
tar -xvf splunk-it-service-intelligence_<latest_version>.spl -C $SPLUNK_HOME/etc/apps
On Windows, rename the file extension from .spl to .tgz first and use a third-party utility like 7-Zip to perform the extraction.
- Start Splunk. For example:
cd $SPLUNK_HOME/bin ./splunk start
The first time you start Splunk Enterprise after installing the new files, a migration script runs to migrate existing ITSI knowledge objects to the new version. To check migration related logs, run the following Splunk search:
index="_internal" source= "*itsi_upgrade.log*"
Step 2. Upgrade indexers
SA-IndexCreation is required on all indexers. For non-clustered distributed environments, copy
$SPLUNK_HOME/etc/apps/ on individual indexers. Indexers must be running Splunk Enterprise 7.1.x -7.2.x. Your search heads must run the same or later version of Splunk Enterprise. If you upgrade your indexers, verify whether you must also upgrade your search heads. For information, see Splunk Enterprise version compatibility.
If you have an indexer cluster, use the configuration bundle method to replicate
SA-IndexCreation across all peer nodes. On the master node, place a copy of
$SPLUNK_HOME/etc/master-apps/. For information about updating peers in an indexer cluster, see Manage app deployment across all peers in the Managing Indexers and Clusters of Indexers manual.
Step 3. Validate the upgrade
The Splunk IT Service Intelligence upgrade process is now complete. Objects disabled during the upgrade process will automatically be enabled.
- In Splunk Web, click Help > About to verify that upgrade was successful.
- Clear the browser cache of the browser you use to access Splunk Web to make sure that you access a fresh version of Splunk Web after upgrading. If you do not clear the browser cache, some pages might fail to load.
Why are things missing after I upgrade?
When objects are exported during a backup or migration, if the number of KPIs linked to a service is high, the instance can hit a KV store memory size limit. This can cause some objects to be dropped from the backup and lost upon upgrade. To fix this issue, increase the KV store bulk get limit in limits.conf.
- Only users with file system access, such as system administrators, can increase the KV store bulk get limit.
- Review the steps in How to edit a configuration file in the Admin Manual.
Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location.
- Open or create a local
- Under the
[kvstore]stanza, increase the
max_size_per_result_mbvalue as necessary.
- Retry the backup or upgrade.
Why is the global team gone after I upgrade?
The Teams feature introduced in version 3.0 requires all services be assigned to a team. When upgrading to ITSI 3.0 or later from a version of ITSI prior to 3.0, the migration script creates the default team, Global, in the KV store and assigns all existing services to the Global team.
If this step in the migration fails, you will receive the error "Failed to import Team settings" and ITSI will be unusable. If you encounter this error when upgrading to version 3.0, download and install ITSI version 3.0.1. ITSI 3.0.1 contains a Python script called
itsi_reset_default_team.py in the
apps/SA-ITOA/bin directory. Running this script manually creates the Global team in the KV store which completes the migration.
If you encounter this error when upgrading from a version prior to 3.0, manually run the Python script.
- Run the following command on any search head in your ITSI deployment:
$SPLUNK_HOME/bin/splunk cmd python itsi_reset_default_team.py
- Provide the splunkd port number and Splunk username and password when prompted.
- Restart Splunk software.
- Click Configure > Teams in ITSI and confirm that the Global team exists.
It is recommended to install version 3.0.1 rather than downloading and running the
itsi_reset_default_team.py file from the ITSI 3.0.1 installation package. Version 3.0.1 has better error logging than version 3.0 so in the event that the script does not work, version 3.0.1 will provide more information on the reason for the failure.
Version-specific upgrade notes
Consider the following guidelines when upgrading to specific versions of IT Service Intelligence.
After upgrading to version 3.1.x
- If you have a dedicated license master, remove
SA-ITOAfrom the license master since ITSI no longer requires it as of version 3.1.x.
- When the objects in ITSI are exported during a backup or migration, if the number of KPIs linked to a service is high, the instance can hit a KV store memory size limit causing some objects to be dropped from the backup and lost after the upgrade.
Workaround: Increase the KV store bulk get limit in
$SPLUNK_HOME/etc/apps/SA-ITOA/local/limits.confand retry the backup or upgrade. Increase the
max_size_per_result_mbvalue as necessary.
[kvstore] # The maximum size, in megabytes (MB), of the result that will be returned for a single query to a collection. # ITSI requires approximately 50MB per 1,000 KPIs. Override this value if necessary. # Default: 500 MB max_size_per_result_mb = 500
This action will increase the memory used by the KV store during operations.
After upgrading to version 4.0.x
- Remove unnecessary XML files from the ITSI OS Module that have been removed or renamed as of ITSI 4.0.0. Remove the following files from
- Version 4.0.x ships with an internal license stack called IT Service Intelligence Internals *DO NOT COPY* stack to ensure that you don't pay for notable events generated by ITSI. The source types used to track notable events and episodes are counted on this special stack with no impact on your Splunk Enterprise license. When calculating your daily license usage, disregard this stack.
After upgrading to version 4.0.4
To initiate the fix for ITSI-1868 concerning entity rules, you need to trigger the service-entity rule change handler. To do this, run the kvstore_to_json mode 4 option which will regenerate your KPI search schedules.
After upgrading to version 4.2.x
Entity Alias Filtering field used in KPI searches was removed in version 4.2.0. With the removal of entity alias filtering, ITSI now strictly matches entities against KPI search results using both the alias key and value, whereas before it only used the alias value. This strict association change can cause some entities to not be included in KPI results. If this is the case, a message appears in Splunk Web with a link to documentation on how to fix potentially broken entities. For information, see Removed features in Splunk IT Service Intelligence.
Planning an upgrade of IT Service Intelligence
Upgrade IT Service Intelligence in a search head cluster environment
This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.2.0, 4.2.1, 4.2.2, 4.2.3