Upgrade IT Service Intelligence on a single instance
This topic describes how to upgrade Splunk IT Service Intelligence (ITSI) on an on-premises search head from version 4.0.x or later to the latest release. Splunk Cloud customers work with Splunk Support to coordinate upgrades to IT Service Intelligence.
- Unzip and run the following script on any search head: Check_kpi_entity_configs.zip. The script outputs a list of entities that might break as a result of the strict entity association change described in Removed features in IT Service Intelligence.
- Make sure no service templates are syncing. Check the sync status of service templates by clicking Configure > Service Templates from the ITSI main menu.
- Perform a full backup of the search head. For information, see Back up and restore ITSI KV store data. To back out of the upgrade, you must restore the prior version of Splunk IT Service Intelligence from a backup.
- If you've made any changes to the
itsi_rules_engine.propertiesfile, make a copy of it before upgrading. Upgrading overwrites any changes you made. After the upgrade is complete, you must re-implement any changes you made to the file.
- Make sure the Splunk admin role inherits from the itoa_admin role. The default settings for admin role inheritance for ITSI are contained in authorize.conf. Problems can occur when these settings have been modified in a local version of the file.
- Review the compatible versions of the Splunk platform. See Splunk Enterprise system requirement.
- Review the hardware requirements to make sure that your server hardware supports ITSI. See Planning your hardware requirements.
- Review known issues with the latest release of IT Service Intelligence. See Known issues in Splunk IT Service Intelligence in the Release Notes.
- Review removed features in the latest release of IT Service Intelligence. See Removed features in the Release Notes.
- Check KV store size limits. The limit of a single batch save to a KV store collection is 50 MB. Check the total amount of data that your services contain, and, if necessary, increase the KV store size limit in limits.conf.
Step 1. Install the latest version of ITSI
On a single-instance deployment, a single Splunk Enterprise instance serves as both the search head and indexer.
You must upgrade ITSI by extracting the ITSI installation package. ITSI does not support installation using the app manager in Splunk Web or using the
splunk install app command at the command line.
- Log in to splunk.com with your Splunk.com ID.
- Download the latest Splunk IT Service Intelligence product.
- Stop your Splunk platform instance:
cd $SPLUNK_HOME/bin ./splunk stop
- Extract the ITSI installation package into
$SPLUNK_HOME/etc/apps. For example:
tar -xvf splunk-it-service-intelligence_<latest_version>.spl -C $SPLUNK_HOME/etc/apps
On Windows, rename the file extension from .spl to .tgz first and use a third-party utility like 7-Zip to perform the extraction.
- Start your Splunk software.
cd $SPLUNK_HOME/bin ./splunk start
The first time you start Splunk Enterprise after installing the new files, a migration script runs to migrate existing ITSI knowledge objects to the new version. To check migration related logs, run the following Splunk search:
index="_internal" source= "*itsi_upgrade.log*"
Step 2. Upgrade Indexers
You must place the
SA-IndexCreation add-on on all indexers. For non-clustered distributed environments, copy
$SPLUNK_HOME/etc/apps/ on individual indexers. Indexers must be running a compatible version of Splunk Enterprise. If you upgrade your indexers, verify whether you must also upgrade your search heads. For information, see Splunk Enterprise version compatibility.
If you have an indexer cluster, use the configuration bundle method to replicate
SA-IndexCreation across all peer nodes. On the master node, place a copy of
$SPLUNK_HOME/etc/master-apps/. For information about updating peers in an indexer cluster, see Manage app deployment across all peers in the Splunk Enterprise Managing Indexers and Clusters of Indexers manual.
Step 3. Validate the upgrade
The Splunk IT Service Intelligence upgrade process is now complete. Objects disabled during the upgrade process are automatically reenabled.
- In Splunk Web, click Help > About to verify that the upgrade was successful.
- Clear the browser cache of the browser you use to access Splunk Web. If you do not clear the browser cache, some pages might fail to load.
Why are things missing after I upgrade?
When objects are exported during a backup or migration, if the number of KPIs linked to a service is high, the instance can hit a KV store memory size limit. This can cause some objects to be dropped from the backup and lost upon upgrade. To fix this issue, increase the KV store bulk get limit in limits.conf.
- Only users with file system access, such as system administrators, can increase the KV store bulk get limit.
- Review the steps in How to edit a configuration file in the Admin Manual.
Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location.
- Open or create a local
- Under the
[kvstore]stanza, increase the
max_size_per_result_mbvalue as necessary.
- Retry the backup or upgrade.
Why is the Global team gone after I upgrade?
The Teams feature introduced in version 3.0 requires that all services be assigned to a team. When upgrading to ITSI 3.0 or later from a version of ITSI prior to 3.0, the migration script creates the default team, Global, in the KV store and assigns all existing services to the Global team.
If this step in the migration fails, you receive the error "Failed to import Team settings" and ITSI is unusable. If you encounter this error when upgrading to version 3.0, download and install ITSI version 3.0.1. ITSI 3.0.1 contains a Python script called
itsi_reset_default_team.py in the
apps/SA-ITOA/bin directory. Running this script manually creates the Global team in the KV store which completes the migration.
If you encounter this error when upgrading to version 3.0.1 from a version prior to 3.0, manually run the Python script included in version 3.0.1.
To run the
itsi_reset_default_team.py script in version 3.0.1:
- Run the following command on any search head in your ITSI deployment:
$SPLUNK_HOME/bin/splunk cmd python itsi_reset_default_team.py
- Provide the splunkd port number and your Splunk username and password when prompted.
- After the script has successfully finished, the Global group is created in the KV store.
- Restart your Splunk software.
Install version 3.0.1 rather than downloading and running the
itsi_reset_default_team.py file from the ITSI 3.0.1 installation package. ITSI version 3.0.1 has better error logging than version 3.0. In the event that the script does not work, version 3.0.1 provides more information on the reason for the failure.
Version-specific upgrade notes
Consider the following guidelines when upgrading to specific versions of IT Service Intelligence.
After upgrading to version 3.1.x
- If you have a dedicated license master, remove
SA-ITOAfrom the license master since ITSI no longer requires the add-on as of version 3.1.x.
- When the objects in ITSI are exported during a backup or migration, if the number of KPIs linked to a service is high, the instance can hit a KV store memory size limit causing some objects to be dropped from the backup and lost after the upgrade.
Workaround: Increase the KV store bulk get limit in
$SPLUNK_HOME/etc/apps/SA-ITOA/local/limits.confand retry the backup or upgrade. Increase the
max_size_per_result_mbvalue as necessary.
[kvstore] # The maximum size, in megabytes (MB), of the result that will be returned for a single query to a collection. # ITSI requires approximately 50MB per 1,000 KPIs. Override this value if necessary. # Default: 500 MB max_size_per_result_mb = 500
This action increases the memory used by the KV store during operations.
After upgrading to version 4.0.x
- Remove unnecessary XML files from the ITSI OS Module that were removed or renamed as of ITSI 4.0.0. Remove the following files from
IT Service Intelligence Internals *DO NOT COPY* stackto ensure that you don't pay for notable events generated by ITSI. The sourcetypes used to track notable events and episodes are counted on this special stack with no impact on your Splunk Enterprise license. When calculating your daily license usage, disregard this stack. For more information, see ITSI license requirements.
After upgrading to version 4.0.4
To initiate the fix for ITSI-1868 concerning entity rules, you need to trigger the service-entity rule change handler. To trigger the handler, run the kvstore_to_json mode 4 option, which will regenerate your KPI search schedules.
After upgrading to version 4.2.x
Entity Alias Filtering field used in KPI searches was removed in version 4.2.0. With the removal of entity alias filtering, ITSI now strictly matches entities against KPI search results using both the alias key and value, whereas before it only used the alias value. This strict association change can cause some entities to not be included in KPI results. If this is the case, a message appears in Splunk Web with a link to documentation on how to fix potentially broken entities. For information, see Removed features in Splunk IT Service Intelligence.
Plan an upgrade of IT Service Intelligence
Upgrade IT Service Intelligence in a search head cluster environment
This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.3.0, 4.3.1