Splunk® IT Service Intelligence

Install and Upgrade Manual

Acrobat logo Download manual as PDF

Splunk IT Service Intelligence version 4.4.x will no longer be supported as of October 22, 2021. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see Before you upgrade IT Service Intelligence.
This documentation does not apply to the most recent version of ITSI. Click here for the latest version.
Acrobat logo Download topic as PDF

Troubleshoot an upgrade of IT Service Intelligence

Use this information to troubleshoot post-upgrade issues.

Teams validation checks, UI loading, and team creation script fail

The ITSI teams validation checks, UI loading, and the team creation script fail when your Splunk Enterprise instance has a role issue. Roles issues often happen on deployments where a role is missing. For example role_A inherits from role_B, but at some point the app where role_B is defined was removed.

First, run the following search to determine whether you're experience this issue:

index=_internal source=splunkd.log ( ERROR "Error retrieving info for role" ) OR ( WARN "Unknown role" )

If there's a role issue, the following errors appear every minute for each broken role:

11-22-2019 09:22:13.260 -0800 ERROR AdminHandler:AuthenticationHandler - Error retrieving info for role: role_B

If this is the case, identify all the roles that are trying to link to the missing roles with the following btool command:

./splunk btool authorize list | grep role_B

For more information, see Use btool to troubleshoot configurations in the Splunk Enterprise Troubleshooting Manual.

To fix the issue, perform one of the following steps:

  • Create a local version of authorize.conf at $SPLUNK_HOME/etc/apps/SA-ITOA/local/ and modify the import list.
  • Use the UI to edit the role.
  • Recreate the missing role.

Why are things missing after I upgrade?

If some objects, such as service analyzers, glass tables, or deep dives, are missing from the UI or unaccessible after you upgrade, the ACL objects corresponding to the objects might be missing or corrupted.

  1. See if the object exists in the KV store. Even if it does exist, there could be duplicates, which you'll address in the next step. Check the list of knowledge objects by name at the following endpoints:
      • curl -k -u admin:password https://<host>:<admin_port>/servicesNS/nobody/SA-ITOA/itoa_interface/deep_dive
      • curl -k -u admin:password https://<host>:<admin_port>/servicesNS/nobody/SA-ITOA/itoa_interface/glass_table
      • curl -k -u admin:password https://<host>:<admin_port>/servicesNS/nobody/SA-ITOA/itoa_interface/home_view
      • curl -k -u admin:password https://<host>:<admin_port>/servicesNS/nobody/SA-ITOA/itoa_interface/event_management_state
      • curl -k -u admin:password https://<host>:<admin_port>/servicesNS/nobody/SA-ITOA/event_management_interface/notable_event_aggregation_policy
      • curl -k -u admin:password https://<host>:<admin_port>/servicesNS/nobody/SA-ITOA/event_management_interface/correlation_search

    The value of the _key attribute is called obj_id or object ID in the next steps.

  2. Check if a corresponding ACL object exists with the ID of the object you're looking for at the following endpoint:
    curl -k -u admin:password https://<host>:<admin_port>/servicesNS/nobody/SA-UserAccess/storage/collections/data/app_acl
    1. If one ACL object exists with the corresponding object ID, and the object is still missing from the UI, contact Splunk Support.
    2. If two ACL objects exists with the corresponding object ID, delete one of them by running the following command:

      curl -k -u admin:password -X DELETE https://<host>:<admin_port>/servicesNS/nobody/SA-UserAccess/storage/collections/data/app_acl/<ACL_ID>

    3. If no ACL object exists with the corresponding object ID, manually create an ACL object with the following command:
      curl -k -u admin:password https://<host>:<admin_port>/servicesNS/nobody/SA-UserAccess/storage/collections/data/app_acl -H "Content-Type: application/json" -X POST -d '{"obj_type":"<OBJ_TYPE>","acl_owner":"nobody","acl_id":"<ACL_ID>","obj_id":"<OBJ_ID>","_user":"nobody","obj_shared_by_inclusion":true,"obj_acl":{"delete":["*"],"write":["*"],"obj_owner":"nobody","read":["*"]},"_key":"<ACL_ID>","obj_storename":"<OBJ_STORENAME>","obj_app":"itsi"}'

      Replace the tokens with the following values:

      Service analyzer home_view itsi_service_analyzer ID of the missing object unique ID
      Deep dive deep_dive itsi_pages ID of the missing object unique ID
      Glass table glass_table itsi_pages ID of the missing object unique ID
      Episode review event_management_state itsi_event_management ID of the missing object unique ID
      Notable event aggregation policy notable_aggregation_policy itsi_notable_event_aggregation_policy ID of the missing object unique ID
      Correlation search correlation_search itsi_correlation_search ID of the missing object unique ID

      ACL_ID must be a unique value.

Why is the Global team gone after I upgrade?

All services in ITSI must be assigned to a team. If migration fails with the error Failed to import Team settings, you can manually run the Python script called itsi_reset_default_team.py. The script manually creates the Global team in the KV store which completes the migration.

To run the script, perform the following steps:

  1. Run the following commands on any search head in your ITSI deployment:
    cd $SPLUNK_HOME/etc/apps/SA-ITOA/bin
    $SPLUNK_HOME/bin/splunk cmd python itsi_reset_default_team.py
  2. Provide the splunkd port number and your Splunk username and password when prompted.
    After the script finishes successfully, the Global team is created in the KV store.
  3. Restart your Splunk software.
Last modified on 02 December, 2020
Version-specific upgrade notes for ITSI
Configure users and roles in ITSI

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.2.0, 4.2.1, 4.2.2, 4.2.3, 4.3.0, 4.3.1, 4.4.0, 4.4.1, 4.4.2, 4.4.3, 4.4.4, 4.4.5, 4.5.0 Cloud only, 4.5.1 Cloud only

Was this documentation topic helpful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters