Troubleshoot ITSI permissions, teams, backups, and restores

Here are some common issues related to ITSI permissions and capabilities, backups, and restores and how to resolve them.

User assigned a custom role can't view objects

A user is assigned a custom role can't view objects in ITSI

Resolution

Make sure you've fully completed steps 1-4 in Create a custom role in ITSI.

User has itoa_admin role but can't view objects

A user is assigned the itoa_admin role but is unable to read services or any other objects on their corresponding lister pages.

Resolution

By default, the itoa_admin role ships with the itoa_analyst and itoa_user roles. The itoa_user ships with read capabilities for ITOA objects like services, entities, glass tables, and deep dives. Make sure these capabilities haven't changed.

Unable to create an external ticket

A user is assigned the itoa_analyst role with the create_external_ticket capability. However, they're unable to create an external ticket.

Resolution

A restriction in Splunk Enterprise means the user needs the itoa_admin role, which inherits from the admin role.

"Access denied. You do not have permission to create this object."

You see access denied errors when attempting to create objects.

Cause

ITSI relies on the fact that your admin role inherit from the roles defined in $SPLUNK_HOME/etc/apps/itsi/default/authorize.conf:

[role_admin]
importRoles = itoa_admin;itoa_analyst;itoa_user;power;user

Resolution

Use btool to check system/local/authorize.conf:

 $SPLUNK_HOME/bin/splunk btool authorize list role_admin --debug

You might have redefined the admin role inheritance in system/local/authorize.conf, or in other apps. If this is the case, add the inheritances added from the UI or through the configuration file.

Default scheduled backup not running

After a fresh install or migration, the default scheduled backup isn't running at 1:00 am.

Cause

The backup runs at 1:00 am in the timezone of the server. If your local timezone is different than the server's, it might appear to run at a different time.

Alternatively, the modular input for the default scheduled backup runs at every restart, and every hour after that. It's possible to see a maximum of one-hour delays. For example, if the next scheduled time is 1:00am, the modular input runs at 12:45am and 1:45am, the backup will start at 1:45am.

Failed to fetch backup information preview

ITSI fails to fetch backup information preview with ID: <backup_id>

Resolution

Check https://localhost:8089/servicesNS/nobody/SA-ITOA/backup_restore_interface/backup_restore/preview/<backup_id> and see if the information exists for the given backup ID.

Failed to upload a backup file

ITSI fails to upload the selected backup file.

Resolution

• Check the network tab of the browser to see if there's a failed request. Check if you can create a restore job by clicking Create.
• Make sure the file is valid and not corrupted.
• Get a new backup file from the backup job. Download this file and try to upload it for restore.

Global team is gone after upgrade

The global team is no longer present after an ITSI upgrade.

Resolution

All services in ITSI must be assigned to a team. If migration fails with the error Failed to import Team settings, you can manually run the Python script called itsi_reset_default_team.py. The script manually creates the Global team in the KV store which completes the migration.

To run the script, perform the following steps:

1. Run the following commands on any search head in your ITSI deployment:

   cd $SPLUNK_HOME/etc/apps/SA-ITOA/bin
   $SPLUNK_HOME/bin/splunk cmd python itsi_reset_default_team.py
   

2. Provide the splunkd port number and your Splunk username and password when prompted.
   After the script finishes successfully, the Global team is created in the KV store.
3. Restart your Splunk software.

Check the ITSI logs

IT Service Intelligence log files have a prefix of itsi_.

• IT Service Intelligence search command logs are located in $SPLUNK_HOME/var/run/splunk/dispatch/<session_id>/itsi_search.log.
• All other ITSI logs are located in $SPLUNK_HOME/var/log/splunk.

All ITSI logs have a source type of itsi_internal_log to make them easy to search.

Steps

1. Run the following Splunk search to search ITSI logs:

   index = _internal sourcetype=itsi_internal_log

2. Click the source field under Selected Fields to see specific log files.

For Windows deployments, the ITSI search command log, itsi_search.log, cannot be searched in Splunk Web. You must open the file on the Windows host using a text editor.