Troubleshoot ITSI backups and restores

Here are some common issues related to ITSI permissions and capabilities, backups, and restores, together with recommendations for how to resolve those issues.

User assigned a custom role can't view objects

A user assigned a custom role can't view objects in ITSI

Resolution

Make sure you've fully completed steps 1-4 in Create a custom role in ITSI.

User has itoa_admin role but can't view objects

A user is assigned the itoa_admin role but is unable to read services or any other objects on their corresponding lister pages.

Resolution

By default, the itoa_admin role ships with the itoa_analyst and itoa_user roles. The itoa_user role ships with read capabilities for ITOA objects like services, entities, glass tables, and deep dives. Make sure these capabilities haven't changed.

Unable to create an external ticket

A user is assigned the itoa_analyst role with the create_external_ticket capability. However, that user is unable to create an external ticket.

Resolution

A restriction in Splunk Enterprise means the user needs the itoa_admin role, which inherits from the admin role.

"Access denied. You do not have permission to create this object."

You see access denied errors when attempting to create objects.

Cause

ITSI relies on the fact that your admin role inherits from the roles defined in $SPLUNK_HOME/etc/apps/itsi/default/authorize.conf:

[role_admin]
importRoles = itoa_admin;itoa_analyst;itoa_user;power;user

Resolution

Use btool to check system/local/authorize.conf:

 $SPLUNK_HOME/bin/splunk btool authorize list role_admin --debug

You might have redefined the admin role inheritance in system/local/authorize.conf, or in other apps. If this is the case, add the inheritances added from the UI or through the configuration file.

Default scheduled backup not running

After a fresh install or migration, the default scheduled backup isn't running at 1:00 am.

Resolution

The backup runs at 1:00 am in the timezone of the server. If your local timezone is different than the server's, it might appear to run at a different time.

Alternatively, the modular input for the default scheduled backup runs at every restart, and every hour after that. It's possible to see a maximum of one-hour delays. For example, if the next scheduled time is 1:00am, the modular input runs at 12:45am and 1:45am, the backup will start at 1:45am.

Failed to fetch backup information preview

ITSI fails to fetch backup information preview with ID: <backup_id>

Resolution

Check https://localhost:8089/servicesNS/nobody/SA-ITOA/backup_restore_interface/backup_restore/preview/<backup_id> to see if the information exists for the given backup ID.

Failed to upload a backup file

ITSI fails to upload the selected backup file.

Resolution

• Check the network tab of the browser to see if there's a failed request. Check if you can create a restore job by clicking Create.
• Make sure the file is valid and not corrupted.
• Get a new backup file from the backup job. Download this file and try to upload it for restore.

Missing macro makes restore fail

Backup restore attempt fails because one or more of the ITSI objects in the environment was created using a macro that was subsequently deleted, and restore cannot reconcile that Splunk object missing from the environment with the artifact that it helped build in ITSI. To ensure consistency, restore operations attempt to validate all ITSI objects, whether those objects are in the environment or in the backup.

Resolution

Avoid deleting macros and saved searches that were used to build ITSI objects. Before deleting Splunk objects from your environment, ensure that they are not used in any ITSI objects, because missing objects impact ITSI performance negatively.

Global team is gone after upgrade

The global team is no longer present after an ITSI upgrade.

Resolution

All services in ITSI must be assigned to a team. If migration fails with the error Failed to import Team settings, you can manually run the Python script called itsi_reset_default_team.py. The script manually creates the Global team in the KV store which completes the migration.

To run the script, perform the following steps:

1. Run the following commands on any search head in your ITSI deployment:

   cd $SPLUNK_HOME/etc/apps/SA-ITOA/bin
   $SPLUNK_HOME/bin/splunk cmd python itsi_reset_default_team.py
   

2. Provide the splunkd port number and your Splunk username and password when prompted.
   After the script finishes successfully, the Global team is created in the KV store.
3. Restart your Splunk software.

How to check the ITSI logs

IT Service Intelligence log files have a prefix of itsi_.

• IT Service Intelligence search command logs are located in $SPLUNK_HOME/var/run/splunk/dispatch/<session_id>/itsi_search.log.
• All other ITSI logs are located in $SPLUNK_HOME/var/log/splunk.

All ITSI logs have a source type of itsi_internal_log to make them easy to search.

Steps

1. Run the following Splunk search to search ITSI logs:

   index = _internal sourcetype=itsi_internal_log

2. Click the source field under Selected Fields to see specific log files.

For Windows deployments, the ITSI search command log, itsi_search.log cannot be searched in Splunk Web. You must open the file on the Windows host using a text editor.