Splunk® IT Service Intelligence

Administration Manual

Acrobat logo Download manual as PDF


Splunk IT Service Intelligence version 4.0.x reached its End of Life on January 19, 2021. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see Plan an upgrade of IT Service Intelligence.
This documentation does not apply to the most recent version of ITSI. Click here for the latest version.
Acrobat logo Download topic as PDF

Backup and restore ITSI data

You can back up and restore ITSI configuration data using either the Backup/Restore UI in Splunk Web, or by manually running the kvstore_to_json.py script from the command line. Splunk Cloud customers must use the Backup/Restore function in the ITSI UI.

The ITSI backup backs up ITSI configuration data stored in the KV store which includes services, entities, KPIs, KPI base searches, teams, glass tables, service analyzer views, and deep dives. Note that the backup does not include summary index data, notable events, or other data stored in Splunk indexes. The backup also does not include settings which exist as .conf files. Back up the indexed data using the same approach you use to back up other Splunk indexes. For information, see Back up indexed data. To back up configuration files, make an archive or copy of $SPLUNK_HOME/etc/.

ITSI version 3.0 and later comes with a default scheduled backup job that runs daily. You can change the frequency and time of the scheduled backup job or disable it from the Backup/Restore Jobs page.

In version 2.4.0 and later, ITSI auto-detects and preserves the app version from which it creates a backup. When you restore from a backup, using either the Backup/Restore Jobs UI or the kvstore_to_json.py command line option, ITSI auto-detects the correct version of the backup and performs the required migration.

You cannot use the UI to restore from backups created prior to ITSI version 2.4.0. If you try to use the UI to restore from a backup prior to 2.4.0, an error message appears instructing you to use the kvstore_to_json.py script to run the restore job. This is because the UI cannot detect the version of a backup prior to 2.4.0, but the kvstore_to_json.py script lets you specify earlier versions of a backup using the -b BR_VERSION parameter.

If you are restoring from a previous version of ITSI to version 3.0 or later, all services and service-related objects (entities, KPI templates, KPI base searches, and KPI threshold templates) are placed in the Global team. Backups and subsequent restores on ITSI version 3.0 or later will retain team information for services and service-related objects. See ITSI service-level permissions for information on teams.

Before restoring a backup, make sure no service templates are syncing. Check the sync status of service templates in the Service Template viewer by selecting Configure > Service Templates from the top menu.

Backup and restore from the UI

ITSI lets you back up your current ITSI configuration and restore from a previous backup directly inside the UI. Only full backups and restores can be performed from the UI. To do a partial backup or restore, use the kvstore_to_json.py script. See kvstore_to_json.py operations for information.

When you run a backup job from the UI, ITSI saves your data to a set of JSON files compressed into a single .zip file located in $SPLUNK_HOME/var/itsi/backups/<_key.zip> on the search head. When performing a restore from the UI, the contents of the backup file are merged with the existing configuration information in ITSI.

Change the scheduled backup

ITSI comes with a default scheduled backup job that runs daily at 1:00 a.m. (01:00) in the server's local time zone. This time is displayed on the Backup/Restore page in the Splunk user's time zone.

You can change the default scheduled backup job to run at a different time or frequency (such as weekly). Additional scheduled backup jobs cannot be created; there is only one scheduled backup job. You can disable the scheduled backup job, but you cannot delete it. Only the most recent backup is kept from the scheduled backup.

Modify the default scheduled backup job to meet your needs.

  1. Click Configure > Backup/Restore.
  2. Click Default Scheduled Backup. The Update Job dialog opens.
  3. Rename the scheduled backup job if desired.
  4. Change the description for the scheduled backup job if desired.
  5. Disable the scheduled backup job if you do not want it to run.
  6. Change the schedule (daily or weekly) and time to run the backup job.
  7. Click Update Job.
    The edited scheduled backup job appears at the top of the Backup/Restore jobs list with the status "Scheduled Weekly" or "Scheduled Daily".

Create a one-time full backup

  1. Click Configure > Backup/Restore.
  2. Click Create new Job > Create Backup Job.
  3. Type in a name for the backup job.
  4. Type in a description for the backup job (optional).
  5. Click Create.
    The backup job appears in the Backup/Restore jobs list with the status "Queued" until the job runs. When the backup job runs, the job status changes to "In Progress." When the backup job finishes, the status changes to "Completed" and a confirmation appears in the Messages drop-down in Splunk web.

Restore a full backup

Restoring a backup merges the JSON data contained in the backup .zip file with your existing KV store data. New elements such as services or KPIs added since the backup are added, existing elements that match an element in the backup are replaced, and all other existing elements are preserved. There is no limitation on the size of the backup file from which you can restore.

Prerequisites

    • Copy over all local ITSI .conf files to the new system. Configuration files are not included in ITSI backups.
    • Make sure all technology add-ons (TAs), supporting add-ons (SAs), and domain add-ons (DAs) that exist on the old system are installed on the new system.
    • If you've made modifications to any add-ons on the old system, manually copy those add-ons over the new system before restoring.

Steps

  1. In the Backup/Restore Jobs list, find the backup from which you want to restore. Click Edit > Restore Backup.
  2. Click Start Restore.
    "Restore from" prepends the backup name in the jobs list. A message stating that the restore job has completed successfully appears in the Message drop-down in Splunk web.

If you restore from a backup that was generated while the Default notable event aggregation policy was running in Smart Mode, you must restart Splunk software.

When restoring a backup taken on an ITSI 3.x or later system to another ITSI 3.x or later system, team ACLs are retained when the teams are restored. Therefore, the roles assigned to the teams must exist on the system the backup is restored to.

For example, a restore creates teams called HR and Finance which have read/write access for the roles hr_admin and finance_admin, respectively. If the current system does not have the hr_admin and finance_admin roles, these teams are only accessible to the itoa_admin role. If the roles assigned to the teams don't already exist on the system, you can create them before or after restoring.

Download and restore from a backup zip file

You can download any backup .zip file that is created when you run a backup job in the UI, then restore from that backup .zip file using the UI. When you restore from a backup .zip file that you have previously downloaded, the maximum file size supported for uploading is 500 MB.

To download a backup .zip file:

  1. In the Backup/Restore jobs list, find the backup file that you want to download.
  2. Click Edit > Download Backup.
    The backup .zip file downloads to your local machine.

To restore from a previously downloaded backup .zip file:

  1. Click Create Job > Create Restore Job.
  2. Type in a name for the restore job.
  3. Type in a description for the restore job (optional).
  4. Click Choose File and select the previously downloaded backup .zip file from which you want to restore.
  5. Click Create.
    ITSI uploads the backup zip file and the new restore job appears in the Backup/Restore jobs list. A message stating that the restore job has completed successfully appears in the Message drop-down in Splunk web.

Backup and restore in a search head cluster environment

ITSI lets you run backup/restore jobs from the UI in search head cluster environments. You can create a backup on any cluster member, then later restore data from that backup on any cluster member, regardless of where the backup was initiated.

For example, if your search head cluster has three cluster members, sh-01, sh-02, and sh-03, and you create a backup on sh-01, you can later restore from that backup on sh-01, sh-02, or sh-03.

When you create a backup on any search head cluster member, configuration data from all cluster members is backed up. Likewise, when you restore from a backup on any cluster member, configuration data is restored across all cluster members.

In a search head cluster environment, the scheduled backup runs only on the captain. However, you can perform a restore of a scheduled backup from any cluster member. If you choose to download the scheduled backup, make sure to download it from the captain because the captain contains the latest backup.

Note: If you restart your Splunk software while a backup or restore job is in progress, the job will resume after the Splunk platform has restarted. Queued jobs will automatically timeout if not completed within 12 hours for any reason. You can change the default timeout duration by updating the value of job_queue_timeout in the [backup_restore] stanza in SA-ITOA/local/itsi_settings_conf.

Restore in the UI from a backup created using the CLI

If you created a backup of ITSI version 2.3.0 or later using the kvstore_to_json.py command line option, and you want to restore that data using the Backup/Restore Jobs UI, the backup JSON files must be contained in a folder named backup and compressed into a .zip file.

Backup and restore from the command line

The kvstore_to_json.py script lets you perform backup and restore operations on ITSI data from the command line. For detailed usage options, see kvstore_to_json.py operations in this manual. Note that you cannot use this script to change the default scheduled backup.

Last modified on 12 March, 2020
PREVIOUS
Schedule maintenance downtime in ITSI
  NEXT
kvstore_to_json.py operations in ITSI

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.0.0, 4.0.1, 4.0.2, 4.0.3, 4.0.4


Was this documentation topic helpful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters