Splunk® IT Service Intelligence

Administer Splunk IT Service Intelligence

Download manual as PDF

This documentation does not apply to the most recent version of ITSI. Click here for the latest version.
Download topic as PDF

Back up and restore ITSI KV store data

Back up the KV store and restore it from backup. You can perform both full backups and partial backups. Taking regular backups from a healthy environment enables you to restore from a backup in the event of a disaster, or if you add a search head to a cluster. Make sure to be familiar with the standard backup and restore tools and procedures used by your organization.

When you run a backup job, ITSI saves your data to a set of JSON files compressed into a single .zip file located in $SPLUNK_HOME/var/itsi/backups/<_key.zip> on the search head. ITSI detects and preserves the application version from which it creates a backup. When you restore from a backup, ITSI detects the correct version of the backup and performs the required migration.

Splunk Cloud customers must use the Backup/Restore page in the ITSI user interface. Others can perform backups and restores from the command line using the kvstore_to_json.py script. See Backup and restore operations (mode 1) for information.

The following table describes the functionality available in each backup and restore method.

Method Backup/Restore UI Command line script Comments
Full backup X X
Partial backup X X Dependent objects are not backed up when performing a partial backup using the command line script.
Partial restore X
Merge changes during restore
X X Merges objects in the backup with existing KV store objects.
Clean restore
X Replaces existing KV store objects with objects in the backup.

Difference between an ITSI backup and a Splunk Enterprise backup

Splunk Enterprise offers an option to back up and restore the KV store. For information, see Back up and restore KV store in the Splunk Enterprise Admin Manual. However, an ITSI backup is structured in a specific format in order to process the content in the backup files. The Splunk Enterprise backup does not have the same format as an ITSI backup, so you cannot use it to back up your ITSI data.

All backup content is processed within ITSI. Many other activities are triggered by ITSI, such as saved search generation and object dependency updates. Therefore, directly restoring Splunk Enterprise KV store data will not restore the ITSI system completely. It is best practice to use the processes described below to back up your ITSI data.

What gets backed up

The following table describes the types of data included and not included in an ITSI backup.

Data Example Included in backup Not included in backup
KV store objects Services, service templates, entities, KPIs, KPI base searches, teams, glass tables, service analyzers, deep dives X
Indexed data ITSI summary index, notable events X

To back up indexed data, use the same approach you use to back up other Splunk indexes. For information, see Back up indexed data in the Splunk Enterprise Managing Indexers and Clusters of Indexers manual.

Default scheduled backup

The default scheduled backup is a full backup that runs daily at 1:00 AM (01:00) in the server's local time zone. The time of the backup job is displayed on the Backup/Restore page in the Splunk user's local time zone. Click Edit in the Actions column to change the frequency and time of the scheduled backup job or disable it from the Backup/Restore Jobs page. You cannot delete the scheduled backup job.

There is only one scheduled backup job. You cannot create additional scheduled backup jobs. Only the most recent backup is retained from the scheduled backup.

Create a full backup

Create a full backup to make a copy of all your ITSI configuration information.

Before creating a backup, make sure no service templates are syncing. Check the sync status of service templates by clicking Configure > Service Templates from the ITSI main menu.

Prerequisites

You must have the itoa_admin role, or be granted the write_itsi_backup_restore capability, to create a backup/restore job.

Steps

  1. Click Configure > Backup/Restore.
  2. Click Create Job > Create Backup Job.
  3. Select Full Backup.
  4. Provide a name and description of the backup job.
  5. (Optional) Enable Include .conf files to back up the following configuration (.conf) files located in $SPLUNK_HOME/etc/apps/SA-ITOA/local and $SPLUNK_HOME/etc/apps/itsi/local.

    ITSI only backs up these .conf files if they exist in a non-default directory such as $SPLUNK_HOME/etc/apps/itsi/local. For more information, see About configuration manuals. Upon restore, the backed up .conf file overrides the existing local version.

  6. Click Create.

The backup job appears on the Backup/Restore Jobs lister page with the status "Queued" until the job runs. When the backup job finishes, the status changes to "Completed" and a confirmation message appears in the Messages drop-down in Splunk Web.

You can run any completed backup job again by clicking Edit > Start Backup in the Actions column. You can also modify the completed backup job before running it again.

Create a partial backup

Create a partial backup if you want to back up a subset of your KV store objects. You can back up services, service templates, teams, glass tables, and configuration (.conf) files. When selecting one of these object types, dependent objects are automatically selected to preserve the functionality of the objects after they are restored. In some cases, you can choose whether or not to include dependent objects in the backup.

Before creating a backup, make sure no service templates are syncing. Check the sync status of service templates by clicking Configure > Service Templates from the ITSI main menu.

Prerequisites

You must have the itoa_admin role, or be granted the write_itsi_backup_restore capability, to create a backup/restore job.

Steps

  1. Click Configure > Backup/Restore.
  2. Click Create Job > Create Backup Job.
  3. Select Partial Backup.
  4. Provide a name and description of the backup.
  5. (Optional) Toggle Include .conf files to back up the following configuration (.conf) files located in $SPLUNK_HOME/etc/apps/SA-ITOA/local and $SPLUNK_HOME/etc/apps/itsi/local.

    ITSI only backs up these .conf files if they exist in a non-default directory such as $SPLUNK_HOME/etc/apps/itsi/local. For more information, see About configuration manuals. Upon restore, the backed up .conf file overrides the existing local version.

  6. Click Next.
  7. On the partial backup page, select the objects to include in the backup. Selections made in one tab can cause selections to be made in another tab if there are dependencies between the objects.
  8. (Optional) Click Change Settings to change the objects that are selected when you select a service. By default, dependent services are selected. The KPI base searches, threshold templates, and team associated with a service are always included in the backup.

    Although entities are not listed in the partial backup page, you can include them in the backup file by selecting Entities in the Settings dialog.

  9. After making your selections, review every tab to confirm the objects that are selected.
  10. Click Save and Backup.

The backup job appears in the Backup/Restore jobs lister page with the status "Queued" until the job runs. When the backup job finishes, the status changes to "Completed" and a confirmation message appears in the Messages drop-down in Splunk web.

You can edit any partial backup job before it starts. If the backup job started, you will see a read-only view that lists the objects contained in the partial backup.

You can run any completed backup job again by clicking Edit > Start Backup in the Actions column. You can also modify the completed backup job before running it again.

Back up a service

When you select a service to back up, ITSI also backs up the following objects:

    • KPI base searches
    • Threshold templates
    • Teams

When you select a service to back up, you can choose whether or not to back up the following objects:

    • Dependent services
    • Entities that match service entity rules
    • Linked service template

The dependency between the service and these objects breaks if you do not choose to back up the associated object:

    • If you back up a service that has dependent services without also backing up the dependent services, the service will no longer be dependent on the services after restoring.
    • If you back up a service without backing up the entities that match the service entity rules, the entities will no longer be associated with the service if the entities do not exist in the restored environment.
    • If you back up a service that is linked to a service template without also backing up the service template, the service will no longer be linked to the service template after restoring.

Back up a service template

When you back up a service template, all the services linked to the service template are also added to the backup. If you deselect a service, it will not exist in the restored environment.

Back up teams

When you select to back up a team, all the services associated with that team are added to the backup. You can deselect any services you do not want to back up.

Back up a glass table

When you select to back up a glass table, all of the services associated with that glass table are also added to the backup. If you choose not to back up a service that the glass table depends on, any widgets that use KPIs from the service will no longer function if the service does not exist in the restored environment.

Glass table images and ACLs are always included in the backup when you back up a glass table.

Restore a full or partial backup

When you restore from a backup listed in the Backup/Restore Jobs page, ITSI merges the JSON data contained in the backup ZIP file with your existing KV store data.

  • New objects added since the backup are added.
  • Existing objects that match an object in the backup are replaced.
  • All other existing objects are preserved.

If you want to delete all existing KV store objects in an ITSI instance and replace them with the objects in the backup for a clean restore, use the command line script.

A restore from the Backup/Restore UI restores all of the data in the backup file. If you want to selectively restore files in a backup, use the command line script.

If you restart the Splunk platform while a backup or restore job is in progress, the job resumes after Splunk restarts. Queued jobs automatically time out if they are not completed within 12 hours for any reason. You can change the default timeout duration by updating the value of job_queue_timeout in the [backup_restore] stanza in itsi_settings_conf.

How restore handles teams

If you are restoring from a previous version of ITSI to version 3.0 or later, all services and service-related objects such as entities, KPI templates, KPI base searches, and KPI threshold templates are placed in the Global team. Backups and subsequent restores on ITSI version 3.0 or later retain team information for services and service-related objects. See Overview of service-level permissions in ITSI for information about teams.

When restoring a backup taken on an ITSI 3.0 or later system to another ITSI 3.0 or later system, team ACLs are retained when the teams are restored. Therefore, the roles assigned to the teams must exist on the system that the backup is restored to. For example, a restore creates teams called "HR" and "Finance", which have read/write access for the hr_admin and finance_admin roles, respectively. If the current system does not have these roles, these teams are only accessible to the itoa_admin role. If the roles assigned to the teams don't already exist on the system, you can create them either before or after restoring.

Prerequisites

    • Make sure no service templates are syncing. Check the sync status of service templates by clicking Configure > Service Templates from the ITSI main menu.
    • Make sure all technology add-ons (TAs), supporting add-ons (SAs), and domain add-ons (DAs) that exist on the old system are installed on the new system.
    • If you've made modifications to any add-ons on the old system, manually copy those add-ons over the new system before restoring.

Restore from a backup

You can restore from a backup that you created.

  1. Click Configure > Backup/Restore and find the backup from which you want to restore.
  2. Click Edit > Restore Backup.
  3. Click Start Restore.
    "Restore from" prepends the backup name in the jobs list. A message stating that the restore job completed successfully appears as a message in Splunk Web.

If you restore from a backup that contains .conf files, you must restart the Splunk platform.

Restore from a backup ZIP file

You can download any backup ZIP file that is created when you run a backup job in the UI and then restore from that backup ZIP file using the UI. The maximum file size supported for uploading is 500 MB.

To download a backup ZIP file:

  1. Click Configure > Backup/Restore and find the backup file that you want to download.
  2. Click Edit > Download Backup. The backup ZIP file downloads to your local machine.

To restore from a downloaded backup ZIP file:

  1. Click Create Job > Create Restore Job.
  2. Provide a name and optional description of the backup.
  3. Click Choose File and select the previously downloaded backup ZIP file from which you want to restore.
  4. (Optional) Toggle Include .conf files to back up any configuration files included in the backup.

    If you restore from a backup that contains .conf files, you must restart the Splunk platform.

  5. Click Create.
    ITSI uploads the backup ZIP file and the new restore job appears in the Backup/Restore Jobs list. A message stating that the restore job has completed successfully appears in the Message drop-down in Splunk web.

Restore from a backup created using the command line

If you created a backup of ITSI using the kvstore_to_json.py command line option and you want to restore that data using the Backup/Restore Jobs page, the backup JSON files must be contained in a folder named backup and compressed into a ZIP file. For information, see kvstore_to_json.py operations in ITSI.

Backup and restore in a search head cluster environment

You can run backup and restore jobs from the Backup/Restore page in search head cluster environments. You can create a backup on any cluster member and then later restore data from that backup on any cluster member, regardless of where the backup was initiated.

For example, if your search head cluster has three cluster members, sh-01, sh-02, and sh-03, and you create a backup on sh-01, you can later restore from that backup on sh-01, sh-02, or sh-03.

When you create a backup on any search head cluster member the configuration data from all cluster members is backed up. Likewise, when you restore from a backup on any cluster member, configuration data is restored across all cluster members.

In a search head cluster environment, the scheduled backup runs only on the captain. However, you can perform a restore of a scheduled backup from any cluster member. If you choose to download the scheduled backup, make sure to download it from the captain because the captain contains the latest backup.

PREVIOUS
Schedule maintenance downtime in ITSI
  NEXT
kvstore_to_json.py operations in ITSI

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.2.0, 4.2.1, 4.2.2, 4.2.3


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters