Splunk® IT Service Intelligence

Administer Splunk IT Service Intelligence

Download manual as PDF

Download topic as PDF

Back up and restore ITSI KV store data

Regularly backing up the KV store lets you restore your data from a backup in the event of a disaster or if you add a search head to a cluster. You can perform both full backups and partial backups of your data.

When you run a backup job, ITSI saves your data to a set of JSON files compressed into a single ZIP file located in $SPLUNK_HOME/var/itsi/backups on the search head. ITSI detects and preserves the application version that it creates a backup from. When you restore from a backup, ITSI detects the correct version of the backup and performs the required migration.

Splunk Cloud customers must use the Backup/Restore Jobs page in the ITSI user interface. All other customers can back up and restore their data from the command line using the kvstore_to_json.py script. For more information, see Backup and restore operations (mode 1).

The following table describes the functionality available in each backup and restore method:

Method Backup/Restore UI Command line script Comments
Full backup X X  
Partial backup X X If you perform a partial backup using the command line script, the backup does not include dependent objects.
Partial restore X  
Merge changes during restore
X X Merges objects in the backup with existing KV store objects.
Clean restore
  X Replaces existing KV store objects with objects in the backup.

Difference between an ITSI backup and a Splunk Enterprise backup

Splunk Enterprise offers an option to back up and restore the KV store. For more information, see Back up and restore KV store in the Splunk Enterprise Admin Manual. However, an ITSI backup is specifically formatted to process the content in the ITSI backup files. The Splunk Enterprise backup is not formatted like an ITSI backup, so you cannot use it to back up your ITSI data.

ITSI processes all backup content. ITSI also triggers many other activities, such as saved search generation and object dependency updates. Directly restoring Splunk Enterprise KV store data does not restore the ITSI system completely. Instead, use the processes described in this topic to back up your ITSI data.

What gets backed up

The following table describes the types of data included and not included in an ITSI backup.

Data Example Included in backup?
KV store objects Services, service templates, entities, KPIs, KPI base searches, teams, glass tables, service analyzers, deep dives Yes
Indexed data ITSI summary index, notable events No

To back up indexed data, use the same approach you use to back up other Splunk indexes. For more information, see Back up indexed data in the Splunk Enterprise Managing Indexers and Clusters of Indexers manual.

Default scheduled backup

The default scheduled backup is a full backup that runs daily at 1:00 AM in the server's local time zone. The time of the backup job is displayed on the Backup/Restore Jobs page. You cannot create additional scheduled backup jobs.

The following limits apply to keeping scheduled backup files in the backup history:

  • A minimum of 7 days if running a daily scheduled backup
  • A minimum of 14 days if running a weekly scheduled backup
  • A maximum of 30 days for all scheduled backup files

The oldest backup file automatically deletes from the backup history after the retention time expires. To set the schedule and retention time period for scheduled backups, edit the settings as defined in the next section.

Edit the default scheduled backup

When you edit the default scheduled backup, you can change the name of the scheduled backup, the frequency and time to run the scheduled backup, how many days to keep the backup file, or you can disable the scheduled backup. You cannot delete the scheduled backup job.

Prerequisites

You must have the itoa_admin role to view and edit the settings for the default scheduled backup job.

Steps

  1. Click Configure > Backup/Restore from the ITSI top menu bar.
  2. Click the default scheduled backup, or in the Actions column, click Edit > Edit.
  3. (Optional) Change the name and add a description for the default scheduled backup.
  4. (Optional) Enable Include .conf files to back up the following configuration files located in $SPLUNK_HOME/etc/apps/SA-ITOA/local and $SPLUNK_HOME/etc/apps/itsi/local:

    ITSI only backs up these .conf files if they exist in a non-default directory, such as $SPLUNK_HOME/etc/apps/itsi/local. For more information, see About configuration files. When restored, the backed up .conf file overrides the existing local version.

  5. Select a daily or weekly schedule, including the time, to run the scheduled backup. The default setting runs the backup job daily at 1:00 AM in the server's local time zone.
  6. Set the number of days you want to keep the backup file. The oldest backup file automatically disappears from the backup history after the retention time expires.
  7. Click Save.

Create a full backup

Create a full backup to make a copy of all your ITSI configuration information.

Before creating a backup, make sure no service templates are syncing. Check the sync status of service templates by clicking Configure > Service Templates from the ITSI top menu bar.

Prerequisites

You must have the itoa_admin role or the write_itsi_backup_restore capability to create a backup job.

Steps

  1. Click Configure > Backup/Restore from the ITSI top menu bar.
  2. Click Create Job > Create Backup Job.
  3. Select Full Backup.
  4. Provide a name and description of the backup job.
  5. (Optional) Enable Include .conf files to back up the following configuration files located in $SPLUNK_HOME/etc/apps/SA-ITOA/local and $SPLUNK_HOME/etc/apps/itsi/local:

    ITSI backs up these .conf files only if they exist in a non-default directory, such as $SPLUNK_HOME/etc/apps/itsi/local. For more information, see About configuration files. When restored, the backed up .conf file overrides the existing local version.

  6. Click Create.

The backup job appears on the Backup/Restore Jobs page with the status Queued until the job runs. When the backup job finishes, the status changes to Completed and a confirmation message appears in the Messages drop-down list in Splunk Web.

You can run any completed backup job again by clicking Edit > Start Backup in the Actions column. You can also modify the completed backup job before running it again.

Create a partial backup

Create a partial backup if you want to back up a subset of your KV store objects. You can back up services, service templates, teams, glass tables, and configuration files. When selecting one of these object types, dependent objects are automatically selected to preserve the functionality of the objects after they are restored. In some cases, you can choose whether or not to include dependent objects in the backup.

Before creating a backup, make sure no service templates are syncing. Check the sync status of service templates by clicking Configure > Service Templates from the ITSI top menu bar.

What happens when you back up a service

When you back up a service, ITSI also backs up the following objects:

    • KPI base searches
    • Threshold templates
    • Teams

You can also choose whether to back up the following associated objects:

    • Dependent services
    • Entities that match service entity rules
    • A linked service template

If you do not choose to back up an associated object, the dependency between a service and the object breaks when you restore.

What happens when you back up a service template

When you back up a service template, all the services linked to the service template are added to the backup. If you choose to not back up a linked service, it will not exist in the restored environment.

What happens when you back up a team

When you back up a team, all the services associated with that team are added to the backup. You can deselect any services you do not want to back up.

What happens when you back up a glass table

When you back up a glass table, all of the services associated with that glass table are added to the backup. If you choose not to back up a service that the glass table depends on, any visualizations that use KPIs from the service will no longer function.

Glass table images and access control lists (ACLs) are always included in the backup when you back up a glass table.

What happens when you back up a deep dive

When you back up a deep dive, all of the services associated with that deep dive are also added to the backup. If you choose not to back up a service that the deep dive depends on, any KPI swimlanes from the service will no longer function if the service does not exist in the restored environment.

Prerequisites

You must have the itoa_admin role or the write_itsi_backup_restore capability to create a backup job.

Steps

  1. Click Configure > Backup/Restore.
  2. Click Create Job > Create Backup Job.
  3. Select Partial Backup.
  4. Provide a name and description of the backup.
  5. (Optional) Toggle Include .conf files to back up the following configuration files located in $SPLUNK_HOME/etc/apps/SA-ITOA/local and $SPLUNK_HOME/etc/apps/itsi/local:

    ITSI backs up these .conf files only if they exist in a non-default directory, such as $SPLUNK_HOME/etc/apps/itsi/local. For more information, see About configuration files. When restored, the backed up .conf file overrides the existing local version.

  6. Click Next.
  7. On the partial backup page, select the objects to include in the backup. If you select one object type, it can cause other object types to be automatically selected if there are dependencies between the objects.
  8. (Optional) Click Change Settings to change the objects that are selected when you select a service. By default, dependent services are selected. The KPI base searches, threshold templates, and team associated with a service are always included in the backup.
  9. (Optional) Although entities are not listed in the partial backup page, you can include them in the backup file by selecting Entities in the Settings dialog box.
  10. After making your selections, verify the objects that you selected.
  11. Click Save and Backup.

The backup job appears on the Backup/Restore Jobs page with the status Queued until the job runs. When the backup job finishes, the status changes to Completed and a confirmation message appears in the Messages drop-down list in Splunk Web.

You can edit any partial backup job before it starts. When the backup job starts, you see a read-only view that lists the objects contained in the partial backup.

You can run any completed backup job again by clicking Edit > Start Backup in the Actions column. You can also modify the completed backup job before running it again.

Restore a full or partial backup

Restoring a backup merges the JSON data contained in the backup ZIP file with your existing KV store data in the following ways:

  • If you added new objects since you created the backup, ITSI keeps these objects.
  • If an existing object matches an object in the backup file, the existing object is replaced.
  • All other existing objects are preserved.

Use the command line script instead of the UI to restore a backup in the following scenarios:

  • You want to delete all existing KV store objects in an ITSI instance and replace them with the objects in the backup for a clean restore.
  • You want to selectively restore files in a backup. Restoring from the UI restores all of the data in the backup file.

If you restart Splunk software while a backup or restore job is in progress, the job resumes after the restart is complete. Queued jobs automatically time out if they are not completed within 12 hours. You can change the default timeout duration by updating the value of job_queue_timeout in the [backup_restore] stanza in a local version of itsi_settings_conf.

How restoring handles teams

ITSI teams were introduced in version 3.0. If you are restoring from an earlier version of ITSI to version 3.0 or later, all services and service-related objects are placed in the Global team. These objects include entities, KPI templates, KPI base searches, and KPI threshold templates. Backups and subsequent restores on ITSI version 3.0 or later retain team information for services and service-related objects. See Overview of service-level permissions in ITSI for information about teams.

When you restore a backup taken on ITSI version 3.0 or later to another ITSI version 3.0 or later, team ACLs are retained when the teams are restored. The roles assigned to the teams must exist on the system that the backup is restored to. For example, suppose a restore creates teams called "HR" and "Finance", which have read/write access for the hr_admin and finance_admin roles. If the current system does not have these roles, only the itoa_admin role can access these teams. If the roles assigned to the teams don't exist on the system, you can create them either before or after restoring.

Prerequisites

    • Before restoring a backup, make sure no service templates are syncing. Check the sync status of service templates by clicking Configure > Service Templates from the ITSI main menu.
    • Make sure all technology add-ons (TAs), supporting add-ons (SAs), and domain add-ons (DAs) that exist on the old system are installed on the new system.
    • If you've made modifications to any add-ons on the old system, manually copy those add-ons over the new system before restoring.

Restore from a backup

You can restore from a default scheduled backup or a backup that you created.

  1. On the ITSI top menu bar, click Configure > Backup/Restore and find the backup that you want to restore from.
  2. Click Edit > Restore Backup.
  3. If you are restoring a scheduled backup, select a saved backup from the list. If you are restoring a created backup, go to the next step.
  4. Click Start Restore.
    Restore from prepends the backup name in the jobs list. A message stating that the restore job successfully completed appears in the messages drop-down list in Splunk Web.
  5. (Optional) If you restore from a backup that contains .conf files, you must restart Splunk software.

Restore from a backup ZIP file

You can download any backup ZIP file that is created when you run a backup job in the UI and then restore from that backup ZIP file using the Backup/Restore Jobs UI. The maximum file size supported for uploading a backup file is 500 MB.

Perform the following steps to download a backup ZIP file:

  1. On the ITSI top menu bar, click Configure > Backup/Restore and find the backup file that you want to download.
  2. Click Edit > Download Backup. If you are restoring a scheduled backup, select a saved backup from the list. If you are restoring a created backup, the backup file displays.
  3. Save the file. The backup ZIP file downloads to your local machine.

Perform the following steps to restore from a downloaded backup ZIP file:

  1. On the Backup/Restore Jobs page, click Create Job > Create Restore Job.
  2. Provide a name and an optional description of the backup.
  3. Click Choose File and select the previously downloaded backup ZIP file that you want to restore from.
  4. (Optional) Toggle Include .conf files to restore any configuration files included in the backup.
  5. Click Create.
    ITSI uploads the backup ZIP file and the new restore job appears in the Backup/Restore Jobs list. A message stating that the restore job has successfully completed appears in the Message drop-down list in Splunk Web.
  6. (Optional) If you restore from a backup that contains .conf files, you must restart Splunk software.

Restore from a backup created using the command line

If you created a backup of ITSI using the kvstore_to_json.py command line option, and you want to restore that data using the Backup/Restore Jobs page, the backup JSON files must be contained in a folder named backup and compressed into a ZIP file. For information, see kvstore_to_json.py operations in ITSI.

Back up and restore in a search head cluster environment

You can run backup and restore jobs from the Backup/Restore page in search head cluster environments. You can create a backup on any cluster member and then later restore data from that backup on any cluster member, regardless of where the backup was initiated.

For example, suppose your search head cluster has three cluster members: sh-01, sh-02, and sh-03. If you create a backup on sh-01, you can later restore from that backup on sh-01, sh-02, or sh-03.

When you create a backup on any search head cluster member, the configuration data from all cluster members is backed up. Likewise, when you restore from a backup on any cluster member, configuration data is restored across all cluster members.

In a search head cluster environment, the scheduled backup runs only on the search head cluster captain. However, you can perform a restore of a scheduled backup from any cluster member. If you download the scheduled backup, make sure to download it from the captain because the captain contains the latest backup.

PREVIOUS
Schedule maintenance downtime in ITSI
  NEXT
kvstore_to_json.py operations in ITSI

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.4.1


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters