
kvstore_to_json.py operations in ITSI
ITSI provides a kvstore_to_json.py
script that lets you backup/restore ITSI configuration data, perform bulk service KPI operations, and apply time zone offsets for ITSI objects.
Usage options
The kvstore_to_json.py
script is located in $SPLUNK_HOME/etc/apps/SA-ITOA/bin/
.
The kvstore_to_json.py
script has these 3 modes:
- Mode 1: Backup and restore operations
- Mode 2: Bulk service KPI operations.
- Mode 3: Time zone offset operations.
To view all kvstore_to_json.py
usage options, specify the -h
option. For example:
[root@myserver splunk]# ./bin/splunk cmd python etc/apps/SA-ITOA/bin/kvstore_to_json.py -h Usage: kvstore_to_json.py [options] Options: -h, --help show this help message and exit -s SPLUNKDPORT, --splunkdport=SPLUNKDPORT splunkd port. If no option is provided, we will default to '8089' -u USERNAME, --username=USERNAME Splunk username -p PASSWORD, --password=PASSWORD Splunk password -n, --no-prompt Use this option when you want to disable the prompt version of this script -v, --verbose Use this option for verbose logging -f FILE_PATH, --filepath=FILE_PATH The full path of a directory. Usage depends on mode. When importing backed up data of version 1.2.0, this could be a file or a set of files. When working with service KPIs, this is a directory containing input.json on entry and output.json on exit. -m MODE, --mode=MODE Specify the mode of operation - what kind of operations to perform. Mode is set to: 1 - for backup/restore operations. 2 - for service KPI operations. Backup and restore operations. This is mode 1.: Use this option when you want to perform backup/restore operations. -i, --importData Use this option when you want to upload data to the KV Store. When importing data from version 1.2.0, you can use filepath as wildcard to upload data from more than one file. However, filepath must be within quotes if it is being used as a wildcard -d, --persist-data Use this option when you want to persist existing configuration in KV Store during import. NOTE: Applicable only if importData option is used -y, --dry-run Use this option when you want only to list objects for import or backup -b BR_VERSION, --base-version=BR_VERSION The original ITSI application version user intends to backup/restore from. -l RULE_FILE_PATH, --rule_file_path=RULE_FILE_PATH The full path of a file which has rules defined for backing up data or importing data -e DUPNAME_TAG, --dupname-tag=DUPNAME_TAG Automatically rename all the duplicated service or entity names from restoring with a tag. If this option is not set, the restoring will halt if duplicate names are detected. The default tag is: _dup_from_restore_<epoch_timestamp> Service KPI operations. This is mode 2.: Use this option when you want to get/create/update/delete KPIs for existing services. -g, --get For input, specify a list of service keys with the keys of KPIs to retrieve. Expected format: [{_key: <service key>, kpis: [{_key: <KPI key>}]]. Specify [] to get all KPIs from all services. Specify [{_key: <service key>, kpis: []] to get all KPIs from a service. Assumes input is available in file_path/input.json -c, --create For input, specify a non-empty list of service keys with their KPIs list. Expected format: [{_key: <service key>, kpis: [{_key: <KPI key>, <rest of KPI structure>}]]. Note that only existing services could be updated with new KPIs only with this option. Assumes input is available in file_path/input.json -t, --update For input, specify a non-empty list of service keys with their KPIs list. Expected format: [{_key: <service key>, kpis: [{_key: <KPI key>, <rest of KPI structure>}]]. Note that only existing services and existing KPIs could be updated using this option. Assumes input is available in file_path/input.json -r, --delete For input, specify a list of service keys with the keys for the KPIs to delete.Expected format: [{_key: <service key>, kpis: [{_key: <KPI key>}]]. Assumes input is available in file_path/input.json Timezone offset operations. This is mode 3.: Use this option when you want to adjust timezone settings for time sensitive fields on object configuration. -q IS_GET, --is_get=IS_GET For input, specify if you are trying to read objects or update their timezone offsets. -o OBJECT_TYPE, --object_type=OBJECT_TYPE For input, specify a valid object type that contains time sensitive configuration. This option will apply offset to all objects on this type unless scoped to a specific object using object_key parameter.Supported object types are: "maintenance_calendar" for maintenance windows, "service" for Services/KPIs (threshold policies) -k OBJECT_TITLE, --object_title=OBJECT_TITLE For input, specify an optional object title of object type that contains time sensitive configuration. Using this option will cause the offset change to only apply to that object. -z OFFSET_IN_SEC, --offset_seconds=OFFSET_IN_SEC For input, specify the offset to apply in seconds as a positive or negative number. This offset should be the number of seconds that you want to add or subtract from the current value.
Backup and restore operations (mode 1)
Use kvstore_to_json.py
mode 1 options to backup and restore your ITSI configuration data to and from JSON files. The kvstore_to_json
script can do full or partial backups and merged or clean restores.
When restoring in a replicated KV store environment, the script works on any member of the search head cluster. It does not require execution on the captain.
You can run the kvstore_to_json
script in either interactive mode or non-interactive mode. In interactive mode, which is the default, the script prompts you to enter required information.
To run the script in non-interactive mode, you must enter the -n
option to disable the default interactive mode.
You can schedule backups through cron by calling the script with all necessary flags for full automation.
Before performing a backup or restore through the CLI, make sure that there are no service template syncs in progress. Check the sync status of service templates in the Service Template viewer by selecting Configure > Service Templates from the top menu.
Backup operations
Interactive mode
- Run
kvstore_to_json.py
, where-s
is the defaultsplunkd
port. If you are runningsplunkd
on any other port besides 8089, you must specify the port. For example:cd $SPLUNK_HOME bin/splunk cmd python etc/apps/SA-ITOA/bin/kvstore_to_json.py -s 8089 -u admin -p changeme -m 1 -f /root/backup
- Enter the requested information at the prompts. For example:
>> Enter the splunkd port number OR press the enter key to use [8089] >> Do you wish to backup data from KV Store OR restore to KV store. Press [y|yes|enter] to backup, [n|no] to restore? > y You have indicated backup of data from current ITSI version 2.4.0. Proceeding.
The script copies your KV store data to a series of JSON files in the/root/backup
directory. If the/root/backup
directory does not exist, the script creates it. - Go to
/root/backup
to confirm that your backup JSON files have been created. Backup file names follow the specific format:<COLLECTION_NAME>___<OBJECT_TYPE>___<ROLLOVER_NUMBER>
. For example:* itsi_migration___migration___0.json * itsi_pages___glass_table___0.json * itsi_services___entity___0.json * itsi_services___kpi_threshold_template___0.json * itsi_pages___deep_dive___0.json * itsi_service_analyzer___home_view___0.json * itsi_services___kpi_template___0.json * itsi_services___service___0.json * itsi_services___correlation_search___0.json
Caution: Do not change the JSON file name. If you change the file name you cannot restore the data.
Non-interactive mode
- Run
kvstore_to_json.py
, using the appropriate backup options. Enter-n
to disable the default interactive mode. Use the-b
option to specify the ITSI version that you want to backup. If you do not specify-b
, the current ITSI version is used. For example:cd $SPLUNK_HOME bin/splunk cmd python etc/apps/SA-ITOA/bin/kvstore_to_json.py -s 8089 -u admin -p changeme -m 1 -f /root/backup -n -b "2.2.2"
- Go to
/root/backup
and confirm that your backup JSON files have been created.
Restore operations
ITSI supports restore migration from the current backup or an earlier version of the backup. You can migrate backup JSON files based on an earlier ITSI version (2.2.0 or later only) to the most recent ITSI version.
To restore from backups based on ITSI version 2.1.0, you must first upgrade to ITSI version 2.2.2, then create the backup. You can then restore from the version 2.2.2 backup after upgrading to version 2.4.0.
Note: Restore from backups based on ITSI 1.2.0 or 2.0.0 is not supported.
When restoring a backup taken on an ITSI 3.x system to another ITSI 3.x system, team ACLs are retained when the teams are restored. Therefore, the roles assigned to the teams must exist on the system the backup is restored to. For example, a restore creates teams called HR and Finance which have read/write access for the roles hr_admin and finance_admin, respectively. If the current system does not have the hr_admin and finance_admin roles, these teams will only be accessible to the itoa_admin role. If the roles assigned to the teams don't already exist on the system, they can be created either prior to restoring or after restoring.
Interactive mode
- Run
kvstore_to_json.py
, using the-i --importData
option to upload data from your JSON backup files to the KV store. For example:cd $SPLUNK_HOME bin/splunk cmd python etc/apps/SA-ITOA/bin/kvstore_to_json.py -s 8089 -u admin -p changeme -m 1 -f /root/backup -i
- Enter the requested information at the prompt. For example:
>> Do you wish to backup data from KV Store OR restore to KV store. Press [y|yes|enter] to backup, [n|no] to restore? > n You would like to restore data from disk to KV Store. Proceeding. >> Enter the previous ITSI version from which you want the data to be restored. Ex: '2.2.0, '2.2.1', '2.2.2' > >> Do you wish to persist the existing data in KV Store during the import [y|n]: > y
Caution: The-i
option deletes all existing data from the KV store. If you do not want to delete all data from the KV store on import, entery
at the prompt to use-d --persist data
option. This option appends data to the existing data in the KV store. - Open the ITSI app and verify that your configuration data has been restored.
Non-interactive mode
To restore data using non-interactive mode, you must use the -b
option to specify the ITSI version of the backup file you plan to restore.
- Run
kvstore_to_json.py
, using the-i --importData
option to upload data from your JSON backup files to the KV store. Use the-b
option to specify the ITSI version of the backup file. Make sure to include the-n
option to disable interactive mode. For example:cd $SPLUNK_HOME bin/splunk cmd python etc/apps/SA-ITOA/bin/kvstore_to_json.py -s 8089 -u admin -p changeme -m 1 -f /root/backup -i -n -b "2.2.0" -d
Caution: The-i
option deletes all data from the KV store. If you do not want to delete all data from the KV store on import, enter the-d --persist data
option. This option appends data to the existing data in the KV store. - Open the ITSI app and verify that your configuration data has been restored.
Use a wildcard to specify groups of files
You can use a wildcard with the -f
option to specify a group of files. For example, to import the files
itsi_export.json
itsi_export_0.json
itsi_export_1.json
you could enter the -f
path as follows:
cd $SPLUNK_HOME bin/splunk cmd python etc/apps/SA-ITOA/bin/kvstore_to_json.py -s 8089 -u admin -p changeme -m 1 -f "/root/itsi_export*" -i
Note: The value of -f
must be in quotes.
Partial backup/restore operations
kvstore_to_json.py
mode 1 also lets you perform partial backup/restore operations on a subset of your ITSI configuration data. Partial backup/restore is useful for moving small numbers of objects between a test environment and a production environment. It also helps you minimize the amount of data involved in backup/restore operations, which can save you time and resources.
To perform partial backup/restore of ITSI data you must first create a JSON file that defines rules for data backup or import. The rules file has the following schema:
[ { object_type (required): <object_type> (Top level objects, valid values are ['service', 'entity', 'home_view', 'kpi_template', 'deep_dive', 'kpi_threshold_template', 'correlation_search', 'glass_table']), title_list (conditional required): regex/string or list of regex/string based titles for import of all objects that match this regex/string (Required or define key_list) key_list (conditional required): list of keys - considered only if title is not defined (Required only if title is empty or not defined) replacement_rules (optional): [ { replacement_key (required): title/_key replacement_type (required): (replace, prefix, postfix) replacement_string (required): "regex/string" replacement_pattern (required only for replace type): "regex/string" } ........ ] ...... } ]
Partial backup
To backup a single object, such as a service:
- In any directory, create a JSON file to use as your rules file. For example:
touch rules.json
- Edit the
rules.json
file as shown in the schema to define a rule for backing up data. For example, to backup a single service called "Database Service" inrules.json
you might specify:[ { "object_type": "service", "title_list": "^Database Service$" } ]
- Run the
kvstore_to_json.py
command using the dry run (-y) option../bin/splunk cmd python etc/apps/SA-ITOA/bin/kvstore_to_json.py -u admin -p changeme -f /root/backup_folder_2 -y -l /root/partial_back_up/rules.json -n
- Verify that the list of objects to backup is correct, before you perform the actual backup. For example:
>>>> This is a dry run. No actual backup or importing will happen, only a list of objects will be displayed. To perform the actual operation, re-run again without the flag for dry run ... <<<<<<<<<<<<<<<<<<< Object Type = service >>>>>>>>>>>>>>>>>>>>>>> Title = Database Service (_key = 97dc7a6a-5706-4d67-864b-b0e7a72e326c) <<<<<<<<<<<<<<<<<<< Object Type = entity >>>>>>>>>>>>>>>>>>>>>>> <<<<<<<<<<<<<<<<<<< Object Type = kpi_template >>>>>>>>>>>>>>>>>>>>>>> <<<<<<<<<<<<<<<<<<< Object Type = kpi_threshold_template >>>>>>>>>>>>>>>>>>>>>>> <<<<<<<<<<<<<<<<<<< Object Type = glass_table >>>>>>>>>>>>>>>>>>>>>>> <<<<<<<<<<<<<<<<<<< Object Type = deep_dive >>>>>>>>>>>>>>>>>>>>>>> <<<<<<<<<<<<<<<<<<< Object Type = home_view >>>>>>>>>>>>>>>>>>>>>>> <<<<<<<<<<<<<<<<<<< Object Type = migration >>>>>>>>>>>>>>>>>>>>>>>
- Run the command without the dry run (
-y
) option to perform the actual backup.
If performing a partial backup of a shared glass table, deep dive, or service analyzer (home view) using the method described in the previous section, the glass table, deep dive, or service analyzer may not be restored due to the fact that the ACL JSON file is not backed up and restored along with the shared object. To work around this issue, change the permissions of the object to Private before backing it up. Then change the permissions on the object back to Shared in App after restoring it.
If the glass table contains external images, use the method described in the following section to back up the glass table.
Partial backup of a glass table with external images
When performing a partial backup of a glass table, images in the glass table may not display after it has been restored. This is because the JSON file containing external images for the glass table needs to be backed up and restored along with the glass table. To back up the images that go with the glass table, as well as the ACLs necessary to restore a shared glass table, do the following:
- Create a full backup without using
rules.json
. - Create a partial backup of the glass table using the
rules.json
file. - Create a
backup_partial
directory with the following files:-
app_acl___app_acl___0.json
From the full backup in step 1. This file contains the information needed to restore any glass tables that have "Shared in App" permissions. -
SA-ITOA_files___glass_table_images___0.json
From the full backup in step 1. This file contains the glass table images. -
itsi_pages___glass_table___0.json
From the partial backup of the glass table in step 2. -
app_info.json
From either the full or partial backup in steps 1 or 2.
-
- Perform a full restore using the backup directory created in step 3. For example: The glass table will be restored with its images.
cd $SPLUNK_HOME bin/splunk cmd python etc/apps/SA-ITOA/bin/kvstore_to_json.py -s 8089 -u admin -p changeme -m 1 -f /root/backup_partial -i
Partial restore
Use partial restore when you have a full backup of your ITSI configuration data and you want to restore only a few objects, instead of all objects.
For example, to restore a single object, such as a service:
- In any directory, create a JSON rules file. For example:
touch rules.json
- Edit
rules.json
to define rules for importing the object. For example:[ { "object_type": "service", "title_list": "^Database Service$" } ]
- Run the
kvstore_to_json.py
command using the dry run (-y) option, where-f
is the path to the folder containing your full data backup,-l
is the path to therules.json
file,-i
is the option to import data, and-d
is the option to persist data (to avoid deletion of existing objects). For example:./bin/splunk cmd python etc/apps/SA-ITOA/bin/kvstore_to_json.py -u admin -p changeme -f /root/full_backup_folder -l /root/partial_back_up/rules.json -i -d -y -n
- After you verify that the list of objects to import in the dry run is correct, run the command without the dry run (
-y
) option to perform the actual restore operation.
Use replacement options
The partial backup/restore rules schema provides replacement options, which let you change the name of an object when you run a partial backup/restore operation. Replacement options are useful for renaming objects when moving from a test environment to a production environment.
For example, to backup a service called test_database_service, but change the name to database_service; and to backup a deep dive called test_database_deep_dive, and change the name to database_deep_dive, you would create a rules.json
file that contains the following:
[ { "object_type": "service", "title_list": "^test_database_service$", "replacement_rules": [ { "replacement_key": "title", "replacement_type": "replace", "replacement_string": "database_service", "replacement_pattern": "^test_database_service$" }] }, { "object_type": "entity", "title_list": ["10.12.*", "*host_database*"] }, { "object_type": "deep_dive", "title_list": "^test_database_deep_dive$", "replacement_rules": [ { "replacement_key": "title", "replacement_type": "replace", "replacement_string": "database_deep_dive", "replacement_pattern": "^test_database_deep_dive$" }] } ]
About dependent objects
When you perform partial backup/restore operations, you must include rules for all dependent objects, such as services, deep dives, glass tables, entities, and so on.
For example, in the ITSI configuration that the above rules file references, test_database_deep_dive contains test_database_service KPI lanes, and thus test_database_service is a dependency of test_database_deep_dive. In this case, you must define rules for all dependent objects, including deep dives, services, and entities (if your service contains entities).
Service KPI operations (mode 2)
kvstore_to_json.py
mode 2 options let you run bulk operations on KPIs, including get (-g
), create (-c
), update (-t
), and delete (-r
). Use these options to replicate, edit, and copy KPIs to multiple services, for example, when moving your ITSI deployment from a test environment to a production environment.
All service KPI options require you to specify the mode 2 parameter -m 2
. You must also specify the file path -f
parameter as the full path to the directory containing the input.json
file.
Before you can run service KPI operations, you must create an input.json
file in the destination directory. The script accepts data input from input.json
and sends data output to an output.json
file that the script creates in the same directory.
All service KPI operations, except get -g
, require you to specify service and/or KPI keys. You can retrieve these keys using the -g
option in output.json
.
Note: See the kvstore_to_json.py
help - h
option for proper input.json
and command syntax.
Get service and KPI keys
Use the get -g
option to retrieve service and KPI data in JSON format, including service and KPI keys.
- Create an
input.json
file in the destination directory.mkdir <directory_containing_input.json> touch input.json
- Edit
input.json
: Add[]
to the file to retrieve JSON data for all services and KPIs, or add specific service and kpi keys to the file to retrieve JSON data for those services and KPIs only. For example:[{"_key": "<service_key>", "kpis": [{"_key": "<kpi_key>"}] } ].
- Run the
kvstore_to_json
script using the get-g
option. Name the full path to the directory containing theinput.json
file as the file path-f
parameter. For example:cd $SPLUNK_HOME bin/splunk cmd python kvstore_to_json.py -u admin -p changeme -m 2 -g -f <directory_containing_input.json> -n
- Review the contents of
output.json
to identify service and KPI keys. For example:[ { "_key": "669c5cec-a492-419d-8659-95a185b4dc5c", "kpis": [ { "_key": "f017cc7b2e67f2b3b9152146", ... } ] } ]
Create KPIs
Use the -c
option to create new KPIs.
- Edit
input.json
to specify the service key of the service for which you want to create the KPI. - Add KPI keys for the KPIs that you want to add to the service and any key-value pairs belonging to the KPI that you want to include in the KPI definition. Leave the key field for each KPI empty for ITSI to auto generate it. For example:
[ { "_key": "<service_key>", "kpis": [ { "title": "<title_of_kpi_to_create>", ... } ] } ]
Update KPIs
Use the -t
option to update KPIs.
In input.json
specify the service and KPI key for each KPI, and any other key/value pair data that you want to update for the KPI.
Delete KPIs
Use the -r
option to delete KPIs.
In input.json
, specify service and kpi keys for all KPIs that you want to delete.
Caution: Make sure to properly validate your JSON input. While the kvstore_to_json
script does provide some schema validation, incorrect JSON formatting can cause errors.
Time zone offset operations (mode 3)
The kvstore_to_json.py
mode 3 option lets you apply a time offset for time-sensitive fields in object configurations. You can use this option to correct time zone discrepancies for the following object types:
maintenance_calendar
: Sets an offset for maintenance window start and end times.service
: Sets an offset for the KPI threshold time policies within a service. See Apply time zone offsets in this manual.kpi_threshold_template
: Sets an offset for a KPI threshold template. Sets an offset for a KPI threshold template. If your services use KPI threshold templates in addition to time policies, you must first run the script on the services, then run it again on each of the KPI threshold templates. The changes propagate down to the KPIs that use that template.
Apply time zone offset
Run the following command to set an offset for one of the supported object types.
- Run
kvstore_to_json.py
, where-o
is the object type,-k
is the name of the specific object, and-z
is the specific time zone offset in seconds. For example:cd $SPLUNK_HOME bin/splunk cmd python etc/apps/SA-ITOA/bin/kvstore_to_json.py -m 3 -o service -k "Database Service" -z 1800
- Enter the requested information at the prompts (default interactive mode only). For example:
>> Enter the splunkd port number OR press the enter key to use [8089] > 8089 >> Enter splunk username or press enter to use "admin" > admin >> Enter splunk password for "admin" >
The script applies the time zone offset to the specified object. For example:
1 object(s) match request Applying timezone change on requested object(s): [u'Database Service'] Timezone offset has been applied on the objects requested.
ITSI time-sensitive configurations are normalized to UTC. For more information on time zones in ITSI, see Time zone handling in this manual.
PREVIOUS Backup and restore ITSI data |
NEXT About configuration files in ITSI |
This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.0.3, 4.0.4
Feedback submitted, thanks!