Splunk® IT Service Intelligence

Administer Splunk IT Service Intelligence

Download manual as PDF

Download topic as PDF

kvstore_to_json.py operations in ITSI

ITSI provides a kvstore_to_json.py script that lets you backup/restore ITSI configuration data, perform bulk service KPI operations, apply time zone offsets for ITSI objects, and regenerate KPI search schedules.

Usage options

The kvstore_to_json.py script is located in $SPLUNK_HOME/etc/apps/SA-ITOA/bin/.

The kvstore_to_json.py script has these 4 modes:

Mode 1: Backup and restore operations
Mode 2: Bulk service KPI operations.
Mode 3: Time zone offset operations.
Mode 4: Regenerate KPI search schedules.

To view all kvstore_to_json.py usage options, specify the -h option.


[root@myserver splunk]# ./bin/splunk cmd python etc/apps/SA-ITOA/bin/kvstore_to_json.py -h

Usage: kvstore_to_json.py [options]

Options:
  -h, --help            show this help message and exit
  -s SPLUNKDPORT, --splunkdport=SPLUNKDPORT
                        splunkd port. If no option is provided, we will
                        default to '8089'
  -u USERNAME, --username=USERNAME
                        Splunk username
  -p PASSWORD, --password=PASSWORD
                        Splunk password
  -n, --no-prompt       Use this option when you want to disable the prompt
                        version of this script
  -v, --verbose         Use this option for verbose logging
  -f FILE_PATH, --filepath=FILE_PATH
                        The full path of a directory. Usage depends on mode.
                        When importing backed up data of version 1.2.0, this
                        could be a file or a set of files. When working with
                        service KPIs, this is a directory containing
                        input.json on entry and output.json on exit.
  -m MODE, --mode=MODE  Specify the mode of operation - what kind of
                        operations to perform. Mode is set to:         1 - for
                        backup/restore operations.         2 - for service KPI
                        operations.

  Backup and restore operations. This is mode 1.:
    Use this option when you want to perform backup/restore operations.

    -i, --importData    Use this option when you want to upload data to the KV
                        Store. When importing data from version 1.2.0, you can
                        use filepath as wildcard to upload data from more than
                        one file. However, filepath must be within quotes if
                        it is being used as a wildcard
    -d, --persist-data  Use this option when you want to persist existing
                        configuration in KV Store during import. NOTE:
                        Applicable only if importData option is used
    -y, --dry-run       Use this option when you want only to list objects for
                        import or backup
    -a, --conf-file     Use this option when you want to back up .conf files.
    -b BR_VERSION, --base-version=BR_VERSION
                        The original ITSI application version user intends to
                        backup/restore from.
    -l RULE_FILE_PATH, --rule_file_path=RULE_FILE_PATH
                        The full path of a file which has rules defined for
                        backing up data or importing data
    -e DUPNAME_TAG, --dupname-tag=DUPNAME_TAG
                        Automatically rename all the duplicated service or
                        entity names from restoring with a tag. If this option
                        is not set, the restoring will halt if duplicate names
                        are detected. The default tag is:
                        _dup_from_restore_<epoch_timestamp>

  Service KPI operations. This is mode 2.:
    Use this option when you want to get/create/update/delete KPIs for
    existing services.

    -g, --get           For input, specify a list of service keys with the
                        keys of KPIs to retrieve. Expected format: [{_key:
                        <service key>, kpis: [{_key: <KPI key>}]]. Specify []
                        to get all KPIs from all services. Specify [{_key:
                        <service key>, kpis: []] to get all KPIs from a
                        service. Assumes input is available in
                        file_path/input.json
    -c, --create        For input, specify a non-empty list of service keys
                        with their KPIs list. Expected format: [{_key:
                        <service key>, kpis: [{_key: <KPI key>, <rest of KPI
                        structure>}]]. Note that only existing services could
                        be updated with new KPIs only with this option.
                        Assumes input is available in file_path/input.json
    -t, --update        For input, specify a non-empty list of service keys
                        with their KPIs list. Expected format: [{_key:
                        <service key>, kpis: [{_key: <KPI key>, <rest of KPI
                        structure>}]]. Note that only existing services and
                        existing KPIs could be updated using this option.
                        Assumes input is available in file_path/input.json
    -r, --delete        For input, specify a list of service keys with the
                        keys for the KPIs to delete.Expected format: [{_key:
                        <service key>, kpis: [{_key: <KPI key>}]]. Assumes
                        input is available in file_path/input.json

Timezone offset operations. This is mode 3.:
    Use this option when you want to adjust timezone settings for time
    sensitive fields on object configuration.

    -q IS_GET, --is_get=IS_GET
                        For input, specify if you are trying to read objects
                        or update their timezone offsets.
    -o OBJECT_TYPE, --object_type=OBJECT_TYPE
                        For input, specify a valid object type that contains
                        time sensitive configuration. This option will apply
                        offset to all objects on this type unless scoped to a
                        specific object using object_key parameter.Supported
                        object types are: "maintenance_calendar" for
                        maintenance windows, "service" for Services/KPIs
                        (threshold policies)
    -k OBJECT_TITLE, --object_title=OBJECT_TITLE
                        For input, specify an optional object title of object
                        type that contains time sensitive configuration. Using
                        this option will cause the offset change to only apply
                        to that object.
    -z OFFSET_IN_SEC, --offset_seconds=OFFSET_IN_SEC
                        For input, specify the offset to apply in seconds as a
                        positive or negative number. This offset should be the
                        number of seconds that you want to add or subtract
                        from the current value.

Running the script in a search head cluster environment

When running the kvstore_to_json.py script in a replicated KV store environment, the script works on any member of the search head cluster. It does not require execution on the captain.

It is best practice to run the script on the MongoDB captain, which might be different than the captain of the search head cluster.

Backup and restore operations (mode 1)

Use kvstore_to_json.py mode 1 to back up and restore your ITSI configuration data to and from JSON files. You can use the kvstore_to_json script to do full or partial backups, full or partial restores, and merged or clean restores.

See Backup and restore ITSI configuration data for a summary of the types of data included and not included in an ITSI backup.

You can run the kvstore_to_json script in either interactive mode or non-interactive mode. In interactive mode, which is the default, the script prompts you to enter required information. To run the script in non-interactive mode, you must enter the -n option to disable the default interactive mode. You can schedule backups through cron by calling the script with all necessary flags for full automation.

Before performing a backup or restore through the CLI, make sure that there are no service template syncs in progress. Check the sync status of service templates in the Service Template viewer by selecting Configure > Service Templates from the top menu.

Perform a full backup using interactive mode

  1. Run kvstore_to_json.py, where -s is the default splunkd port. If you are running splunkd on any other port besides 8089, you must specify the port. For example:
    cd $SPLUNK_HOME
    bin/splunk cmd python etc/apps/SA-ITOA/bin/kvstore_to_json.py -s 8089 -u admin -p changeme -m 1 -f /root/backup
    
  2. Enter the requested information at the prompts. For example:
    >> Enter the splunkd port number OR press the enter key to use [8089]
    >> Do you wish to backup data from KV Store OR restore to KV store.
     Press [y|yes|enter] to backup, [n|no] to restore? > y
    You have indicated backup of data from current ITSI version 4.0.3. Proceeding.
    


    The script copies your KV store data to a series of JSON files in the /root/backup directory. If the /root/backup directory does not exist, the script creates it.

  3. Go to /root/backup to confirm that your backup JSON files have been created. Backup file names follow the specific format: <COLLECTION_NAME>___<OBJECT_TYPE>___<ROLLOVER_NUMBER>. For example:
    * itsi_migration___migration___0.json
    * itsi_pages___glass_table___0.json
    * itsi_services___entity___0.json
    * itsi_services___kpi_threshold_template___0.json
    * itsi_pages___deep_dive___0.json
    * itsi_service_analyzer___home_view___0.json
    * itsi_services___kpi_template___0.json
    * itsi_services___service___0.json
    * itsi_services___correlation_search___0.json
    

    Do not change the JSON file name. If you change the file name you cannot restore the data.

Perform a full backup using non-interactive mode

  1. Run kvstore_to_json.py, using the appropriate backup options. Enter -n to disable the default interactive mode. Use the -b option to specify the ITSI version that you want to back up. If you do not specify -b, the current ITSI version is used. For example:
    cd $SPLUNK_HOME
    bin/splunk cmd python etc/apps/SA-ITOA/bin/kvstore_to_json.py -s 8089 -u admin -p changeme -m 1 -f /root/backup -n  -b "4.0.3"
    
  2. Go to /root/backup and confirm that your backup JSON files have been created.

Restore from a backup using interactive mode

ITSI supports restore migration from the current backup or an earlier version of the backup. You can migrate backup JSON files based on an earlier ITSI version to the most recent ITSI version. You can restore all the files in a backup or selected files.

Prerequisite

Team access control lists are retained when the teams are restored. Therefore, the roles assigned to the teams must exist on the system to which the backup is restored. For example, a restore creates teams called HR and Finance which have read/write access for the roles hr_admin and finance_admin, respectively. If the current system does not have the hr_admin and finance_admin roles, these teams will only be accessible to the itoa_admin role. If the roles assigned to the teams don't already exist on the system, create them either prior to restoring or after restoring.

Steps

  1. Run kvstore_to_json.py, using the -i --importData option to upload data from your JSON backup files to the KV store. For example:
    cd $SPLUNK_HOME
    bin/splunk cmd python etc/apps/SA-ITOA/bin/kvstore_to_json.py -s 8089 -u admin -p changeme -m 1 -f /root/backup -i 
    
  2. Enter the requested information at the prompt. For example:
    >> Do you wish to backup data from KV Store OR restore to KV store.
     Press [y|yes|enter] to backup, [n|no] to restore? > n
    >> Do you wish to include .conf files? [y|n|enter]: > y
    You would like to restore data from disk to KV Store. Proceeding.
    >> Enter the previous ITSI version from which you want the data to be restored. Ex: '2.2.0, '2.2.1', '2.2.2' >
    >> Do you wish to persist the existing data in KV Store during the import [y|n]: > y
    


    Caution: The -i option deletes all existing data from the KV store. If you do not want to delete all data from the KV store on import, enter y at the prompt to use -d --persist data option. This option appends data to the existing data in the KV store.

  3. Open the ITSI app and verify that your configuration data has been restored.

Restore from a backup using non-interactive mode

To restore data using non-interactive mode, you must use the -b option to specify the ITSI version of the backup file you plan to restore.

ITSI supports restore migration from the current backup or an earlier version of the backup. You can migrate backup JSON files based on an earlier ITSI version to the most recent ITSI version. You can restore all the files in a backup or selected files.

Prerequisite

Team access control lists are retained when the teams are restored. Therefore, the roles assigned to the teams must exist on the system to which the backup is restored. For example, a restore creates teams called HR and Finance which have read/write access for the roles hr_admin and finance_admin, respectively. If the current system does not have the hr_admin and finance_admin roles, these teams will only be accessible to the itoa_admin role. If the roles assigned to the teams don't already exist on the system, create them either prior to restoring or after restoring.

Steps

  1. Run kvstore_to_json.py, using the -i --importData option to upload data from your JSON backup files to the KV store. Use the -b option to specify the ITSI version of the backup file. Make sure to include the -n option to disable interactive mode. For example:
    cd $SPLUNK_HOME
    bin/splunk cmd python etc/apps/SA-ITOA/bin/kvstore_to_json.py -s 8089 -u admin -p changeme -m 1 -f /root/backup -i -n -b "4.0.3" -d
    

    The -i option deletes all data from the KV store. If you do not want to delete all data from the KV store on import, enter the -d --persist data option. This option appends data to the existing data in the KV store.

  2. Open the ITSI app and verify that your configuration data has been restored.

Use a wildcard to specify groups of files

You can use a wildcard with the -f option to specify a group of files. For example, to import the files

itsi_export.json 
itsi_export_0.json
itsi_export_1.json 

you could enter the -f path as follows:

cd $SPLUNK_HOME
bin/splunk cmd python etc/apps/SA-ITOA/bin/kvstore_to_json.py -s 8089 -u admin -p changeme -m 1 -f  "/root/itsi_export*" -i

The value of -f must be in quotes.

Perform a partial backup

Support for partial backups using the kvstore_to_json.py script is removed in version 4.3.0. See Removed features in Splunk IT Service Intelligence. Use the Backup/Restore page in the ITSI UI to do partial backups. For information, see Create a partial backup.

Perform a partial restore

You can use the kvstore_to_json.py script to perform a partial restore on a subset of your ITSI configuration data.

Prerequisites

To perform a partial restore of ITSI data, you must first create a JSON file that defines rules for the data restore. The rules file has the following schema:

 
[
 {
  object_type (required): <object_type>  (Top level objects, valid values are ['service', 'entity', 'home_view', 'kpi_template', 'deep_dive', 'kpi_threshold_template', 'correlation_search', 'glass_table']),
  title_list (conditional required): regex/string or list of regex/string based titles for import of all objects that match this regex/string (Required or define key_list)
  key_list (conditional required): list of keys - considered only if title is not defined (Required only if title is empty or not defined)
  replacement_rules (optional): [
     {
         replacement_key (required): title/_key
         replacement_type (required): (replace, prefix, postfix)
         replacement_string (required): "regex/string"
         replacement_pattern (required only for replace type): "regex/string"
     }
     ........
   ]
  ......
 }
]

For example, perform the following steps to restore a single object, such as a service:

  1. In any directory, create a JSON rules file. For example:
    touch rules.json
    
  2. Edit rules.json to define rules for importing the object. For example:
    [ 
     {
        "object_type": "service",
        "title_list": "^Database Service$"
      }
    ]
    

Steps

Use partial restore when you have a full backup of your ITSI configuration data and you want to restore only a few objects, instead of all objects.

For example, to restore a single object, such as a service:

  1. In any directory, create a JSON rules file. For example:
    touch rules.json
    
  2. Edit rules.json to define rules for importing the object. For example:
    [ 
     {
        "object_type": "service",
        "title_list": "^Database Service$"
      }
    ]
    
  3. Run the kvstore_to_json.py command using the dry run (-y) option, where -f is the path to the folder containing your full data backup, -l is the path to the rules.json file, -iis the option to import data, and -d is the option to persist data (to avoid deletion of existing objects). For example:
     ./bin/splunk cmd python etc/apps/SA-ITOA/bin/kvstore_to_json.py -u admin -p changeme -f /root/full_backup_folder -l /root/partial_back_up/rules.json -i -d -y -n
    
  4. After you verify that the list of objects to import in the dry run is correct, run the command without the dry run (-y) option to perform the actual restore operation.

Use replacement options

The partial restore rules schema provides replacement options, which let you change the name of an object when you run a partial backup/restore operation. Replacement options are useful for renaming objects when moving from a test environment to a production environment.

For example, to backup a service called test_database_service, but change the name to database_service; and to backup a deep dive called test_database_deep_dive, and change the name to database_deep_dive, you would create a rules.json file that contains the following:

[ 
 {
    "object_type": "service",
    "title_list": "^test_database_service$",
    "replacement_rules": [ { 
        "replacement_key": "title",
        "replacement_type": "replace",
        "replacement_string": "database_service",
        "replacement_pattern": "^test_database_service$"
    }]
  },
  {
    "object_type": "entity",
    "title_list": ["10.12.*", "*host_database*"]
  },
  {
    "object_type": "deep_dive",
    "title_list": "^test_database_deep_dive$",
    "replacement_rules": [ {
        "replacement_key": "title",
        "replacement_type": "replace",
        "replacement_string": "database_deep_dive",
        "replacement_pattern": "^test_database_deep_dive$"
    }]
  }
]

Service KPI operations (mode 2)

kvstore_to_json.py mode 2 options let you run bulk operations on KPIs, including get (-g), create (-c), update (-t), and delete (-r). Use these options to replicate, edit, and copy KPIs to multiple services, for example, when moving your ITSI deployment from a test environment to a production environment.

All service KPI options require you to specify the mode 2 parameter -m 2. You must also specify the file path -f parameter as the full path to the directory containing the input.json file.

Before you can run service KPI operations, you must create an input.json file in the destination directory. The script accepts data input from input.json and sends data output to an output.json file that the script creates in the same directory.

All service KPI operations, except get -g, require you to specify service and/or KPI keys. You can retrieve these keys using the -g option in output.json.

See the kvstore_to_json.py help - h option for proper input.json and command syntax.

Get service and KPI keys

Use the get -g option to retrieve service and KPI data in JSON format, including service and KPI keys.

  1. Create an input.json file in the destination directory.
    mkdir <directory_containing_input.json>
    touch input.json
    
  2. Edit input.json: Add [] to the file to retrieve JSON data for all services and KPIs, or add specific service and kpi keys to the file to retrieve JSON data for those services and KPIs only. For example:
    [{"_key": "<service_key>", "kpis": [{"_key": "<kpi_key>"}] } ].
    
  3. Run the kvstore_to_json script using the get -g option. Name the full path to the directory containing the input.json file as the file path -f parameter. For example:
    cd $SPLUNK_HOME
    bin/splunk cmd python kvstore_to_json.py -u admin -p changeme -m 2 -g -f <directory_containing_input.json> -n
    
  4. Review the contents of output.json to identify service and KPI keys. For example:
    [
        {
            "_key": "669c5cec-a492-419d-8659-95a185b4dc5c", 
            "kpis": [
                {
                    "_key": "f017cc7b2e67f2b3b9152146", 
    ... 
    }
    ]
    }
    ]
    

Create KPIs

Use the -c option to create new KPIs.

  1. Edit input.json to specify the service key of the service for which you want to create the KPI.
  2. Add KPI keys for the KPIs that you want to add to the service and any key-value pairs belonging to the KPI that you want to include in the KPI definition. Leave the key field for each KPI empty for ITSI to auto generate it. For example:
    [
        {
            "_key": "<service_key>", 
            "kpis": [
                {
                    "title": "<title_of_kpi_to_create>",
    ... 
    }
    ]
    }
    ]
    

Update KPIs

Use the -t option to update KPIs.

In input.json specify the service and KPI key for each KPI, and any other key/value pair data that you want to update for the KPI.

Delete KPIs

Use the -r option to delete KPIs.

In input.json, specify service and kpi keys for all KPIs that you want to delete.

Caution: Make sure to properly validate your JSON input. While the kvstore_to_json script does provide some schema validation, incorrect JSON formatting can cause errors.

Time zone offset operations (mode 3)

The kvstore_to_json.py mode 3 option lets you apply a time offset for time-sensitive fields in object configurations. You can use this option to correct time zone discrepancies for the following object types:

  • maintenance_calendar: Sets an offset for maintenance window start and end times.
  • service: Sets an offset for the KPI threshold time policies within a service. See Apply time zone offsets in this manual.
  • kpi_threshold_template: Sets an offset for a KPI threshold template. After running the command to set an offset for a KPI threshold template, you must run the command again for each service that uses the KPI threshold template and set the same offset so that they are in sync.

Apply time zone offset

Run the following command to set an offset for one of the supported object types.

If you use the command to set an offset for a KPI threshold template, you must run the command again for each service that uses the KPI threshold template and set the same offset so that they are in sync.

  1. Run kvstore_to_json.py, where -o is the object type, -k is the title of the specific object, and -z is the specific time zone offset in seconds. For example:
    cd $SPLUNK_HOME
    bin/splunk cmd python etc/apps/SA-ITOA/bin/kvstore_to_json.py -m 3 -o service -k "Database Service" -z 1800
    
  2. Enter the requested information at the prompts (default interactive mode only). For example:
    >> Enter the splunkd port number OR press the enter key to use [8089] > 8089
    >> Enter splunk username or press enter to use "admin" > admin
    >> Enter splunk password for "admin" >
    
  3. The script applies the time zone offset to the specified object. For example:
    1 object(s) match request
    Applying timezone change on requested object(s): [u'Database Service']
    Timezone offset has been applied on the objects requested.
    

ITSI time-sensitive configurations are normalized to UTC.

Regenerate KPI search schedules (mode 4)

The kvstore_to_json.py mode 4 option regenerates the search schedules for your KPIs. Use this command if you have set your KPI saved search schedules to run at the same time in itsi_settings.conf. Run this command to reset the search schedules of all your KPIs to use the new search schedule. See Schedule KPI searches to update at the same time for more information.

  1. Run kvstore_to_json.py in mode 4.
    For example:
    cd $SPLUNK_HOME
    bin/splunk cmd python etc/apps/SA-ITOA/bin/kvstore_to_json.py -m 4
    
  2. Enter the requested information at the prompts (default interactive mode only).
  3. You will see the following message after the KPI search schedules have been reset:
    Retrieving KPIs to reset their saved search scheduling
    Saving updated KPI scheduling
    Done.
    
PREVIOUS
Back up and restore ITSI KV store data
  NEXT
Create and manage saved searches in ITSI

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.3.0, 4.3.1


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters