KV Store integration for custom alert actions

Integrate custom alert actions with the KV Store

Integrate custom alert actions with the KV Store to track state and implement complex workflows. Here are some example use cases for KV Store integration.

• Alert queue for review and approval. To defer immediate alert actions, use the KV Store as a queue for alert action requests. Send alert action parameters, metadata, or an invocation string to the KV Store. Admin or other authorized users can review and approve queued alert action requests.
  

• Alert action throttling. Use the KV Store to track and retrieve state, such as most recent alert actions or an alert action count. An alert action script with custom throttling logic can use state information to suppress or run alert actions.
  

• Logic to create and update service tickets. Use a custom alert action script to create or update service tickets when an alert triggers. The script can log alerts and ticket information in the KV Store. When a new alert triggers, the script can check the KV Store for ticket history on similar alerts. If a ticket already exists for an alert with similar properties, then the script can update the ticket. If no ticket exists, the script can file a new one.

Example code

Here is a code selection from a KV Store custom alert action script. The example app updates one field in a KV Store record. This script has been made cross-compatible with Python 2 and Python 3 using python-future.

from __future__ import print_function
from future import standard_library
standard_library.install_aliases()

import sys
import json
import urllib.request, urllib.parse, urllib.error
import urllib.request, urllib.error, urllib.parse

def request(method, url, data, headers):
    """Helper function to fetch JSON data from the given URL"""
    req = urllib.request.Request(url, data, headers)
    req.get_method = lambda: method
    res = urllib.request.urlopen(req)
    return json.loads(res.read())

payload = json.loads(sys.stdin.read())

config = payload.get('configuration', dict())
collection = config.get('collection')
record_name = config.get('name')
field = config.get('field')
value = config.get('value')

# Build the URL for the Splunkd REST endpoint
url_tmpl = '%(server_uri)s/servicesNS/%(owner)s/%(app)s/storage/collections/data/%(collection)s/%(name)s?output_mode=json'
record_url = url_tmpl % dict(
    server_uri=payload.get('server_uri'),
    owner='nobody',
    app=urllib.parse.quote(config.get('app') if 'app' in config else payload.get('app')),
    collection=urllib.parse.quote(collection),
    name=urllib.parse.quote(record_name))
print('DEBUG Built kvstore record url=%s' % record_url, file=sys.stderr)
headers = {
    'Authorization': 'Splunk %s' % payload.get('session_key'),
    'Content-Type': 'application/json'}

# Fetch the record from the kvstore collection
try:
    record = request('GET', record_url, None, headers)
    print("DEBUG Retrieved record:", json.dumps(record), file=sys.stderr)
except urllib.error.HTTPError as e:
    print('ERROR Failed to fetch record at url=%s. Server response: %s' % (
        record_url, json.dumps(json.loads(e.read()))), file=sys.stderr)
    sys.exit(2)

# Update the record with the user supplied field value
data = {field: value}
record.update(data)

print('INFO Updating kvstore record=%s in collection=%s with data=%s' % (
    record_name, collection, json.dumps(data)), file=sys.stderr)

# Send the updated record to the server
try:
    response = request('POST', record_url, json.dumps(record), headers)
    print('DEBUG server response:', json.dumps(response), file=sys.stderr)
except urllib.error.HTTPError as e:
    print('ERROR Failed to update record:', json.dumps(json.loads(e.read())), file=sys.stderr)
    sys.exit(3)