Use alert-triggered scripts for the Splunk Add-on for ServiceNow

The Splunk Add-on for ServiceNow supports automatic incident and event creation and incident update from scripts triggered by alerts.

Before you can use these scripts, see configure ServiceNow to integrate with the Splunk platform.

Depending on the search that you run, alert-triggered scripts can create multiple events or incidents in ServiceNow. This can occur when the search string in the alert logic returns multiple events. The number of events returned by a search in your alert equals the number of incidents or events created in ServiceNow.

Scripted Alert is now a deprecated Splunk feature. The best practice is to use custom alert actions instead.

Create an incident or event from an alert using the snow_incident.py or snow_event.py script

Create an incident or event based on an alert.

1. In Splunk Web, click Settings > Searches, Reports, and Alerts.
2. Click New.
3. Set the Destination app to Splunk Add-on for ServiceNow (Splunk_TA_snow).
4. Enter a Search name that describes the alert you want to create.
5. Enter a Search that meets the following criteria:
   • To create an incident, the search must include the mandatory arguments account. account is required to identify the ServiceNow instance on which incident is to be created. The Splunk platform passes the arguments to the alert result to trigger the script.

   To use the custom fields in the snowincident command, you have to pass || separated key-value pairs of custom fields and values as part of --custom_fields argument in the command. All the parameters which are passed in the --custom_fields are needed to be configured first in the ServiceNow instance. Parameters passed in custom fields must be configured at ServiceNow Incident table, Splunk import set (x_splu2_splunk_ser_u_splunk_incident) table and appropriate transform map in ServiceNow. For example, u_affected_user and u_caller_id must be present in your ServiceNow incident table. It must be present in intermediate Splunk Import Set table (x_splu2_splunk_ser_u_splunk_incident table) and must be mapped in the appropriate transform map.

   • To create an event, the search must include the mandatory arguments account, node, resource, type, and severity. account is required to identify the ServiceNow instance on which event is to be created. The other arguments are required by ServiceNow to create an event. The Splunk platform passes the arguments to the alert result to trigger the script.
     • account is required by ServiceNow to create an event.
     • node is required by ServiceNow to create an event.
     • resource is required by ServiceNow to create an event.
     • type is required by ServiceNow to create an event.
     • severity is required by ServiceNow to create an event.
     • account is required to identify the ServiceNow instance on which an event is to be created.
     • custom fields are a list of optional fields in a key-value format that you can specify while creating a ServiceNow event.
   • The search can include any of the optional arguments supported by ServiceNow incident or event creation. See About the commands and scripts for a table detailing each of these arguments.
   • The search must be in tabular format.

   
   The following search is an example that demonstrates how to trigger the script to create an incident when CPU usage is 95 or higher.

   sourcetype="CPURates" earliest=-5m latest=now
   | stats avg(CPU) as CPU last(_time) as time by host
   | where CPU>=95 | eval account="user"
   | eval contact_type="email"
   | eval custom_fields="u_affected_user=nobody||u_caller_id=12345"
   | eval ci_identifier=host
   | eval priority=1 | eval category="Software"
   | eval subcategory="database"
   | eval short_description="CPU on ". host ." is at ". CPU
   | table account, category, subcategory, short_description, contact_type, custom_fields, ci_identifier, priority
   

   
   The following search is an example that demonstrates how to trigger the script to create an event when CPU usage is 95 or higher:

   sourcetype="CPURates" earliest=-5m latest=now
   | stats avg(CPU) as CPU last(_time) as time by host
   | where CPU>=95 | eval account="user"
   | eval node=host
   | eval custom_fields="u_affected_user=nobody||u_caller_id=12345"
   | eval resource="CPU"
   | eval type="CPUAlert" | eval severity="2"
   | eval description="CPU on ". host ." is at ". CPU.""
   | table account, node, resource, description, type, custom_fields, severity
   

6. Under Schedule and alert, click Schedule this search.
7. Select values for Schedule type, Run every, Expiration, and Severity according to your alert requirements.
8. Under Alert actions, check the box next to Enable under Run a script.
9. Enter the name of the script in File name of shell script to run.
   • For an incident, enter snow_incident.py
   • For an event, enter snow_event.py
10. Click Save.

Update an incident from an alert using the snow_incident.py script

The procedure to update an incident from an alert is the same procedure to create an incident from an alert.

account is required, and is used to identify the ServiceNow instance on which incident is to be updated. The argument correlation_id is required to update an existing incident.

You can update incident using an alert-triggered script only if you created the incident from the Splunk platform. You cannot update incidents created in ServiceNow.

The following example search demonstrates how you can trigger the script to update a previously-created incident when CPU usage drops below 15.

sourcetype="CPURates" earliest=-5m latest=now
| stats avg(CPU) as CPU last(_time) as time by host
| where CPU<15 | eval contact_type="email"
| eval ci_identifier=host
| eval state="7"
| eval correlation_id="23c24b34-c98b-4da1-997e-9992edb80863"
| eval category="Software"
| eval subcategory="database"
| eval short_description="CPU on ". host ." is at ". CPU ""
| eval custom_fields="u_affected_user=nobody||u_caller_id=12345"
| eval account="user1"
| table account, category, subcategory, short_description, ciIdentifier, host, custom_fields