Use custom alert actions for the Splunk Add-on for ServiceNow

Use the Splunk Add-on for ServiceNow to create custom alert actions that automatically create incidents and events or update existing incidents. Custom alert actions are available in Splunk platform version 6.3.0 and later. Custom alert actions are a user-friendly implementation of the alert-triggered scripts available in previous versions.

Before you can use the custom alert actions, see configure ServiceNow to integrate with the Splunk platform.

Depending on the search that you save as an alert, the custom alert action might create multiple events or incidents in ServiceNow. This can occur if the search string that you save as an alert returns multiple events. The number of events returned by the search equals the number of incidents or events created in ServiceNow.

Splunk Import set

The #Splunk Import set is the Splunk Integration plugin found in the ServiceNow store that you can install onto your ServiceNow instance. You can use this plugin to manage, insert or update an incident on the Incident table. If the correlation ID exists on the Incident table, the plugin would update the incident. If the correlation ID doesn't exist, the plugin would create a new incident.

ServiceNow Incident table

The #ServiceNow Incident table is where you can see all created or updated incidents. This is a default table provided by ServiceNow.

Transform map in ServiceNow

The #Transform map in ServiceNow is the mapping between the Splunk Integration Plugin and the ServiceNow Incident table. This mapping is active once you install the Splunk integration plugin onto the ServiceNow instance.

Create an incident or event from a custom alert action

You can create an incident or event from a custom alert action.

1. Write a search string to trigger incident or event creation in ServiceNow.
2. Click Save As > Alert.
3. Fill out the Alert form. Give your alert a unique name and indicate whether the alert is a real-time alert or a scheduled alert. See Getting started with Alerts in the Alerting Manual.

   During incident creation, to use the custom fields in the alert action, the user has to pass all the custom fields separated by the || sign. For example, u_affected_user=nobody||u_caller_id=12345. Parameters passed in custom fields must be configured at the ServiceNow incident table, Splunk import set (x_splu2_splunk_ser_u_splunk_incident) table and appropriate transform map in ServiceNow. For example, u_affected_user and u_caller_id must be present in your ServiceNow incident table. It must be present in the intermediate Splunk Import Set table (x_splu2_splunk_ser_u_splunk_incident table) and must be mapped in the appropriate transform map.

4. Under Trigger Actions, click Add Actions.
5. From the list, select ServiceNow Event Integration if you want the alert to create an event in ServiceNow, or ServiceNow Incident Integration if you want to create an incident in ServiceNow.
6. Enter values for the fields to specify parameters for your event or incident. See About the commands, alert actions, and scripts available with the Splunk Add-on for ServiceNow for an explanation of the fields. Give each alert that uses a custom alert action a unique name across your Splunk deployment.

   If you are creating an incident, note that the behavior for the Correlation ID field is slightly different in a custom alert action than it is in the commands and scripts. This variation supports the ability to update incidents using the correlation ID in subsequent custom alert actions. In a custom alert action, if you leave this value blank, the Splunk software does not generate a random UUID, but generates a correlation ID based on a MD5 hash of your alert name, owner and the app name. Ensure that you give each alert using a custom alert action a unique name across your Splunk platform deployment.

7. Click Save.

Update an incident from a custom alert action

To update a previously-created incident from a custom alert action follow steps 1 - 5 in the previous section, "Create an incident or event from a custom alert action," exactly as if you were creating a new incident. Then, when you enter the values for the ServiceNow Incident Integration under Trigger Actions, include the Correlation ID of the incident that you want to update.

Examples

These examples demonstrate how to create custom alert actions that create and update incidents in ServiceNow.

ServiceNow Incident Integration alert action that creates a new incident

This example creates an incident in ServiceNow from a custom alert action when an employee who is about to leave the company downloads data from the company's Salesforce instance.

1. Write a search that triggers the alert action when it returns events. For this example, the search is:

   (sourcetype="snow:hr_change" category = offboarding closed_at="") OR (sourcetype="sfdc:logfile" EVENT_TYPE=ReportExport) | transaction Email maxspan=1d | where sourcetype="snow:hr_change" AND sourcetype="sfdc:logfile"

2. Run the search over a time period that makes sense for your use case. In this example, the time period could be Last 24 hours.
3. Click Save As > Alert.
4. Give your alert the title Data Leak Alert.
5. For this use case, schedule the alert to Run every day to sweep for the condition once per day. Leave the default set to Number of Results is greater than 0.
6. Set the Trigger to For each result, because you want a separate alert to fire for each search result.
7. Under Trigger Actions, select ServiceNow Incident Integration.
8. Enter values for the following fields, as shown.

   Field	Example Value
   State	1
   Configuration Item	Leave blank. Does not apply in this example.
   Contact Type	Phone
   Assignment Group	Security
   Incident Category	Security
   Incident Subcategory	Data leak
   Impact	2 (Enter the numerical value as defined in Supported Arguments for Incidents)
   Urgency	2 (Enter the numerical value as defined in Supported Arguments for Incidents)
   Priority	4 (Enter the numerical value as defined in Supported Arguments for Incidents) Priority value passed in the alert action will be set in incidents only if there is no business rule configured on your ServiceNow instance for Incident table to calculate priority. By default, priority is calculated based on the "impact" and "urgency" values, and hence in such cases "Priority" value sent from alert action will not be reflected.
   Short Description	Data Leak: off-boarded user $result.Name$ with email $result.Email$ is exporting data from SFDC.
   Splunk URL	https://<splunk_server>: port
   Custom Fields	u_affected_user=nobody||u_caller_id=12345
   Correlation ID	A unique ID to support third-party application integration. It should only consist of alphanumeric characters, underscore (_), and hyphen (-) in its value. Leave blank for the Splunk Add-on for ServiceNow to generate a unique ID for you.
   Account	Select an account from the drop-down menu, or use the Create new account link to configure a new account.

9. Click Save.

In this example, each fired alert creates an incident with an account name and Correlation ID that is unique to the email and the alert that you have created. If you want to update this incident using another custom alert action, you can supply the same account name and correlation ID to find and update the existing incident.

You can create other custom alert actions using the $result.Email$ value and some other concatenated unique strings so that you can handle unlimited incidents based on the same parameters but different alert conditions.

ServiceNow Incident Integration alert action that updates the above incident

To update incidents that you created using the example above, create another custom alert action. In this example, when someone has closed the HR change ticket for that employee in ServiceNow, this alert closes the corresponding incident regarding the Salesforce data download.

1. Write a search to find the same events again, filtering to those with a closed_at field that is not null.

   (sourcetype="snow:hr_change" category = offboarding closed_at!="") OR (sourcetype="sfdc:logfile" EVENT_TYPE=ReportExport) | transaction Email maxspan=1d | where sourcetype="snow:hr_change" AND sourcetype="sfdc:logfile"

2. Search over the last 24 hours.
3. Save As > Alert.
4. Give your alert the title Resolved Data Leak Alert.
5. Schedule the alert to run once per day. Leave the default set to Number of Results is greater than 0.
6. Set the alert to trigger For each result.
7. Under Trigger Actions, select ServiceNow Incident Integration.
8. Enter values for the following fields, as shown. The only values that are different from the alert in the previous example are the State and the Short Description.

   Field	Example Value
   State	7
   Configuration Item	(Leave blank. Does not apply in this example.)
   Contact Type	Phone
   Assignment Group	Security
   Incident Category	Security
   Incident Subcategory	Data leak
   Impact	2 (Enter the numerical value as defined in Supported Arguments for Incidents)
   Urgency	2 (Enter the numerical value as defined in Supported Arguments for Incidents)
   Priority	4 (Enter the numerical value as defined in Supported Arguments for Incidents) Priority value passed in the alert action will be set in incidents only if there is no business rule configured on your ServiceNow instance for Incident table to calculate priority. By default, priority is calculated based on the "impact" and "urgency" values, and hence in such cases "Priority" value sent from alert action will not be reflected.
   Short Description	Resolved Data Leak: off-boarded user $result.Name$ with email $result.Email$ is exporting data from SFDC.
   Splunk URL	https://<splunk_server>: port
   Custom Fields	u_affected_user=nobody||u_caller_id=12345
   Correlation ID	A unique ID to support third-party application integration. It should only consist of alphanumeric characters, underscore (_), and hyphen (-) in its value. Leave blank for the Splunk Add-on for ServiceNow to generate a unique ID for you.
   Account	Select the previously entered account

9. Click Save.

When ServiceNow finds an existing incident with an account name and correlation ID that matches this value, it updates the existing incident with the new State and Short Description. If ServiceNow cannot find an incident matching this correlation ID, it creates an incident.

ServiceNow Event Integration alert action that creates a new event

This example creates an event in ServiceNow from a custom alert action when CPU usage exceeds 90% on any host.

1. Create a search that triggers the alert action when it returns events. For this example, the search is:

   sourcetype="linux:server:metrics" | stats avg(cpu) by host | rename avg(cpu) AS cpu_utilization | where cpu_utilization>90

2. Run the search over the last 24 hours.
3. Click Save As > Alert.
4. Give your alert the title High CPU on Linux Server.
5. Schedule your alert to Run every day, because you want to sweep for this condition every hour.
6. Set the Trigger to For each result, because you want a separate alert to fire for each search result.
7. Under Trigger Actions, select ServiceNow Event Integration.
8. Enter values for the following fields, as shown.

   Field	Example Value
   Enter ServiceNow Account(s)	Enter the locally configured ServiceNow accounts in the add-on comma-delimited. Each ServiceNow account that you enter should be one that is configured in the add-on on the same machine as the one from where alert actions will be triggered. For example, accounts configured in the Splunk Add-on for ServiceNow on your search head should be used, and not an account that is configured your heavy forwarders or inputs data manager (IDM).
   Node	$result.host$
   Type	MySQL Server
   Resource	CPU
   Severity	1
   CI identifier	k1=v1||k2=v2
   Description	CPU usage is very high on $result.host$.
   Custom Fields	<code>u_affected_user=nobody||u_caller_id=12345</code>

9. Click Save.

In this example, for a given account, each alert fired by the Splunk software triggers a separate event in ServiceNow. However, ServiceNow combines multiple events with the same values for Node, Resource, and Type fields under the same ServiceNow Alert. Subsequent firings of the same alert with the same parameters are part of the same ServiceNow Alert.

Unless you modified the Alert rules in ServiceNow, the creation of a Critical event automatically triggers the creation of a new incident. In this example, for a given account, when the conditions are met and the alert fires, ServiceNow triggers the creation of a matching incident. Subsequent firings of this alert do not create additional incidents because they are associated with the same ServiceNow Alert. Use ServiceNow's Event Management functionality of the given account to close this incident using a clear event (severity 0) with a matching Node, Resource, and Type.