Splunk® Supported Add-ons

Splunk Add-on for ServiceNow

Download manual as PDF

Download topic as PDF

Use custom alert actions for the Splunk Add-on for ServiceNow

Use the Splunk Add-on for ServiceNow to create custom alert actions that automatically create incidents and events or update existing incidents. Custom alert actions are available in Splunk platform version 6.3.0 and later. Custom alert actions are a user-friendly implementation of the alert-triggered scripts available in previous versions.

Before you can use the custom alert actions, see configure ServiceNow to integrate with the Splunk platform.

Depending on the search that you save as an alert, the custom alert action might create multiple events or incidents in ServiceNow. This can occur if the search string that you save as an alert returns multiple events. The number of events returned by the search equals the number of incidents or events created in ServiceNow.

Create an incident or event from a custom alert action

You can create an incident or event from a custom alert action.

  1. Write a search string to trigger incident or event creation in ServiceNow.
  2. Click Save As > Alert.
  3. Fill out the Alert form. Give your alert a unique name and indicate whether the alert is a real-time alert or a scheduled alert. See Getting started with Alerts in the Alerting Manual.
  4. Under Trigger Actions, click Add Actions.
  5. From the list, select ServiceNow Event Integration if you want the alert to create an event in ServiceNow, or ServiceNow Incident Integration if you want to create an incident in ServiceNow.
  6. Enter values for the fields to specify parameters for your event or incident. See About the commands, alert actions, and scripts available with the Splunk Add-on for ServiceNow for an explanation of the fields.

    If you are creating an incident, note that the behavior for the Correlation ID field is slightly different in a custom alert action than it is in the commands and scripts. This variation supports the ability to update incidents using the correlation ID in subsequent custom alert actions. In a custom alert action, if you leave this value blank, the Splunk platform does not generate a random UUID, but generates a correlation ID based on a MD5 hash of your alert name and the app name. Ensure that you give each alert using a custom alert action a unique name across your Splunk deployment.

  7. Click Save.

Update an incident from a custom alert action

To update a previously-created incident from a custom alert action follow steps 1 - 5 in the previous section, "Create an incident or event from a custom alert action," exactly as if you were creating a new incident. Then, when you enter the values for the ServiceNow Incident Integration under Trigger Actions, include the Correlation ID of the incident that you want to update.

Examples

These examples demonstrate how to create custom alert actions that create and update incidents in ServiceNow.

ServiceNow Incident Integration alert action that creates a new incident

This example creates an incident in ServiceNow from a custom alert action when an employee who is about to leave the company downloads data from the company's Salesforce instance.

  1. Write a search that triggers the alert action when it returns events. For this example, the search is:

    (sourcetype="snow:hr_change" category = offboarding closed_at="") OR (sourcetype="sfdc:logfile" EVENT_TYPE=ReportExport) | transaction Email maxspan=1d | where sourcetype="snow:hr_change" AND sourcetype="sfdc:logfile"

  2. Run the search over a time period that makes sense for your use case. In this example, the time period could be Last 24 hours.
  3. Click Save As > Alert.
  4. Give your alert the title Data Leak Alert.
  5. For this use case, schedule the alert to Run every day to sweep for the condition once per day. Leave the default set to Number of Results is greater than 0.
  6. Set the Trigger to For each result, because you want a separate alert to fire for each search result.
  7. Under Trigger Actions, select ServiceNow Incident Integration.
  8. Enter values for all required fields, as shown.
    Field Example Value
    State 1
    Configuration Item Leave blank. Does not apply in this example.
    Contact Type Phone
    Assignment Group Security
    Incident Category Security
    Incident Subcategory Data leak
    Short Description Data Leak: off-boarded user $result.Name$ with email $result.Email$ is exporting data from SFDC.
    Correlation ID DataLeakAlert.$result.Email$
    Account Select an account from the drop-down menu, or use the Create new account link to configure a new account.
  9. Click Save.

In this example, each fired alert creates an incident with an account name and correlation ID that is unique to the email and the alert that you have created. If you want to update this incident using another custom alert action, you can supply the same account name and correlation ID to find and update the existing incident.

You can create other custom alert actions using the $result.Email$ value and some other concatenated unique strings so that you can handle unlimited incidents based on the same parameters but different alert conditions.

ServiceNow Incident Integration alert action that updates the above incident

If you want to update incidents that you created using the example above, you can do so using another custom alert action. In this example, when someone has closed the HR change ticket for that employee in ServiceNow, this alert closes the corresponding incident regarding the Salesforce data download.

  1. Write a search to find the same events again, filtering to those with a closed_at field that is not null.

    (sourcetype="snow:hr_change" category = offboarding closed_at!="") OR (sourcetype="sfdc:logfile" EVENT_TYPE=ReportExport) | transaction Email maxspan=1d | where sourcetype="snow:hr_change" AND sourcetype="sfdc:logfile"

  2. Search over the last 24 hours.
  3. Save As > Alert.
  4. Give your alert the title Resolved Data Leak Alert.
  5. Schedule the alert to run once per day. Leave the default set to Number of Results is greater than 0.
  6. Set the alert to trigger For each result.
  7. Under Trigger Actions, select ServiceNow Incident Integration.
  8. Enter values for all required fields, as shown. The only values that are different from the alert in the previous example are the State and the Short Description.
    Field Example Value
    State 7
    Configuration Item (Leave blank. Does not apply in this example.)
    Contact Type Phone
    Assignment Group Security
    Incident Category Security
    Incident Subcategory Data leak
    Short Description Resolved Data Leak: off-boarded user $result.Name$ with email $result.Email$ is exporting data from SFDC.
    Correlation ID DataLeakAlert.$result.Email$
    Account Select the previously entered account
  9. Click Save.

When ServiceNow finds an existing incident with an account name and correlation ID that matches this value, it updates the existing incident with the new State and Short Description. If ServiceNow cannot find an incident matching this correlation ID, it creates an incident.

ServiceNow Event Integration alert action that creates a new event

This example creates an event in ServiceNow from a custom alert action when CPU usage exceeds 90% on any host.

  1. Create a search that triggers the alert action when it returns events. For this example, the search is:

    sourcetype="linux:server:metrics" | stats avg(cpu) by host | rename avg(cpu) AS cpu_utilization | where cpu_utilization>90

  2. Run the search over the last 24 hours.
  3. Click Save As > Alert.
  4. Give your alert the title High CPU on Linux Server.
  5. Schedule your alert to Run every day, because you want to sweep for this condition every hour.
  6. Set the Trigger to For each result, because you want a separate alert to fire for each search result.
  7. Under Trigger Actions, select ServiceNow Event Integration.
  8. Enter values for required fields, as shown.
    Field Example Value
    Node $result.host$
    Type MySQL Server
    Resource CPU
    Severity 1
    Description CPU usage is very high on $result.host$.
    Account Select an account from the drop-down menu, or use the Create new account link to configure a new account
  9. Click Save.

In this example, for a given account, each alert fired by the Splunk software triggers a separate event in ServiceNow. However, ServiceNow combines multiple events with the same values for Node, Resource, and Type fields under the same ServiceNow Alert. Subsequent firings of the same alert with the same parameters are part of the same ServiceNow Alert.

Unless you modified the Alert rules in ServiceNow, the creation of a Critical event automatically triggers the creation of a new incident. In this example, for a given account, when the conditions are met and the alert fires, ServiceNow triggers the creation of a matching incident. Subsequent firings of this alert do not create additional incidents because they are associated with the same ServiceNow Alert. Use ServiceNow's Event Management functionality of the given account to close this incident using a clear event (severity 0) with a matching Node, Resource, and Type.

PREVIOUS
Use custom generating search commands for the Splunk Add-on for ServiceNow
  NEXT
Use alert-triggered scripts for the Splunk Add-on for ServiceNow

This documentation applies to the following versions of Splunk® Supported Add-ons: released


Comments

In order for an incident or event to yield >1 record in Service Now, you must change the alert's Trigger Conditions to "For each result."

Yorokobi
September 16, 2019

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters