Take action on an episode in ITSI

ITSI provides the following actions that you can run on episodes:

• Share the episode
• Link a ticket
• Ping a host
• Run a custom script
• Send an email
• Create a ticket in an external ticketing system

ITSI also provides a python-based notable event actions SDK that lets you build post-action, state-changing capability into episode actions. You can use this SDK to enable execution of additional tasks for episode actions, such as creating a ticket in a third-party system. To create your own custom episode actions, see Set up custom episode actions below.

Not all actions are available if role-based permissions are set. All episode actions are Splunk platform alert actions that you can manage in the Alert Actions manager. For more information, see Using the alert actions manager in the Alerting Manual. Permissions can be set per user role for each episode action. These permissions also determine which actions are available in a notable event aggregation policy.

Share episode

Generate a URL that links to a filtered view of Episode Review. For example, you might want to link directly to the "Events Timeline" tab within a specific episode. Generate a custom link to that episode that you can save, send, or bookmark.

1. Select an episode
2. (Optional) Select a specific tab within the episode.
3. Click Actions > Share episode.
4. Copy the link.

Link a ticket

You can associate a ticket from your internal ticketing system to an episode. For example, you might see an episode related to a disk failure in Episode Review and remember that a ticket has been created for this issue in your internal ticketing system. You can add the ticket information to the episode. You can quickly access the ticket in the future to review information on the status and progress of investigation into the episode.

1. Select an episode.
2. Click Actions > Link ticket.
3. Configure the ticket information and click Done.
4. Click the Activity tab to confirm that the ticket was linked.
5. Click the Impact tab to see a link to the ticket under All Tickets. If you linked a ticket to an episode, the ticket is linked to each notable event in the episode.

Display a ticket column

Add a new column in Episode Review to display linked tickets for episodes

1. Click the gear icon .
2. Click Add Column and select All Tickets.
3. Click Done.

Ping a host

Determine whether a host is still active on the network by pinging the host.

1. Select an episode.
2. Click Actions > Ping host.
3. Type the event field that contains the host that you want to ping in the Host field. For example, %server%.
4. Click Done.

Run a script

The run a script functionality is officially deprecated and will be removed in a future release. It will be replaced with custom alert actions as a more scalable and robust framework for integrating custom actions. To learn how to migrate existing alert action scripts to the custom alert action framework, see Convert a script alert action to a custom alert action in the Splunk Enterprise Developing Views and Apps for Splunk Web manual.

You can run an alert script on an IT Service Intelligence (ITSI) episode. For example, you can configure an alert to run a script that generates a Simple Network Management Protocol (SNMP) trap notification. The script sends the notification to another system such as a Network Systems Management console. You can configure a different alert that runs a script that calls an API, which in turn sends the triggering event to another system.

For security reasons, place all scripts in $SPLUNK_HOME/bin/scripts or $SPLUNK_HOME/etc/SA-ITOA/bin/scripts.

1. Select an episode.
2. Click Actions > Run a script.
3. Type the file name of the script.
4. Click Done.

For more information about scripted alerts, see the following topics in the Splunk Enterprise documentation:

• For Splunk Enterprise, see Configure scripted alerts in the Splunk Enterprise Alerting Manual.
• For Splunk Cloud, see Configure scripted alerts in the Splunk Cloud Alerting Manual.

Send an email

Send an email as a result of an episode. ITSI sends one email even though there are multiple events in the episode.

You can use tokens in the email subject or message. The tokens are replaced with field values in the email message. You can use the following fields which are available from the episode:

• owner
• severity
• status
• title
• description
• start_time
• last_time
• is_active
• event_count

You can also use fields that are contained in the last event in the episode. If a field is not present in the last event of the episode (although it may exist in other events in the episode), the token will not be replaced with the field value in the email message.

Prerequisite

Make sure that the mail server is configured in the Splunk platform before performing this action.

• For Splunk Enterprise, see Configure email notification settings in the Splunk Enterprise Alerting Manual.
• For Splunk Cloud, see Configure email notification settings in the Splunk Cloud Alerting Manual.

Steps

1. Select an episode.
2. Click Actions > Send email.
3. In the To field, type a comma-separated list of email addresses to send the email to.
4. (Optional) Change the priority of the email. Defaults to Lowest.
5. Type a subject for the email. The subject defaults to "Splunk Results". You can include tokens that insert text based on the results of the search using the format $result.<fieldname>$. For example: $result.title$
6. Type a message to include as the body of the email. You can include tokens that insert text based on the results of the search using the format $result.<fieldname>$. For example: $result.event_id$
7. Select whether to send an HTML & plain text, or just a plain text email message.
8. Click Done.

Create a ticket in ServiceNow or Remedy

You can create a ticket in a ServiceNow or Remedy incident tracking system for an episode. After you install and configure the corresponding add-on on your Splunk platform, an option to create a ticket in that system appears in Episode Review Actions menu.

You can also create a ticket in an external ticketing system as an action to take on an episode in the Action Rules section of a notable event aggregation policy.

Requirements for Remedy

• BMC Remedy version 9.1 or later.
• Splunk Add-on for Remedy 2.0 or later must be installed and configured. See the Splunk Add-on for BMC Remedy manual for instructions.
• The user creating the incident must be assigned a role of admin, itoa_admin, or itoa_analyst.

An itoa_user role can create incidents if the following capabilities have been assigned:

• execute-notable_event_action = enabled
• write-notable_event = enabled
• delete-notable_event = enabled
• delete_by_keyword = enabled
• deleteIndexesAllowed = itsi_tracked_alerts

For information about ITSI roles and capabilities, see Configure users and roles in ITSI in the ITSI Installation and Configuration Manual.

A role must also have the Splunk platform list_storage_passwords capability to create incidents in Remedy. This capability needs to be assigned to custom roles. The capability is assigned to itoa_user and itoa_analyst in Splunk platform version 6.5 and later (and is not necessary for the itoa_admin and itoa_team_admin roles). For information about Splunk platform capabilities, see About defining roles with capabilities in the Securing Splunk Enterprise manual.

Requirements for ServiceNow

• ServiceNow version Fuji or later.
• Splunk Add-on for ServiceNow 2.9 or later must be installed and configured. See the Splunk Add-on for ServiceNow manual for instructions.

The user creating the incident must be assigned a role of admin, itoa_admin, or itoa_analyst.

An itoa_user role can create incidents if the following capabilities have been assigned:

• execute-notable_event_action = enabled
• write-notable_event = enabled
• delete-notable_event = enabled
• delete_by_keyword = enabled
• deleteIndexesAllowed = itsi_tracked_alerts

For information about ITSI roles and capabilities, see Configure users and roles in ITSI in the ITSI Installation and Configuration Manual.

A role must also have the Splunk platform list_storage_passwords capability to create incidents in ServiceNow. This capability needs to be assigned to custom roles. The capability is assigned to itoa_user and itoa_analyst in Splunk platform version 6.5 and later (and is not necessary for the itoa_admin and itoa_team_admin roles). For information about Splunk platform capabilities, see About defining roles with capabilities in the Securing Splunk Enterprise manual.

Steps

1. Select an episode.
2. Click Actions > Remedy Incident Integration or ServiceNow Incident Integration.
3. Configure the fields corresponding to fields in your incident tracking system.
   • Do not enter a Correlation ID for either Remedy or ServiceNow (even though this field is marked as required for Remedy). ITSI takes care of associating the episode with the external ticket for you.
4. Click Done. After a few seconds the following message appears: "Successfully dispatched actions. View in Activity".
5. Click View in Activity to see one or more entries related to the external ticketing system.
6. Go to the Impact tab to see the incident number listed under All Tickets. Click this link to open the ticket in your ticketing system.

When you create a ticket in ServiceNow, the name that appears in the "Opened by" field for the incident is the name of the Splunk user that configured the Splunk Add-on for ServiceNow, no matter which Splunk user creates the ticket in ITSI.

Create a ticket in VictorOps

You can create an incident in a VictorOps incident management system for an episode. After you install and configure the VictorOps add-on on your Splunk platform, an option to create an incident in VictorOps appears in the Episode Review Actions menu.

Prerequisites

• VictorOps version 1.16 or later.
• VictorOps for Splunk version 1.0.8 or later must be installed and configured. See the Splunk Integration Guide for VictorOps for instructions.
• The user creating the incident must be assigned a role of admin, itoa_admin, or itoa_analyst.

An itoa_user role can create incidents if the following capabilities have been assigned:

• execute-notable_event_action = enabled
• write-notable_event = enabled
• delete-notable_event = enabled
• delete_by_keyword = enabled
• deleteIndexesAllowed = itsi_tracked_alerts

For information about ITSI roles and capabilities, see Configure users and roles in ITSI in the ITSI Installation and Configuration Manual.

A role must also have the Splunk platform list_storage_passwords capability to create incidents in VictorOps. This capability needs to be assigned to custom roles. The capability is assigned to itoa_user and itoa_analyst in Splunk platform version 6.5 and later (and is not necessary for the itoa_admin and itoa_team_admin roles). For information about Splunk platform capabilities, see About defining roles with capabilities in the Securing Splunk Enterprise manual.

Steps

1. Select an episode.
2. Click Actions > VictorOps.
3. Configure the following fields:

   Field	Description
   Message Type	INFO - creates an alert WARNING - creates an alert CRITICAL - creates an incident ACKNOWLEDGEMENT - acknowledges the incident RECOVERY - resolves the incident
   Monitoring Tool	The VictorOps monitoring tool. Set this field to Splunk ITSI so that the incident and alert are branded with the Splunk ITSI logo.
   Alert Entity ID	The unique identifier for an incident. It is best practice to use a token to insert the value of a field. For example, you could use ITSI Alert: $result.itsi_group_title$.
   Alert Entity Display Name	The title of the incident. If you do not provide a display name, ITSI uses the Entity ID field.
   State Message	The status message to send to VictorOps.
   Routing Key	Optionally, configure a routing key to override the global VictorOps routing key.

4. Click Done. After a few seconds the following message appears: "Successfully dispatched actions. View in Activity".
5. Click View in Activity to see one or more entries related to VictorOps.

To set the above fields to reasonable defaults, create a local version of alert_actions.conf in $SPLUNK_HOME/etc/apps/victorops_app/local and add the following stanza:

[victorops]
disabled = 0
param.entity_id = ITSI Alert: $result.itsi_group_id$
param.entity_display_name = ITSI Alert: $result.itsi_group_title$
param.monitoring_tool = Splunk ITSI

Create a ticket in an external ticketing system

You can create a ticket in an external ticketing system from an ITSI episode.

1. Create a custom alert action in the Splunk platform. See Custom alert actions overview in Developing Views and Apps for Splunk Web.
2. Consume the Notable Event Action SDK to update external ticket information for a given episode using the Episode ID. See Notable event actions SDK reference.
3. Add a stanza for the custom alert action in $SPLUNK_HOME/etc/apps/SA-ITOA/local/notable_event_actions.conf.

If you have a custom alert action that exposes APIs along the lines of those exposed by the Splunk Add-on for ServiceNow or Splunk Add-on for Remedy, use the stanzas for [snow_incident] and [remedy_incident] in default/notable_event_actions.conf as examples.

Refer to the notable_event_actions.conf spec and example files located in $SPLUNK_HOME/etc/apps/SA-ITOA/README for more information.