Integrate ITSI with Splunk On-Call (VictorOps)

Integrate IT Service Intelligence (ITSI) with Splunk On-Call (VictorOps) to correlate Splunk On-Call incidents with ITSI episodes. Your teams can collaborate with monitoring data inside the Splunk On-Call timeline to speed up incident response and remediation.

Prerequisites

Requirement	Description
Splunk On-Call	You must have Splunk On-Call (VictorOps) Alert Admin Permissions or greater.
ITSI roles	You must have the itoa_admin or itoa_team_admin role to set up the integration with VictorOps.

Set up the integration

Perform the following steps to integrate ITSI with Splunk On-Call (VictorOps):

1. Install and configure the Splunk On-Call (VictorOps) app

1. Install Splunk On-Call (VictorOps) version 1.0.20 or later. See Splunk On-Call on Splunkbase.
2. From the Splunk On-Call web portal, click the Integrations tab.
3. Click Splunk ITSI.
4. Click Enable Integration and copy the API key to the clipboard.
5. Open the Splunk On-Call (VictorOps) app and click Configuration > Alert API Key Configuration.
6. Configure the following fields:

   Field	Description
   Name	Provide a name for the API key.
   API Key	The API key you copied in the previous step.
   Routing Key	<Leave blank>

2. Test the integration

You're now ready to start creating Splunk On-Call incidents from ITSI.

• Within ITSI, navigate to Episode Review. If you configured the integration correctly, the option to Create an incident in Splunk On-Call appears in the Actions dropdown menu.
• To create an incident, follow the steps in Create a ticket in Splunk On-Call.

Note: The integration is unidirectional, so updates to Splunk On-Call tickets aren't reflected in ITSI.

Automate the process of creating Splunk On-Call (VictorOps) incidents

Configure aggregation policy action rules to create and update Splunk On-Call incidents when certain trigger conditions are met. In the following example, you configure ITSI to create a critical incident in Splunk On-Call when an episode's severity is greater than Normal. You also create a rule to resolve the Splunk On-Call incident when the episode breaks, and automatically set the incident's status to RECOVERY, which resolves the incident.

1. Define a rule to create incidents in Splunk On-Call

Define the following aggregation policy action rule to open a critical Splunk On-Call incident when an episode is created in ITSI:

1. Within ITSI, click Configuration > Notable Event Aggregation Policies.
2. Open the aggregation policy you want to integrate with Splunk On-Call.
3. Go to the Action Rules tab.
4. Click Add Rule and configure trigger conditions for when to open a Splunk On-Call incident. For example, the following rule opens an incident each time an episode is created:
   
5. Click Configure and configure the following fields:

   Field	Description
   Message Type	CRITICAL
   Monitoring Tool	Splunk-ITSI
   Alert Entity ID	$result.itsi_group_id$. Keep this ID consistent for all message types across related actions. Splunk On-Call uses this field to identify incidents and correlate subsequent alerts with the original incident. Once configured correctly, ITSI pipes directly into your Splunk On-Call timeline.
   Alert Entity Display Name	$result.itsi_group_title$
   State Message	$result.itsi_group_title$
   Routing Key	Optionally, configure a routing key to override the global Splunk On-Call routing key.

   For more information about configuring aggregation policy action rules, see Configure episode action rules in ITSI.

2. Define a rule to resolve incidents in Splunk On-Call

Within the same aggregation policy, configure the following action rule to resolve a Splunk On-Call incident when its corresponding episode is resolved in ITSI.

1. Click Add Rule and configure an additional action rule for when an episode breaks:
2. Click Configure and configure the policy to change the incident's status to RECOVERY when the ITSI incident is resolved.
3. Click Save to save the policy.

Manually acknowledge a Splunk On-Call incident

Once a Splunk On-Call ticket is created from an ITSI episode, you can manually acknowledge that incident.

1. From the ITSI main menu, click Episode Review.
2. Select an episode that is currently linked to a Splunk On-Call ticket.
3. Click Actions > Create Splunk On-Call incident.
4. Configure the following fields:

   Field	Description
   Message Type	ACKNOWLEDGEMENT
   Monitoring Tool	Splunk-ITSI.
   Alert Entity ID	$result.itsi_group_id$
   Alert Entity Display Name	$result.itsi_group_title$
   State Message	$result.itsi_group_title$
   Routing Key	Optionally, configure a routing key to override the global Splunk On-Call routing key.

5. Click Done. Check the Activity tab to confirm the action ran successfully.

Enable non-admins to test and send alerts

Non-admin roles creating VictorOps incidents through the Episode Review Actions must be assigned certain permissions. To provide non-admin roles the ability to create and test VictorOps incidents, assign them the following capabilities:

• execute-notable_event_action
• write-notable_event
• delete-notable_event
• list_storage_passwords
• victorops_user

Set Splunk On-Call fields to reasonable defaults

Rather than manually typing in the Splunk On-Call incident creation fields each time you create an incident, you can set the fields to reasonable defaults.

Prerequisites

• Only users with file system access, such as system administrators, can modify default fields using a configuration file.
• Review the steps in How to edit a configuration file in the Admin Manual.

Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location.

Steps

1. Open or create a local version of alert_actions.conf in $SPLUNK_HOME/etc/apps/victorops_app/local.
2. Add the following stanza:

   [victorops]
   disabled = 0
   param.entity_id = ITSI Alert: $result.itsi_group_id$
   param.entity_display_name = ITSI Alert: $result.itsi_group_title$
   param.monitoring_tool = Splunk-ITSI