Splunk® IT Service Intelligence

Event Analytics Manual

Download manual as PDF

Download topic as PDF

Integrate ITSI with VictorOps

Integrate IT Service Intelligence (ITSI) with VictorOps to correlate VictorOps incidents with ITSI episodes. Your teams can collaborate with monitoring data inside the VictorOps timeline to speed up incident response and remediation.

Prerequisites

Requirement Description
VictorOps You must have VictorOps version 1.16 or later.
ITSI roles You must have the itoa_admin or itoa_team_admin role to set up the integration with VictorOps.

Set up the integration

Perform the following steps to integrate ITSI with VictorOps:

1. Install and configure the VictorOps for Splunk app

  1. Install VictorOps for Splunk version 1.0.20 or later. See VictorOps For Splunk on Splunkbase.
  2. From the VictorOps web portal, click the Integrations tab.
  3. Click Splunk ITSI.
  4. Click Enable Integration and copy the API key to the clipboard.
  5. Open the VictorOps for Splunk app and click Configuration > Alert API Key Configuration.
  6. Configure the following fields:
    Field Description
    Name Provide a name for the API key.
    API Key The API key you copied in the previous step.
    Routing Key <Leave blank>

2. Test the integration

You're now ready to start creating VictorOps incidents from ITSI.

  • Within ITSI, navigate to Episode Review. If you configured the integration correctly, the option to Create an incident in VictorOps appears in the Actions dropdown menu.
  • To create an incident, follow the steps in Create a ticket in VictorOps in the User Manual.

Note: The integration is unidirectional, so updates to VictorOps tickets aren't reflected in ITSI.

Automate the process of creating VictorOps incidents

Configure aggregation policy action rules to create and update VictorOps incidents when certain trigger conditions are met. In the following example, you configure ITSI to create a critical incident in VictorOps when an episode's severity is greater than Normal. You also create a rule to resolve the VictorOps incident when the episode breaks, and automatically set the incident's status to RECOVERY, which resolves the incident.

1. Define a rule to create incidents in VictorOps

Define the following aggregation policy action rule to open a critical VictorOps incident when an episode is created in ITSI:

  1. Within ITSI, click Configuration > Notable Event Aggregation Policies.
  2. Open the aggregation policy you want to integrate with VictorOps.
  3. Go to the Action Rules tab.
  4. Click Add Rule and configure trigger conditions for when to open a VictorOps incident. For example, the following rule opens an incident each time an episode is created:
    VictorOpsaction.png
  5. Click Configure and configure the following fields:
    Field Description
    Message Type CRITICAL
    Monitoring Tool Splunk-ITSI
    Alert Entity ID $result.itsi_group_id$.

    Keep this ID consistent for all message types across related actions. VictorOps uses this field to identify incidents and correlate subsequent alerts with the original incident. Once configured correctly, ITSI pipes directly into your VictorOps timeline.

    Alert Entity Display Name $result.itsi_group_title$
    State Message $result.itsi_group_title$
    Routing Key Optionally, configure a routing key to override the global VictorOps routing key.
    For more information about configuring aggregation policy action rules, see Configure episode action rules in ITSI.

2. Define a rule to resolve incidents in VictorOps

Within the same aggregation policy, configure the following action rule to resolve a VictorOps incident when its corresponding episode is resolved in ITSI.

  1. Click Add Rule and configure an additional action rule for when an episode breaks: VictorOpsresolve.png
  2. Click Configure and configure the policy to change the incident's status to RECOVERY when the ITSI incident is resolved.
  3. Click Save to save the policy.

Manually acknowledge a VictorOps incident

Once a VictorOps ticket is created from an ITSI episode, you can manually acknowledge that VictorOps incident.

  1. From the ITSI main menu, click Episode Review.
  2. Select an episode that is currently linked to a VictorOps ticket.
  3. Click Actions > Create VictorOps incident.
  4. Configure the following fields:
    Field Description
    Message Type ACKNOWLEDGEMENT
    Monitoring Tool Splunk-ITSI.
    Alert Entity ID $result.itsi_group_id$
    Alert Entity Display Name $result.itsi_group_title$
    State Message $result.itsi_group_title$
    Routing Key Optionally, configure a routing key to override the global VictorOps routing key.
  5. Click Done. Check the Activity tab to confirm the action ran successfully.

Enable non-admins to test and send alerts

Non-admin roles creating VictorOps incidents through the Episode Review Actions must be assigned certain permissions. To provide non-admin roles the ability to create and test VictorOps incidents, assign them the following capabilities:

  • execute-notable_event_action
  • write-notable_event
  • delete-notable_event
  • list_storage_passwords
  • victorops_user

Set VictorOps fields to reasonable defaults

Rather than manually typing in the VictorOps incident creation fields each time you create an incident, you can set the fields to reasonable defaults.

Prerequisites

  • Only users with file system access, such as system administrators, can modify default fields using a configuration file.
  • Review the steps in How to edit a configuration file in the Admin Manual.

Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location.

Steps

  1. Open or create a local version of alert_actions.conf in $SPLUNK_HOME/etc/apps/victorops_app/local.
  2. Add the following stanza:
    [victorops]
    disabled = 0
    param.entity_id = ITSI Alert: $result.itsi_group_id$
    param.entity_display_name = ITSI Alert: $result.itsi_group_title$
    param.monitoring_tool = Splunk-ITSI
    
Last modified on 15 September, 2020
PREVIOUS
Integrate ITSI with BMC Remedy
  NEXT
Set up custom episode actions in ITSI

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.5.0 Cloud only, 4.5.1 Cloud only, 4.6.0 Cloud only, 4.6.1 Cloud only, 4.6.2 Cloud only, 4.7.0


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters