Splunk® IT Service Intelligence

Event Analytics Manual

Integrate ITSI with PagerDuty

Integrate IT Service Intelligence (ITSI) with PagerDuty to correlate PagerDuty incidents with ITSI episodes and accelerate resolution. This integration leverages PagerDuty's Events API to send alerts.

Prerequisites

Requirement Description
PagerDuty You must be an administrator or account owner in PagerDuty.
ITSI roles You must have the admin role to set up an integration with PagerDuty.

Set up the integration

Before you integrate ITSI with PagerDuty, configure the API access token and a routing/incident key of an integration service.

Configure PagerDuty

Set up API access token

  1. Log in to PagerDuty.
  2. From the home page, select Integrations, then Developer Tools then API Access Keys.
  3. Provide a name for the key and select Create.
  4. Copy the API token.

Set up an integration service

  1. From the navigation menu, select Services then Service Directory.
  2. Select Create New Service.
  3. Enter a name and description.
  4. Assign an escalation policy. You can add an existing policy, otherwise the default policy is assigned. Pagerduty Escalation policy.png
  5. Select Turn Off Alert Grouping in the Noise section.
  6. Note: Turning off alert grouping is required, otherwise PagerDuty groups all incoming alerts into a single incident. For multiple episodes with unique deduplication keys, the same incident will be linked to all of them.

  7. Select the Events V2 API integration in the Integrations section. Pagerdutyintegration.png
  8. Expand the panel for the Event V2 API integration and copy the integration key.

Configure ITSI

You need to configure a PagerDuty account in ITSI.

  1. In ITSI, select Configuration then Event Management then PagerDuty.
  2. Select Add PagerDuty account.
  3. Add the following configuration information and then select Save:
    Option Description
    Account Name Enter an account name.
    PagerDuty URL The REST API URL of PagerDuty. By default, it is https://api.pagerduty.com, and you won't be able to edit this. Even if this is edited in the conf file itsi_pagerduty_accounts.conf, ITSI reads the URL as https://api.pagerduty.com.
    Routing Key The Incident Key from the Service in PagerDuty which you have created previously
    API Token The API Access Token of your PagerDuty instance which you have generated previously

Test the integration

Create PagerDuty incidents from ITSI.

  1. Select Alerts and Episodes from the ITSI navigation bar.
  2. Select an episode. Select on Actions then Send to PagerDuty.
  3. To create an incident, follow the steps in the Create an Incident in PagerDuty section above.
  4. Note: The integration is unidirectional, so updates to the PagerDuty incident don't reflect in ITSI.

Automate creating PagerDuty incidents

You can configure action rules in notable event aggregation policies to create and update PagerDuty incidents when certain trigger conditions are met. In the following example, you configure ITSI to create a critical incident in PagerDuty when a new episode is created. You also create a rule to resolve the PagerDuty incident when the episode breaks.

Define an action rule to create incidents in PagerDuty

  1. In ITSI, select Configuration then Notable Event Aggregation Policies.
  2. Select the aggregation policy you want to integrate with PagerDuty.
  3. Select the Action Rules tab. Select Add Rule and configure trigger conditions for when to create an incident in PagerDuty. For example, the following rule about PagerDuty opens an incident each time an episode is created. For example, you can create a rule that states "If the rule to If the number of events in this episode is 1, then send a PagerDuty event for the episode" by choosing the condition, if the number of events in this episode is exactly equal to 1, then send to PagerDuty. Pagerdutyrule1.png
  4. Select Configure and set up the following fields:
    Field Description
    PagerDuty Account The account name which you configured in ITSI.
    PagerDuty Event Action Trigger
    Source $result.source$
    Summary $result.itsi_group_title$
    Severity Critical

For more information about configuring aggregation policy action rules, see Configure episode action rules in ITSI.

Define an action rule to resolve incidents in PagerDuty

In the same aggregation policy, configure the following action rule to resolve a PagerDuty incident when its corresponding episode is resolved in ITSI.

  1. Select Add Rule and configure an additional action rule for when an episode breaks. For example, create a rule with the condition that if the episode is broken, then send to PagerDuty. Pagerdutyintegrationrule2.png
  2. Select Configure and configure the PagerDuty Event Action as Resolve.
  3. Select Save to save the policy.
Last modified on 28 May, 2024
Integrate ITSI with Splunk Add-on for Jira Cloud   Integrate ITSI with a webhook

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.19.0, 4.19.1


Was this topic useful?







You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters