Docs » Send alert notifications to third-party services using Splunk Observability Cloud » Send alert notifications to Splunk On-Call using Splunk Observability Cloud

Send alert notifications to Splunk On-Call using Splunk Observability Cloud 🔗

You can configure Splunk Observability Cloud to automatically send alert notifications to Splunk On-Call (formerly VictorOps) when a detector alert condition is met and when the alert clears.

To send Observability Cloud alert notifications to Splunk On-Call, complete the following configuration tasks:

Step 1: Get your Splunk On-Call service API endpoint URL 🔗

You must be a Splunk On-Call global admin or alert admin to perform this task.

To get the service API endpoint URL in Splunk On-Call:

  1. Log in to Splunk On-Call.

  2. Click the Integrations tab.

  3. Click the 3rd Party Integrations tab.

  4. Click the SignalFX Signal Monitoring tile. The Service API Endpoint value displays.

    If you don’t see an endpoint URL value, click Enable Integration to generate one..

  5. Copy the entire endpoint URL, including the $routing_key text, for use in Step 3: Create a Splunk On-Call integration in Observability Cloud.

Step 2: Get your Splunk On-Call alert routing key 🔗

You must be a Splunk On-Call global admin or alert admin to perform this task.

For information about how to get your Splunk On-Call alert routing key, see Create & Manage Alert Routing Keys.

You’ll need your alert routing key in Step 3: Create a Splunk On-Call integration in Observability Cloud.

Step 3: Create a Splunk On-Call integration in Observability Cloud 🔗

You must be an Observability Cloud administrator to perform this task.

To create a Splunk On-Call integration in Observability Cloud:

  1. In the Observability Cloud navigation menu, select Data Setup.

  2. In the CATEGORIES menu, select Notification Services.

  3. Click the Splunk On-Call (VictorOps) tile.

  4. Click New Integration to display the configuration options.

  5. By default, the name of the integration is VictorOps. Splunk recommends that you give your integration a unique and descriptive name. For information about the downstream use of this name, see About naming your integrations.

  6. In the Post URL field, enter the service API endpoint URL value you copied from Splunk On-Call in Step 1: Get your Splunk On-Call service API endpoint URL.

  7. Click Save.

  8. If Observability Cloud is able to validate the Splunk On-Call service API endpoint URL, a Validated! success message displays. If an error displays instead, make sure that the URL value you entered matches the value displayed in Splunk On-Call in Step 1: Get your Splunk On-Call service API endpoint URL.

Step 4: Add a Splunk On-Call integration as a detector alert recipient in Observability Cloud 🔗

To add a Splunk On-Call integration as a detector alert recipient in Observability Cloud:

  1. Create or edit a detector that you want to configure to send alert notifications using your Splunk On-Call integration.

    For more information about working with detectors, see Create detectors to trigger alerts and Subscribe to alerts using the Detector menu.

  2. In the Alert recipients step, click Add Recipient.

  3. Select VictorOps and then select the name of the Splunk On-Call integration you want to use to send alert notifications. This is the integration name you created in Step 3: Create a Splunk On-Call integration in Observability Cloud.

  4. Enter the routing key you got in Step 2: Get your Splunk On-Call alert routing key

  5. Activate and save the detector.

Observability Cloud will send an alert notification to your Splunk On-Call timeline when an alert is triggered by the detector and when the alert clears.

Observability Cloud alert notification fields sent to Splunk On-Call 🔗

Here are the Observability Cloud alert notification fields that are sent to Splunk On-Call.

Field

Description

Detector Definition

Displays a link to view the Observability Cloud detector and corresponding alert rules.

Graph

Displays a snapshot view of the signal that triggered the alert.

detector

Displays the name of the Observability Cloud detector.

inputs

Displays detailed information about the Observability Cloud alert, including the rule and detector names, alert triggering conditions, and signal details.

rule

Displays the name of the Observability Cloud alert rule where the conditions to trigger and clear alert events are defined.

entity_display_name

Displays the Observability Cloud rule and detector name. This information also appears in the rule and detector notification fields.

state_message

When the alert is triggered, displays the alert’s severity. Valid values include: critical, major, minor, warning, or info. When the alert is resolved, displays the alert’s resolution. Valid values include: back to normal, stopped, or manually resolved.

entity_id

Displays the incident’s ID.

monitoring_tool

Displays signalfx.

message_type

Displays the alert’s severity. Valid values include: critical, warning, acknowledgement, info, or recovery.