Docs » Splunk On-Call integrations » Datadog integration for Splunk On-Call

Datadog integration for Splunk On-Call 🔗

Use the Datadog integration with Splunk On-Call to send content from Datadog into the timeline using @ mentions in the Datadog event stream.

Requirements 🔗

  • Splunk On-Call version required:** Starter, Growth, or Enterprise

  • Annotating the links to your incidents provided by Datadog requires the use of the Splunk On-Call rules engine which is an Enterprise feature

Enable the integration in Splunk On-Call 🔗

  1. Go to Integrations then 3rd Party Integrations then Datadog.

  2. Select Enable Integration.

  3. Copy the Service API Key to your clipboard.

Configure Datadog 🔗

  1. Select Integrations in the sidebar menu, find the VictorOps integration, and select Available then Install.

    Datadog integrations

  2. In the dialog box, select the configuration tab.

  3. Paste in your service API key you obtained from Splunk On-Call, as well as, the Splunk On-Call routing key you want to use and select Update Configuration.

    Datadog VictorOps integration configuration

  4. Add @victorops to your Datadog metric monitors. To ensure a monitor autoresolves the corresponding VictorOps incident, make sure that @victorops is selected in the monitor’s notification step.

    VictorOps mention in Datadog metric monitors

Resolve the DataDog monitor from Splunk On-Call 🔗

Typically it is best to let Datadog, the source of the incident, resolve monitor incidents. However, if you want to resolve the incident from Splunk On-Call, here are the steps to do so.

Requirements 🔗

Custom Outgoing Webhooks and the Alert Rules Engine are Enterprise features.

Steps 🔗

  1. Go to Integrations then Outgoing Webhooks in Splunk On-Call.

  2. Create a new Custom Outgoing Webhook with the following details. Add your specific Datadog API credentials to the end of the URL, as in the example.

    • In the Event field, select Incident-Resolved.

    • For Method select POST.

    • For Content Type select application/json.

    • In the To field, enter your Datadog API credentials as in this example: https://api.datadoghq.com/monitor/bulk_resolve?api_key=<datadpg_api_key>&application_key=<datadog_app_key>

    • Paste the following payload:

      {
    "resolve": [
       {
          "${{ALERT.datadog_monitor_id}}": "ALL_GROUPS"
       }
    ]
 }

  3. Next, create an alert rules engine rule to conditionally fire the webhook you created. To do so, go to Settings then Alert Rules Engine and create the following rule: #. When monitoring_tool matches Datadog #. Under Transform these alert fields configure the following transformation: select datadog_monitor_id then ${{monitor_data.id}}.

Splunk On-Call alert rules engine rule transformation

(Optional) Additional alert rules engine rule transformations 🔗

Datadog alerts typically include links to your alerts that you can annotate to automatically render snapshot images and provide links to Datadog. You can also make some additional rules engine transformations:

  1. When monitoring_tool matches Datadog

  2. Under Transform these alert fields configure the following transformations:

    • Select Image URL then Snapshot then ${{snapshot_url}}. This transformation renders a snapshot image in the Splunk On-Call timeline.

    • Select URL then Snapshot Link then ${{snapshot_link}}}. This transformation provides link to the image in Datadog.

    • Select URL then Event Link then ${{event_url}}. This transformation provides a link to the event in Datadog.

Splunk On-Call optional alert rules engine rule transformation

This page was last updated on Feb 22, 2024.