Grafana integration for Splunk On-Call 🔗
Use the Splunk On-Call Grafana integration to forward Grafana alerts and dashboards into Splunk On-Call to notify the correct on-call users. Create on-call schedules, rotations, and escalation policies in Splunk On-Call, then route Grafana alerts and charts based on those parameters. Monitor and visualize logs, metrics and traces in Grafana and set thresholds to optimize alerts, create charts and surface actionable system insights.
When events meet predetermined monitoring criteria, Grafana sends an alert notification. Then, in the Splunk On-Call timeline, users can route and escalate critical alert data to the correct people. With the Splunk On-Call and Grafana integration, on-call responders can collaborate in real time around system data to reduce mean time to acknowledge (MTTA) and mean time to repair (MTTR) and resolve incidents faster.
With the Splunk On-Call Grafama integration, teams are able to:
Track system performance over time, visualize service health and alert on-call teams when applicable monitoring thresholds are met
Surface alert context in the form of metrics, logs, graphs and links to runbooks
Automatically route and escalate Grafana alert data through Splunk On-Call software and collaborate in a single-pane-of-glass to drive incident workflows forward
Chat in-line with incident context to improve collaboration and quickly share critical infrastructure information and dashboards, reducing MTTA and MTTR
Activate Grafana in Splunk On-Call 🔗
From the main timeline go to Integrations then Grafana.
Select the Grafana integration and copy the Service API Endpoint for later use.
Connect Splunk On-Call in Grafana 🔗
From the main dashboard, select the options menu then Alerting then Notification Channels.
Add a new channel.
Give the notification a name select the VictorOps type.
In the VictorOps settings paste in your Service API Endpoint. Replace the
$routing_keywith the routing key you intend to use. For more information on routing keys, see Create Routing Keys in Splunk On-Call.Save the integration.
The resulting test alert in Splunk On-Call looks similar to this:
Connect the notification channel to your alert 🔗
Locate the Grafana Dashboard panel you want to send alerts to Splunk On-Call and select Edit under dropdown arrow next to the panel name.
Go to the Alert section.
Define your alert conditions.
Under Notifications, select the plus button to select the notification channel you configured.
Save your changes.
You have completed the standard configuration. Repeat these steps as necessary, creating separate notification channels for each routing key.
Include an image on the alert 🔗
For those with Enterprise Splunk On-Call features, you can include the Grafana image of the alert in the notification.
When configuring the notification channel in Grafana, select Include image.
In Splunk On-Call, go to Settings then Alert Rules Engine and add the following Rules Engine rule to surface the image_url as an annotation to the incident. Ensure that the image is hosted in a publicly accessible location so that Splunk On-Call can display it. For further information, see Use images in notifications in Grafana documentation.
Your resulting Splunk On-Call incident now includes the image as an annotation:
