Send alert notifications to a webhook using Splunk Observability Cloud ๐
You can configure Splunk Observability Cloud to automatically send alert notifications to a webhook when a detector alert condition is met and when the alert clears.
To send Observability Cloud alert notifications to a webhook, complete the following configuration tasks:
Step 2: Create a webhook integration in Observability Cloud
You must be an Observability Cloud administrator to perform this task.
Step 3: Add a webhook integration as a detector alert recipient in Observability Cloud
Caution
If the target destination is not available, Observability Cloud retries sending the notification for up to a day. After 24 hours of failed attempts the notification is not delivered.
Step 1: Create a webhook ๐
Create a webhook that listens for and receives Observability Cloud alert notification requests.
Your webhook must use a secure (HTTPS) connection and must support Transport Layer Security (TLS) 1.2 or higher.
To help secure your webhook, use a shared secret that Observability Cloud can provide to your webhook to confirm that its requests are friendly traffic.
Ensure that your webhook promptly returns a 200 response code after it has successfully received the request from Observability Cloud. If Observability Cloud does not receive a 200 response code within a certain timeframe, it will retry the request.
Hereโs some useful information about the request that will come from Observability Cloud:
Uses the POST method
Uses a
content-type
value ofapplication/json
Includes the parameters covered in Observability Cloud webhook request parameters.
In Step 2: Create a webhook integration in Observability Cloud, youโll need to provide the following information about your webhook:
URL
Shared secret
Header parameter values
Observability Cloud webhook request parameters ๐
Observability Cloud provides the following parameters in a JSON object in its request body.
For request body examples, see Observability Cloud webhook request examples.
Parameter |
Format |
Description |
---|---|---|
|
string |
Name of the detector |
|
string |
URL of the detector, which includes a parameter to select this specific incident |
|
string |
ID of the detector |
|
string |
(Optional) Description of the detector |
|
string |
URL of the alert preview image |
|
string |
Unique identifier for this alert notification |
|
string |
Unique identifier for the version of the detector that sent the notification |
|
string |
Name of the detector rule that triggered the alert |
|
string |
Severity level of the rule |
|
string |
(Optional) Runbook URL specified in this rule |
|
string |
(Optional) Tip specified in this rule |
|
string |
Notification title for this rule |
|
string |
Notification message for this rule |
|
string |
Trigger metric data and detection criteria in this rule, in SignalFlow format |
|
string |
(Optional) Clear metric data and detection criteria in this rule, in SignalFlow format. |
|
string |
Kept for backwards compatibility. Use |
|
string |
This is the state of the incident. Valid values include: |
|
string |
Time the event occurred, in ISO 8601 format |
|
array |
Map of the inputs involved in this rule. For more information, see inputs array. |
|
integer |
This is the schema version for this event. The value is always set to |
inputs
array ๐
The Observability Cloud webhook request includes an inputs
array. Each object in the array is named after the program variable itโs bound to. If it isnโt bound to a program variable, it uses a name like _S0
, _S1
, and so forth.
Each input object contains the following elements:
Element |
Description |
---|---|
|
(Optional) This is a map of the dimensions of the input signal. This element might be empty if there are no dimensions, but this is rare. This element might not be present if the input was a static value and not a comparison against scalar values. |
|
Value of the input when the alert triggered or when it cleared |
|
(Optional) This is the fragment of the SignalFlow program that represents the input. This element might not be present for some detectors or for static, anonymous inputs. |
Observability Cloud webhook request examples ๐
This section provides examples of the JSON request body that Observability Cloud can send to a webhook.
Note
Note about realms
A realm is a self-contained deployment of Splunk Observability Cloud in which your organization is hosted. Different realms have different API endpoints. For example, the endpoint for sending data in the us1
realm is https://ingest.us1.signalfx.com
, while the endpoint for sending data in the eu0
realm is
https://ingest.eu0.signalfx.com
.
When you see a placeholder realm name in the documentation, such as <YOUR_REALM>
, replace it with your actual realm name. To find your realm name, open the left navigation menu in Observability Cloud, select , and select your username. The realm name appears in the Organizations section. If you donโt include the realm name when specifying an endpoint, Observability Cloud defaults to the us0
realm.
This request provides an alert notification for a detector that alerts when memory use reaches or exceeds 90% for 10 minutes.
{
"sf_schema": 2,
"detector": "Memory usage detector",
"detectorUrl": "https://app.<YOUR_REALM>.signalfx.com/#/detector/ABCDEFGHIJK/edit",
"description": "A detector that alerts when memory usage exceeds 90% for 10 minutes",
"incidentId": "BCDEFGHIJKL",
"eventType": "foo",
"rule": "Running out of memory",
"severity": "Minor",
"description": "Memory has reached 90% of maximum for 10 minutes",
"detectOnCondition": "when(A > 90, '10m')",
"detectOffCondition": "when(A < 90, '15m')",
"status": "ok",
"statusExtended": "ok",
"imageUrl": "https://org.<YOUR_REALM>.signalfx.com/#/chart/abCDefGHij",
"timestamp": "2016-11-08T19:43:30Z",
"inputs": {
"_S1": {
"dimensions": {
"host": "i-346235qa",
"plugin": "signalfx-metadata"
},
"value": 96.235234634345,
"fragment": "data('memory.utilization')"
}
}
}
This request provides alert notifications for a detector that alerts when host latency is greater than the data center latency and the data center latency is greater than 40 ms.
{
"sf_schema": 2,
"detector": "My detector",
"detectorUrl": "https://app.<YOUR_REALM>.signalfx.com/#/detector/<id>/edit",
"incidentId": "<id>",
"eventType": "<event-type>",
"rule": "My detector rule",
"severity": "Critical",
"description": "Latency of host myserver is 43.4, over a datacenter-wide latency of 42.9",
"status": "anomalous",
"statusExtended": "anomalous",
"imageUrl": "https://org.<YOUR_REALM>.signalfx.com/#/chart/abCDefGHij",
"timestamp": "2016-10-25T21:19:38Z",
"detectOnCondition": "when(a > b and b > 40)",
"inputs": {
"a": {
"key": {
"host": "myserver",
"dc": "us-west-1"
},
"value": 43.4,
"fragment": "data('latency').p99(by=['host', 'dc'])"
},
"b": {
"key": {
"dc": "us-west-1"
},
"value": 42.9,
"fragment": "data('latency').p99(by='dc')"
},
"_S2": {
"value": 40,
"fragment": "40"
}
}
}
Step 2: Create a webhook integration in Observability Cloud ๐
You must be an Observability Cloud administrator to perform this task.
Log in to Splunk Observability Cloud.
In the left navigation menu, select
.Select Add Integration.
In the integration filter menu, select All.
In the Search field, search for Webhook, and select it.
Click New Integration to display the configuration options.
Enter a name for this integration. Give your integration a unique and descriptive name. For information about the downstream use of this name, see About naming your integrations.
In the URL field, enter the webhook URL you created in Step 1: Create a webhook.
In the Shared Secret field, enter the shared secret you created in Step 1: Create a webhook.
In the Headers section, enter any header parameters required by the webhook you created in Step 1: Create a webhook.
Save.
If Observability Cloud is able to validate the URL, shared secret, and headers you provided for your webhook, a Validated! success message displays. If you get an error, make sure that the values you entered match the values you defined in Step 1: Create a webhook.
Step 3: Add a webhook integration as a detector alert recipient in Observability Cloud ๐
To add a webhook integration as a detector alert recipient in Observability Cloud:
Create or edit a detector that you want to configure to send alert notifications using your webhook integration.
For more information about working with detectors, see Create detectors to trigger alerts and Subscribe to alerts using the Detector menu.
In the Alert recipients step, click Add Recipient.
Select Webhook and then select the name of the webhook integration you want to use to send alert notifications. This is the integration name you created in Step 2: Create a webhook integration in Observability Cloud.
Activate and save the detector.
Observability Cloud will send an alert notification to the webhook when an alert is triggered by the detector and when the alert clears.