Create and modify alerts in Splunk App for Infrastructure

Use alerts to monitor and respond to specific events. Alerts trigger when alert thresholds set for a metric on an entity or group meet specific conditions. Creating an alert includes:

• Selecting metrics that the alert will track.
• Configuring a threshold for the alert that triggers when a tracked metric reaches the threshold.
• Configuring alert notifications to receive an email or VictorOps notification when a tracked metric triggers the alert.

When you create an alert, it appears in the Alerts drop-down in the Data section of the Analysis Workspace for the entity or group, and displays in the Entity or Group view of the Alerts page if triggered.

Create an alert

Follow these steps to configure an alert for an entity or group. Before creating an alert and sending an alert notification, you need to configure notification settings. For more information, see Configure Alert Notification Settings in Splunk App for Infrastructure.

1. Select an entity or group from the Entity or Group view to drill down into the Analysis Workspace.
2. From the Data section of the entity or group's Analysis Workspace, select a metric for which you want to create an alert.
3. (Optional) Select a metric and click Split by (when viewing a metric for a group, Split all by) to split the metric by a specific dimension. You can split a metric by any dimension for an entity or group when creating an alert. If you split by a host-identifying dimension when creating a group alert, entities in the group that reach the threshold for the alert will trigger the alert and appear in the Entities view of the Alerts tab. If you do not split by a dimension when creating a group alert, or split by any dimension that's not a host-identifying dimension, the alert will trigger for the whole group, and will appear in the Groups view of the Alerts tab. Here are some examples of host-identifying dimensions:
• host
• ip
• InstanceId
• VolumeId
• LoadBalancerName
4. (Optional) Drag your cursor over a time area and data point in the chart to pinpoint what data to use to create the alert.
5. In the top-right corner of the chart, click the icon.
6. Click Create Alert. If you do not see the Create Alert option, you might not be logged in as a user with permissions to create alerts. The metrics panel also needs to contain data to create an alert.
7. In the Create Alert window, set alert thresholds for the metric. The alert chart in the dialog visually displays the thresholds.
8. (Optional) Enter a custom name for the alert following the character requirements. A Name for the alert is automatically generated.
9. Set up trigger conditions for thresholds. The Critical threshold is required. You can adjust this threshold value, but the threshold cannot be deleted.
1. (Optional) Click Add New Threshold to create a Warning threshold as well.
2. For the If field, select greater than or less than to set the threshold hierarchy. If you select greater than, the Critical threshold is a maximum threshold. If you select less than, the Critical threshold is a minimum threshold.
3. Modify the value to meet each threshold. You can enter a value or drag the point on the y-axis of the chart in the Create Alert window.
10. Click Alert Notification to set up a notification. You can configure an alert to send notifications via email, VictorOps for Splunk, a Slack webhook, or a custom webhook when the alert severity improves, degrades, or changes at all. If you select one of the webhook notification options, you can select the default webhook you entered in the notification settings or enter a new one. For more information about configuring alert notifications, see Configure alert notifications in Splunk App for Infrastructure.
11. When you are done configuring the alert, click Submit.

Edit an alert

Edit an alert to change threshold trigger conditions, or to add or change email recipients for notification for when the alert triggers.

1. In the Analysis Workspace > Data > Alerts section, click a metric alert.
2. In the chart for the alert, click the .
3. Click Edit Alert.
4. In the Edit Alert dialog, you can edit the alert name, threshold values and alert notification information.
5. Click Submit.