Monitor and investigate alerts in Splunk App for Infrastructure

Admin privileges are required to create and manage alerts.

Use alerts to monitor triggered events and perform root cause analysis for a specific behavior in your data. The Alerts page displays a table of triggered alerts. From here you can link to the Analysis Workspace, where you can continue to investigate performance issues during the time of the alert. This topic includes information about how to:

• Monitor and investigate alerts in the Analysis Workspace
• Monitor and investigate alerts from the Alerts tab
• View alert notifications
• View alert details

Monitor and investigate alerts in the Analysis Workspace

Investigate alerts in the Analysis Workspace. To view alerts, follow these steps:

1. Go to the Investigate page.
2. Select the Entities or Groups view to investigate entities or groups, respectively.
3. From the list, select an entity or group you want to investigate to open the Analysis Workspace.
4. If you selected an individual entity, click Analysis to view the Analysis Workspace.
5. From the Data section of the Analysis Workspace, select Alerts to view alerts for the entity or group.

The alerts are listed in a tree structure according to the data source they use. Click a data source name to see a list of alerts that are based on it, and to view the alert details in the workspace. The chart displays color-coded severity levels based upon thresholds you set when creating the alert.

Monitor and investigate alerts from the Alerts tab

Investigate triggered alerts from the Alerts tab. When you view alerts from here, you can filter the list by dimensions of entities or groups to search for triggered alerts that are affecting entities or groups of similar dimensions.

A color-coded severity status displays the state of the alert. This allows you to easily view the state of the alert by color. Severity colors include: Green for Information, Yellow for Medium, and Red for Critical. A severity status arrow displays alongside the severity status color in the Current Severity column, indicating whether the alert has improved or degraded recently.

The row size in the Alerts tab is limited to two rows per entity. If resizing the window, or if the slide-out panel is activated, the dimensions column truncates and only displays dimensions that fit into the available two-row column size. The row does not expand or wrap to more than two rows, so all dimensions might not display. To view all of the dimensions for each entity or group, close the slide-out panel or resize your window to a larger view.

To start investigating an alert, follow these steps:

1. Go to the Alerts tab.
2. Select the Entities or Groups view to investigate entities or groups, respectively.
3. (Optional) Filter for triggered alerts according by entity dimensions or group names to find triggered alerts for specific types of entities or groups. This enables you to monitor triggered alerts for specific dimensions across entities and groups.
4. Select an entity or group, depending on the view you selected. From the list of triggered alerts in the slide-out panel on the right side of the page, view the triggered alerts for the selected entity or group. The slide-out panel displays every triggered alert for the entity or group you selected.
5. For the alert that you want to investigate, click the icon and select Investigate. This takes you to the Analysis Workspace for the alert.
6. (For admins and power users) To view the alert configuration and learn more about what the alert is tracking, click the icon and select Clone this Panel. From the cloned alert panel, you can modify Aggregations for the alert's metric, modify the Split by dimension for the alert's metric, display a Time comparison for the alert's metric, and Filter the alert's metric by dimension.
   1. (Optional) After investigating the alert and modifying any configurations for it using the cloned panel, you can use the cloned panel create a new alert with the modified configuration. To do so, click the icon and select Create Alert. For more information about creating alerts, see Create and modify alerts in Splunk App for Infrastructure.

Alert notifications

When an alert is created by an administrator, there is an option to include an alert notification. This can be an alert notification sent by email or using VictorOps for Splunk. When the alert triggers, one or more notifications are sent depending on the type of alert notification and recipients selected during the alert creation, including:

• An email notification with details of the alert.
• A VictorOps notification, with the details displaying in your VictorOps account timeline. In your VictorOps account, click Alert Payload link to view details of the alert.

Alert details

Select an alert from the Alerts tab or the Analysis Workspace to view its details. These details include the threshold conditions and severity levels configured for the alert, settings, and triggered instances. Triggered instances appear as annotations on the chart, and up to 100 annotations can display on the chart. Triggered instance annotations appear at the time the alert triggers, not the precise time the alert threshold is crossed.

Alert badges gauge the alert severity level. To help you monitor alert activity, badge colors are based on the most recent severity level of a triggered alert.

Severity level	Badge color
No trigger	Gray
Info	Green
Medium	Yellow
Critical	Red

Example

The following alert shows CPU user average for the cpu.user metric.

This alert is based on the aggregate average of cpu.user metric values. The green alert badge indicates a severity level of Info, yellow for medium, and red for critical. The horizontal lines show the alert threshold values. The annotations show triggered instances for the alert.