Use the Content Pack for Splunk Infrastructure Monitoring

After you install and configure the Content Pack for Splunk Infrastructure Monitoring, it's time to start actively monitoring your environment. The content pack includes three preconfigured correlation searches that pull Splunk Infrastructure Monitoring event data, including anomalous events and clear events, from each of your integrated AWS, Azure, and GCP instances. The correlation searches create corresponding notable events in ITSI with drilldown links back to the event in Infrastructure Monitoring. The content pack also includes three aggregation policies that group these events into meaningful episodes.

The following image shows an Azure service in the Service Analyzer and its corresponding episode:

Infrastructure Monitoring records an event when monitoring data meets an anomalous condition defined in a detector. When the detector's trigger condition is no longer true, it records a clear event. For more information, see About Detectors and Alerts in the Splunk Infrastructure Monitoring documentation.

Tune correlation searches

The correlation searches in the content pack use the event subcommand from the Splunk Infrastructure Monitoring Add-on to identify any detectors that have fired for a given cloud provider. For example, if a detector creates an event when you exceed a limit on unreserved instances of AWS, the AWS correlation search creates a corresponding notable event in ITSI.

The following correlation searches are included in the content pack.

• Splunk Infrastructure Monitoring Events AWS EC2 Search
• Splunk Infrastructure Monitoring Events Azure VM Search
• Splunk Infrastructure Monitoring Events GCP Compute Engine Search

Each search is enabled by default. To view, modify, or disable them, go to Configuration > Correlation Searches from the ITSI main menu.

Each correlation search executes the sim command to get all events within your monitored AWS EC2, Azure VM, or GCP environment. The searches extract the event ID and detector ID from the data, and the eval function gets the instance_id, which is one of the entity aliases in ITSI. The join function assigns the alias field to a value. Each search runs every minute with a 15-minute lookback time. You can tune these parameters to run at a frequency matching your Infrastructure Monitoring data collection interval.

For more information about correlation searches, see Overview of correlation searches in ITSI in the Event Analytics manual.

event subcommand

Each correlation search includes the event subcommand from the Splunk Infrastructure Monitoring Add-on, which uses the GET /v2/event API to retrieve events. This endpoint returns events generated by Infrastructure Monitoring, which have a name prefixed by sf_ to indicate that they originated from Infrastructure Monitoring.

For usage examples of the event command, see event query syntax in the Splunk Infrastructure Monitoring Add-on manual.

Tune aggregation policies

This content pack includes the following notable event aggregation policies that group notable events from Infrastructure Monitoring into episodes:

• SIM AWS EC2 Alerts
• SIM Azure VM Alerts
• SIM GCP Compute Engine Alerts

The aggregation policies are enabled by default. To view, modify, or disable them, go to Configuration > Notable Event Aggregation Policies from the ITSI main menu.

The aggregation policies group events into episodes based on whether they came from AWS, Azure, or GCP. You can modify the filtering and breaking criteria to meet your specific use cases. For instructions, see Configure episode filtering and breaking criteria in ITSI.

Each policy also contains action rules that change the severity to High if the episode has more than 30 events, and to Critical if is has more than 90 events. You can modify the existing action rules or build additional rules, such as adding a comment, sending an email, or creating a ticket in an external system. For more information about creating action rules, see Configure episode action rules in ITSI in the Event Analytics manual.

View and manage Infrastructure Monitoring events in ITSI

All notable events generated by a Infrastructure Monitoring correlation search have a title with the format Splunk Infrastructure Monitoring Events for <entity_type>. For example, Splunk Infrastructure Monitoring Events for AWS EC2. They each have a severity of High and a status of New. Each notable event includes a drilldown link back to the detector that created the event in Infrastructure Monitoring. The description field of the ITSI notable event is the informational field in the Infrastructure Monitoring event. You can modify this description using the Notable Event Description field of the correlation search.

For more information about Episode Review and episode management, see Overview of Episode Review in ITSI.

The following image shows events in Episode Review generated from the GCP Compute Engine correlation search. Each event includes a drilldown link back to the event in Infrastructure Monitoring:

Bidirectional integration with Infrastructure Monitoring

ITSI includes a bidirectional integration with Infrastructure Monitoring that lets you clear all Infrastructure Monitoring incidents within an episode. An Infrastructure Monitoring incident is the combination of a trigger event and a clear event.

To clear all incidents within an ITSI episode, select the episode in Episode Review and click Actions > Clear Splunk Infrastructure Monitoring Incidents. If the detector's alert condition is still active after clearing, Infrastructure Monitoring creates a new incident for the alert.

The option to clear Infrastructure Monitoring incidents only appears after you install and configure the Splunk Infrastructure Monitoring Add-on.

Monitor AWS, Azure, and GCP entities in the Infrastructure Overview

Each Infrastructure Monitoring entity type contains a set of vital metrics, which are statistical calculations based on SPL searches that represent the overall health of entities of that type. You can view each entity type in the Infrastructure Overview and drill down further into each of their vital metrics. For more information about the Infrastructure Overview and vital metrics, see About the Infrastructure Overview in ITSI.

The following table lists the default vital metrics for each Infrastructure Monitoring entity type:

Entity type	Vital metrics
AWS EC2	CPU Utilization Network In Network Out Disk Read Ops Disk Write Ops
AWS Lambda	Duration Invocations
Azure Virtual Machines	CPU Utilization Network In Network Out Disk Read Ops Disk Write Ops
Azure Functions	Executions Memory
GCP Compute Engine	CPU Utilization Network In Network Out Disk Read Ops Disk Write Ops
GCP Cloud Functions	Executions Memory