Generate events with correlation searches in ITSI

A correlation search is a recurring search within IT Service Intelligence (ITSI) that scans multiple data sources for defined patterns. You can configure a correlation search to generate a notable event when search results meet specific conditions. Use Episode Review to review notable events that your correlation searches generate and to initiate the investigative process of determining root cause.

Configure correlation searches to update the settings associated with how they run, change the search logic, and throttle alerts. See Overview of correlation searches in ITSI to learn more about correlation searches.

Prerequisites

Requirement	Description
ITSI role	You must have the write_itsi_correlation_search capability to create a correlation search. The itoa_admin and itoa_team_admin ITSI roles have this capabilities by default.

Steps

1. From the ITSI main menu, click Configuration > Correlation Searches.
2. Click Create New Search > Create Correlation Search.
3. Configure the following fields:

Search Properties

Field	Description	Defaults
Search Name	A name that describes the correlation search. For example, "cpu_load_percent".	None
Description	(Optional) A description of the type of issue the search is intended to detect.	None
Search Type	The correlation search type: Data Model: Create a search based on a Splunk data model. Click Select to select a data model. Ad hoc: Create a search based on a custom search string that you provide. Predefined: Choose from a library of third-party search templates. For more information, see Correlation search templates in ITSI.	Ad hoc
Time range	The time range over which the correlation search applies.	Last 15 minutes

Association

Field	Description	Defaults
Service	Select one or more ITSI services to which this correlation search applies. You can only select services that belong to teams for which you have read access.	None
Entity Lookup Field	The field in the data retrieved by the correlation search that is used to look up corresponding entities. For example, host.	None

Schedule

Field	Description	Defaults
Schedule Type	Configure the schedule for the correlation search: Basic: Schedule searches to run at regular intervals. Configure the search interval in the Run Every menu. Cron: Schedule searches to run periodically at fixed times, dates, or intervals. Enter a schedule in Cron Schedule. For more information, see Use cron expressions for scheduling in the Alerting Manual	Basic, Every 5 minutes

Notable Events

Use this section to configure the notable event that is generated when search results meet a specific condition. The Splunk platform indexes the event object like any other event. You can track, manage, and update notable events in Episode Review.

ITSI correlation searches support field substitution with tokens in the format %fieldname%. Use field substitution to map third-party alert field values to corresponding notable event fields. See Ingest third-party alerts into ITSI for specific examples.

Field	Description	Defaults
Notable Event Title	The title of the notable event in Episode Review. For example, mysql-01 server cpu Load %.	None
Notable Event Description	A brief phrase to describe the notable event. For example, "This alert triggers when DB CPU load on the mysql-01 server reaches 80%."	None
Owner	The ITSI role to which the notable event is assigned in Episode Review. If using advanced mode, the value must resolve to a username in the system.	Unassigned
Severity	The level of importance of the event. If using advanced mode, the values must resolve to an integer specified in the default version of itsi_notable_event_severity.conf (or the local version if you created one).	1 - Info 2 - Normal 3 - Low 4 - Medium 5 - High 6 - Critical
Status	The triage status of the event in Episode Review. If using advanced mode, the values must resolve to an integer specified in the default version of itsi_notable_event_status.conf (or the local version if you created one).	0 - Unassigned 1 - New 2 - In Progress 3 - Pending 4 - Resolved 5 - Closed
Instructions	Specific instructions for how to handle the notable event. You must use token replacement to substitute the value of a third-party alert field. When you configure aggregation policies you can specify how you want to display these instructions in Episode Review.	%itsi_instruction%
Drilldown Search Name	Include a drilldown to a specific Splunk search. The drilldown links are displayed on the All Events tab of an episode you select in Episode Review, under the Drilldown Search column.	None
Drilldown Search	The Splunk search you drill down to.	None
Drilldown earliest offset	Defines how far back from the time of the event to start looking for related events.	Last 5 minutes
Drilldown latest offset	Defines how far ahead from the time of the event to look for related events.	Next 5 minutes
Notable Event Identifier Fields	Set of fields used to determine whether a notable event is unique. These identifier fields form the event hash field, which is added to every notable event to help identify unique alarm types. This field supports field substitution in the format %fieldname%.	source
Drilldown Website Name	You can drill down to a specific website from the Overview tab of Episode Review. Set the name of the drilldown website link.	None
Drilldown Website URL	The website you drill down to.	None

Advanced Options

Throttling

When correlation search results meet specific conditions, the search generates a new notable event (alert). A correlation search can create multiple alerts for the same condition. In most cases, it is best to have a single alert for the same condition. Throttling blocks the correlation search from creating duplicate alerts for the same issue every time it runs.

If you group by one or more fields, throttling is applied to each unique field-value combination. For example, throttling by host once a day means that only one notable event of this type is created per day, per server.

Throttling applies to any correlation search alert type, including notable events and actions (RSS feed, email, run script, and ticketing).

Field	Description
Suppress Period	During the suppress period, any additional event that matches any of the Fields to group by does not create a new alert. After the suppress period passes, the next matching event creates a new alert, and throttling conditions resume. Enter a relative time range in seconds. For example "60s" (60 seconds).
Fields to group by	Fields to compare for similar events. For example, cpu_load_percent. During the suppress period, any additional matches in the correlation search results are compared to the fields defined here. If a field matches, it stops the creation of a new alert. You can define multiple fields. The fields available depend on the search fields that the correlation search returns.

Actions

Actions are other alert types that a correlation search can trigger. You configure action alerts independently from other alert types, such as Notable Events and Risk Scoring.

Action	Description
Include in RSS feed	Posts the correlation search alert on the Splunk Enterprise RSS feed.
Send email	Sends an email about the correlation search alert. Email subject: The email subject defaults to "Splunk Alert: $name$", where $name$ is the correlation search Search Name. Email address(es): Insert email addresses and/or distribution lists that should receive the alert. Include entity information: Appends entity information to the subject of the email. Include results in email: Adds the correlation search results in the body of the email in the format you specify. Attach results in a PDF: Includes the correlation search results as a PDF attachment. The schedule_search capability and the admin_all_objects capability are required for PDF delivery scheduling. Attach results in a CSV: Includes the correlation search results as a CSV attachment. Note: Email actions require that you configure the mail server in Splunk Enterprise. See Configure email notification settings in the Alerting Manual.
Run a script	Triggers a shell script. See Configure a script for an alert action in the Alerting Manual.