Generate events with correlation searches in ITSI
A correlation search is a recurring search within IT Service Intelligence (ITSI) that scans multiple data sources for defined patterns. You can configure a correlation search to generate a notable event when search results meet specific conditions. Use Episode Review to review notable events that your correlation searches generate and to initiate the investigative process of determining root cause.
Configure correlation searches to update the settings associated with how they run, change the search logic, and throttle alerts. See Overview of correlation searches in ITSI to learn more about correlation searches.
Prerequisites
Requirement | Description |
---|---|
ITSI role | You must have the write_itsi_correlation_search capability to create a correlation search. The itoa_admin and itoa_team_admin ITSI roles have this capabilities by default. |
Steps
- From the ITSI main menu, click Configuration > Correlation Searches.
- Click Create New Search > Create Correlation Search.
- Configure the following fields:
Search Properties
Field | Description | Defaults |
---|---|---|
Search Name | A name that describes the correlation search. For example, "cpu_load_percent". | None |
Description | (Optional) A description of the type of issue the search is intended to detect. | None |
Search Type | The correlation search type:
|
Ad hoc
|
Time range | The time range over which the correlation search applies. | Last 15 minutes
|
Association
Field | Description | Defaults |
---|---|---|
Service | Select one or more ITSI services to which this correlation search applies. You can only select services that belong to teams for which you have read access. | None |
Entity Lookup Field | The field in the data retrieved by the correlation search that is used to look up corresponding entities. For example, host .
|
None |
Schedule
Field | Description | Defaults |
---|---|---|
Schedule Type | Configure the schedule for the correlation search:
|
Basic , Every 5 minutes
|
Notable Events
Use this section to configure the notable event that is generated when search results meet a specific condition. The Splunk platform indexes the event object like any other event. You can track, manage, and update notable events in Episode Review.
ITSI correlation searches support field substitution with tokens in the format %fieldname%
. Use field substitution to map third-party alert field values to corresponding notable event fields. See Ingest third-party alerts into ITSI for specific examples.
Field | Description | Defaults |
---|---|---|
Notable Event Title | The title of the notable event in Episode Review. For example, mysql-01 server cpu Load % .
|
None |
Notable Event Description | A brief phrase to describe the notable event. For example, "This alert triggers when DB CPU load on the mysql-01 server reaches 80%." | None |
Owner | The ITSI role to which the notable event is assigned in Episode Review. If using advanced mode, the value must resolve to a username in the system. | Unassigned
|
Severity | The level of importance of the event.
|
1 - Info2 - Normal3 - Low4 - Medium5 - High6 - Critical
|
Status | The triage status of the event in Episode Review.
|
0 - Unassigned1 - New2 - In Progress3 - Pending4 - Resolved5 - Closed
|
Instructions | Specific instructions for how to handle the notable event. You must use token replacement to substitute the value of a third-party alert field. When you configure aggregation policies you can specify how you want to display these instructions in Episode Review. | %itsi_instruction%
|
Drilldown Search Name | Include a drilldown to a specific Splunk search. The drilldown links are displayed on the All Events tab of an episode you select in Episode Review, under the Drilldown Search column. | None |
Drilldown Search | The Splunk search you drill down to. | None |
Drilldown earliest offset | Defines how far back from the time of the event to start looking for related events. | Last 5 minutes
|
Drilldown latest offset | Defines how far ahead from the time of the event to look for related events. | Next 5 minutes
|
Notable Event Identifier Fields | Set of fields used to determine whether a notable event is unique. These identifier fields form the event hash field, which is added to every notable event to help identify unique alarm types. This field supports field substitution in the format %fieldname% .
|
source
|
Drilldown Website Name | You can drill down to a specific website from the Overview tab of Episode Review. Set the name of the drilldown website link. | None |
Drilldown Website URL | The website you drill down to. | None |
Advanced Options
Throttling
When correlation search results meet specific conditions, the search generates a new notable event (alert). A correlation search can create multiple alerts for the same condition. In most cases, it is best to have a single alert for the same condition. Throttling blocks the correlation search from creating duplicate alerts for the same issue every time it runs.
If you group by one or more fields, throttling is applied to each unique field-value combination. For example, throttling by host once a day means that only one notable event of this type is created per day, per server.
Throttling applies to any correlation search alert type, including notable events and actions (RSS feed, email, run script, and ticketing).
Field | Description |
---|---|
Suppress Period | During the suppress period, any additional event that matches any of the Fields to group by does not create a new alert. After the suppress period passes, the next matching event creates a new alert, and throttling conditions resume. Enter a relative time range in seconds. For example "60s" (60 seconds). |
Fields to group by | Fields to compare for similar events. For example, cpu_load_percent .
During the suppress period, any additional matches in the correlation search results are compared to the fields defined here. If a field matches, it stops the creation of a new alert. You can define multiple fields. The fields available depend on the search fields that the correlation search returns. |
Actions
Actions are other alert types that a correlation search can trigger. You configure action alerts independently from other alert types, such as Notable Events and Risk Scoring.
Action | Description |
---|---|
Include in RSS feed | Posts the correlation search alert on the Splunk Enterprise RSS feed. |
Send email | Sends an email about the correlation search alert.
The schedule_search capability and the admin_all_objects capability are required for PDF delivery scheduling. Note: Email actions require that you configure the mail server in Splunk Enterprise. See Configure email notification settings in the Alerting Manual. |
Run a script | Triggers a shell script. See Configure a script for an alert action in the Alerting Manual. |
Overview of correlation searches in ITSI | Normalize alerts with correlation search templates in ITSI |
This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.11.0, 4.11.1, 4.11.2, 4.11.3, 4.11.4, 4.11.5, 4.11.6, 4.12.0 Cloud only, 4.12.1 Cloud only, 4.12.2 Cloud only, 4.13.0, 4.13.1, 4.13.2, 4.13.3, 4.14.0 Cloud only, 4.14.1 Cloud only, 4.14.2 Cloud only, 4.15.0, 4.15.1, 4.15.2, 4.15.3, 4.16.0 Cloud only, 4.17.0, 4.17.1, 4.18.0, 4.18.1, 4.19.0, 4.19.1
Feedback submitted, thanks!