Splunk® IT Service Intelligence

Event Analytics Manual

Acrobat logo Download manual as PDF


Acrobat logo Download topic as PDF

Generate events with correlation searches in ITSI

A correlation search is a recurring search within IT Service Intelligence (ITSI) that scans multiple data sources for defined patterns. You can configure a correlation search to generate a notable event when search results meet specific conditions. Use Episode Review to review notable events that your correlation searches generate and to initiate the investigative process of determining root cause.

Configure correlation searches to update the settings associated with how they run, change the search logic, and throttle alerts. See Overview of correlation searches in ITSI to learn more about correlation searches.

Prerequisites

Requirement Description
ITSI role You must have the write_itsi_correlation_search capability to create a correlation search. The itoa_admin and itoa_team_admin ITSI roles have this capabilities by default.

Steps

  1. From the ITSI main menu, click Configuration > Correlation Searches.
  2. Click Create New Search > Create Correlation Search.
  3. Configure the following fields:

Search Properties

Field Description Defaults
Search Name A name that describes the correlation search. For example, "cpu_load_percent". None
Description (Optional) A description of the type of issue the search is intended to detect. None
Search Type The correlation search type:
    • Data Model: Create a search based on a Splunk data model. Click Select to select a data model.
    • Ad hoc: Create a search based on a custom search string that you provide.
    • Predefined: Choose from a library of third-party search templates. For more information, see Correlation search templates in ITSI.
Ad hoc
Time range The time range over which the correlation search applies. Last 15 minutes

Association

Field Description Defaults
Service Select one or more ITSI services to which this correlation search applies. You can only select services that belong to teams for which you have read access. None
Entity Lookup Field The field in the data retrieved by the correlation search that is used to look up corresponding entities. For example, host. None

Schedule

Field Description Defaults
Schedule Type Configure the schedule for the correlation search:
  • Basic: Schedule searches to run at regular intervals. Configure the search interval in the Run Every menu.
  • Cron: Schedule searches to run periodically at fixed times, dates, or intervals. Enter a schedule in Cron Schedule. For more information, see Use cron expressions for scheduling in the Alerting Manual
Basic, Every 5 minutes

Notable Events

Use this section to configure the notable event that is generated when search results meet a specific condition. The Splunk platform indexes the event object like any other event. You can track, manage, and update notable events in Episode Review.

ITSI correlation searches support field substitution with tokens in the format %fieldname%. Use field substitution to map third-party alert field values to corresponding notable event fields. See Ingest third-party alerts into ITSI for specific examples.

Field Description Defaults
Notable Event Title The title of the notable event in Episode Review. For example, mysql-01 server cpu Load %. None
Notable Event Description A brief phrase to describe the notable event. For example, "This alert triggers when DB CPU load on the mysql-01 server reaches 80%." None
Owner The ITSI role to which the notable event is assigned in Episode Review. If using advanced mode, the value must resolve to a username in the system. Unassigned
Severity The level of importance of the event.


If using advanced mode, the values must resolve to an integer specified in the default version of itsi_notable_event_severity.conf (or the local version if you created one).

1 - Info
2 - Normal
3 - Low
4 - Medium
5 - High
6 - Critical
Status The triage status of the event in Episode Review.


If using advanced mode, the values must resolve to an integer specified in the default version of itsi_notable_event_status.conf (or the local version if you created one).

0 - Unassigned
1 - New
2 - In Progress
3 - Pending
4 - Resolved
5 - Closed
Instructions Specific instructions for how to handle the notable event. You must use token replacement to substitute the value of a third-party alert field. When you configure aggregation policies you can specify how you want to display these instructions in Episode Review. %itsi_instruction%
Drilldown Search Name Include a drilldown to a specific Splunk search. The drilldown links are displayed on the All Events tab of an episode you select in Episode Review, under the Drilldown Search column. None
Drilldown Search The Splunk search you drill down to. None
Drilldown earliest offset Defines how far back from the time of the event to start looking for related events. Last 5 minutes
Drilldown latest offset Defines how far ahead from the time of the event to look for related events. Next 5 minutes
Notable Event Identifier Fields Set of fields used to determine whether a notable event is unique. These identifier fields form the event hash field, which is added to every notable event to help identify unique alarm types. This field supports field substitution in the format %fieldname%. source
Drilldown Website Name You can drill down to a specific website from the Overview tab of Episode Review. Set the name of the drilldown website link. None
Drilldown Website URL The website you drill down to. None

Advanced Options

Throttling

When correlation search results meet specific conditions, the search generates a new notable event (alert). A correlation search can create multiple alerts for the same condition. In most cases, it is best to have a single alert for the same condition. Throttling blocks the correlation search from creating duplicate alerts for the same issue every time it runs.

If you group by one or more fields, throttling is applied to each unique field-value combination. For example, throttling by host once a day means that only one notable event of this type is created per day, per server.

Throttling applies to any correlation search alert type, including notable events and actions (RSS feed, email, run script, and ticketing).

Field Description
Suppress Period During the suppress period, any additional event that matches any of the Fields to group by does not create a new alert. After the suppress period passes, the next matching event creates a new alert, and throttling conditions resume. Enter a relative time range in seconds. For example "60s" (60 seconds).
Fields to group by Fields to compare for similar events. For example, cpu_load_percent.

During the suppress period, any additional matches in the correlation search results are compared to the fields defined here. If a field matches, it stops the creation of a new alert. You can define multiple fields. The fields available depend on the search fields that the correlation search returns.

Actions

Actions are other alert types that a correlation search can trigger. You configure action alerts independently from other alert types, such as Notable Events and Risk Scoring.

Action Description
Include in RSS feed Posts the correlation search alert on the Splunk Enterprise RSS feed.
Send email Sends an email about the correlation search alert.
    • Email subject: The email subject defaults to "Splunk Alert: $name$", where $name$ is the correlation search Search Name.
    • Email address(es): Insert email addresses and/or distribution lists that should receive the alert.
    • Include entity information: Appends entity information to the subject of the email.
    • Include results in email: Adds the correlation search results in the body of the email in the format you specify.
    • Attach results in a PDF: Includes the correlation search results as a PDF attachment.

    The schedule_search capability and the admin_all_objects capability are required for PDF delivery scheduling.

    • Attach results in a CSV: Includes the correlation search results as a CSV attachment.

    Note: Email actions require that you configure the mail server in Splunk Enterprise. See Configure email notification settings in the Alerting Manual.

Run a script Triggers a shell script. See Configure a script for an alert action in the Alerting Manual.
Last modified on 28 April, 2023
PREVIOUS
Overview of correlation searches in ITSI
  NEXT
Ingest third-party alerts into ITSI

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.11.0, 4.11.1, 4.11.2, 4.11.3, 4.11.4, 4.11.5, 4.11.6, 4.12.0 Cloud only, 4.12.1 Cloud only, 4.12.2 Cloud only, 4.13.0, 4.13.1, 4.13.2, 4.13.3, 4.14.0 Cloud only, 4.14.1 Cloud only, 4.14.2 Cloud only, 4.15.0, 4.15.1, 4.15.2, 4.15.3, 4.16.0 Cloud only, 4.17.0, 4.17.1, 4.18.0, 4.18.1


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters