Splunk® IT Service Intelligence

Administer Splunk IT Service Intelligence

Download manual as PDF

Download topic as PDF

Configure correlation searches in ITSI

A correlation search is a recurring search that scans multiple data sources for defined patterns. You can configure a correlation search to generate a notable event (alert) when search results meet specific conditions. Use Episode Review to review notable events that your correlation searches generate and to initiate the investigative process of determining root cause.

You can use an ITSI correlation search to ingest third-party alerts as ITSI notable events. If you are creating a correlation search to ingest alerts from a third-party product, such as Nagios or SCOM, see Ingest third-party alerts as ITSI notable events for specific instructions.

Configure correlation searches to update the settings associated with how they run, change the search logic, and throttle alerts. See Correlation search overview for ITSI to learn more about correlation searches.

Search Properties

Field Description Defaults
Search Name A name that describes the correlation search. For example, "cpu_load_percent". None
Description (Optional) A description of the type of issue the search is intended to detect. None
Search Type The correlation search type:
    • Data Model: Create a search based on a Splunk data model. Click Select to select a data model.
    • Ad hoc: Create a search based on a custom search string that you provide.
Ad hoc
Time range The time range over which the correlation search applies. Last 15 minutes

Association

Field Description Defaults
Service Select one or more ITSI services to which this correlation search applies. You can only select services that belong to teams for which you have read access. None
Entity Lookup Field The field in the data retrieved by the correlation search that is used to look up corresponding entities. For example, host. None

Schedule

Field Description Defaults
Schedule Type Configure the schedule for the correlation search:
  • Basic: Schedule searches to run at regular intervals. Configure the search interval in the Run Every menu.
  • Cron: Schedule searches to run periodically at fixed times, dates, or intervals. Enter a schedule in Cron Schedule. For more information, see Use cron expressions for scheduling in the Alerting Manual
Basic, Every 5 minutes

Notable Events

Use this section to configure the notable event that is generated when search results meet a specific condition. The Splunk platform indexes the event object like any other event. You can track, manage, and update notable events in Episode Review.

ITSI correlation searches support field substitution with tokens in the format %fieldname%. Use field substitution to map third-party alert field values to corresponding notable event fields. See Use ITSI correlation search to ingest alerts for specific examples.

Field Description Defaults
Notable Event Title The title of the notable event in Episode Review. For example, mysql-01 server cpu Load %. None
Notable Event Description A brief phrase to describe the notable event. For example, "This alert triggers when DB CPU load on the mysql-01 server reaches 80%." None
Owner The ITSI role to which the notable event is assigned in Episode Review. If using advanced mode, the value must resolve to a username in the system. Unassigned
Severity The level of importance of the event.


If using advanced mode, the values must resolve to an integer specified in the default version of itsi_notable_event_severity.conf (or the local version if you created one).

1 - Info
2 - Normal
3 - Low
4 - Medium
5 - High
6 - Critical
Status The triage status of the event in Episode Review.


If using advanced mode, the values must resolve to an integer specified in the default version of itsi_notable_event_status.conf (or the local version if you created one).

0 - Unassigned
1 - New
2 - In Progress
3 - Pending
4 - Resolved
5 - Closed
Drilldown Search Name You can drill down to a specific Splunk search from the Overview tab of Episode Review for an individual notable event or from the Grouped Events tab for a group of notable events. Set the name of the drilldown search link. None
Drilldown Search The search you drill down to. None
Drilldown earliest offset Defines how far back from the time of the event to start looking for related events. Last 5 minutes
Drilldown latest offset Defines how far ahead from the time of the event to look for related events. Next 5 minutes
Notable Event Identifier Fields Determine whether a notable event is unique or not. These identifier fields form the event hash field, which is added to every notable event to help identify unique alarm types. source
Drilldown Website Name You can drill down to a specific website from the Overview tab of Episode Review. Set the name of the drilldown website link. None
Drilldown Website URL The website you drill down to. None

Advanced Options

Throttling

When correlation search results meet specific conditions, the search generates a new notable event (alert). A correlation search can create multiple alerts for the same condition. In most cases, it is best to have a single alert for the same condition. Throttling blocks the correlation search from creating duplicate alerts for the same issue every time it runs.

If you group by one or more fields, throttling is applied to each unique field-value combination. For example, throttling by host once a day means that only one notable event of this type is created per day, per server.

Throttling applies to any correlation search alert type, including notable events and actions (RSS feed, email, run script, and ticketing).

Field Description
Suppress Period During the suppress period, any additional event that matches any of the Fields to group by does not create a new alert. After the suppress period passes, the next matching event creates a new alert, and throttling conditions resume. Enter a relative time range in seconds. For example "60s" (60 seconds).
Fields to group by Fields to compare for similar events. For example, cpu_load_percent.

During the suppress period, any additional matches in the correlation search results are compared to the fields defined here. If a field matches, it stops the creation of a new alert. You can define multiple fields. The fields available depend on the search fields that the correlation search returns.

Actions

Actions are other alert types that a correlation search can trigger. You configure action alerts independently from other alert types, such as Notable Events and Risk Scoring.

Action Description
Include in RSS feed Posts the correlation search alert on the Splunk Enterprise RSS feed.
Send email Sends an email about the correlation search alert.
    • Email subject: The email subject defaults to "Splunk Alert: $name$", where $name$ is the correlation search Search Name.
    • Email address(es): Insert email addresses and/or distribution lists that should receive the alert.
    • Include entity information: Appends entity information to the subject of the email.
    • Include results in email: Adds the correlation search results in the body of the email in the format you specify.
    • Attach results in a PDF: Includes the correlation search results as a PDF attachment.

    The schedule_search capability and the admin_all_objects capability are required for PDF delivery scheduling.

    • Attach results in a CSV: Includes the correlation search results as a CSV attachment.

    Note: Email actions require that you configure the mail server in Splunk Enterprise. See Configure email notification settings in the Alerting Manual.

Run a script Triggers a shell script. See Configure scripted alerts in the Alerting Manual.
PREVIOUS
Create correlation searches in ITSI
  NEXT
Create multi-KPI alerts in ITSI

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.2.0, 4.2.1, 4.2.2, 4.2.3, 4.3.0, 4.3.1, 4.4.0, 4.4.1


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters