Ingest third-party alerts into ITSI with correlation searches

You can ingest alerts from third-party products, such as Nagios and SCOM, into IT Service Intelligence (ITSI) as notable events. You can then manage those notable events in Episode Review. The ability to ingest third-party alerts as notable events lets you use ITSI as an efficient central management console, where you can view, triage, and investigate alerts from integrated products.

Universal Alerting is part of the Content Pack for Monitoring and Alerting and provides a reusable way to ingest third party alerts into ITSI, without the need to create correlation searches or notable event aggregation policies.

For more information about Universal Alerting, see About Universal Alerting in the Content Pack for Monitoring and Alerting.

For more information about correlation searches, see Overview of correlation searches in ITSI.

Prerequisite

Requirement	Description
Ingest the event data into ITSI	Before you can normalize and correlate external events in ITSI, you must first ingest the event data into Splunk. There are various methods to get third-party alerts into Splunk. It is recommended that for each alert sender, you have a different source type. Use any of the following methods to ingest alerts into ITSI: Set up an email receiver bot. Email alerts are received, parsed, and stored to disk in a directory where a Splunk universal forwarder monitors and collects the data. Set up a polling mechanism that uses the third-party service's API calls to fetch new alerts every minute and store this information into Splunk. Use a Splunk universal forwarder to monitor the application logs on the alert-generating servers. The universal forwarder catches the alert in the log files when the tool fires alerts.
ITSI role	You must have the write_itsi_correlation_search capability to ingest third-party alerts into ITSI. The itoa_admin and itoa_team_admin ITSI roles have this capabilities by default.

Steps

Configure a correlation search to normalize third-party alerts as ITSI notable events. A correlation search is an ad hoc Splunk search that enriches, normalizes, and deduplicates raw alerts. Each resulting row from the search becomes a notable event in ITSI.

ITSI correlation searches support field substitution with tokens in the format %fieldname%. Use field substitution to map third-party alert field values to corresponding notable event fields.

Token variables cannot be the same as correlation search field names. For example, you cannot use "title," "description," "owner," "severity," "status," and so on, as variables. Instead use token variables such as %orig_description%, %orig_owner%, %orig_severity%, %orig_status%, and so on.

1. In ITSI, click Configuration > Correlation Searches.
2. Click Create New Search > Create Correlation Search.
3. Configure a correlation search to ingest third party alerts.

Search Properties

Field name	Description	Defaults
Search Name	A name that describes the correlation search. For example, "nagios_alert".	None
Description	A description of the type of issue the search is intended to detect.	None
Search Type	Enter an ad hoc search string with appropriate eval command expressions to map fields in your third-party alert data to ITSI notable event fields. For example, the following ad hoc search maps Nagios severity-level fields to ITSI severity-level fields: index="main" sourcetype="csv" | head 100 | eval orig_severity=case(severity=="OK", 3, severity=="WARNING", 4, severity=="CRITICAL", 6, severity=="UNKNOWN", 5) Alternatively, click Predefined and choose from a library of third-party search templates. For more information, see Correlation search templates in ITSI.	Ad hoc
Time range	The time range over which the correlation search applies. Choose Relative and run the search over the last 1 minute or last 5 minutes.	Last 15 minutes

Association

Field name	Description	Defaults
Service	The ITSI services to which this correlation search applies. Leave this field blank.	None
Entity Lookup Field	The field in the data retrieved by the correlation search that is used to look up corresponding entities. This is usually a host or node field (for example, itsiNode). If the host or node of the alert is listed as a filtered entity in one or more ITSI services, these service names are added to the notable event as "affected service(s)". The entity lookup field must be an actual entity alias field for the match to occur. For example, if the entity was created with an alias of hostname but the entity lookup field in the correlation search as a value of myhost, the entity is not matched. The entity lookup field must be hostname as well. You must provide a field in order to see a list of impacted entities in corresponding episodes.	None

Schedule

Field name	Description	Defaults
Schedule Type	Choose a Basic schedule. Configure the search interval in the Run Every menu to run every 1 minute or 5 minutes (depending on the time range above).	Basic, Every 5 minutes

Notable Events

Use this section to configure the notable event that is generated when search results meet a specific condition. The Splunk platform indexes the event object like any other event. The events are displayed in Episode Review. After you configure a correlation search, create an aggregation policy to determine how to individual events are grouped into episodes. For more information, see Overview of aggregation policies in ITSI.

ITSI correlation searches support field substitution with tokens in the format %fieldname%. Use field substitution to map third-party alert field values to corresponding notable event fields. To see the available fields that you can use as tokens, click Run Search under Search Properties. You can use any of the fields listed in the left panel.

Field name	Description	Defaults
Notable Event Title	The title of the notable event in Episode Review. For example, you might enter %name% to substitute Nagios alert names as notable event titles.	None
Notable Event Description	A brief phrase to describe the notable event. For example, you might enter %reason% to substitute Nagios descriptions for notable event descriptions.	None
Owner	The ITSI role to which the notable event is assigned in Episode Review. Click Advanced Mode, then enter the owner using a token in the format %fieldname%. For example, %owner_field%. In advanced mode, owners must match the names of owners in the system.	Unassigned
Severity	The level of importance of the event. Click Advanced Mode, then provide a token in the format %fieldname% to substitute the value of a third-party alert field. Values must match an integer specified in the default version of itsi_notable_event_severity.conf (or the local version if you created one).	1 - Info 2 - Normal 3 - Low 4 - Medium 5 - High 6 - Critical
Status	The triage status of the event in Episode Review. Click Advanced Mode, then provide a token in the format %fieldname% to substitute the value of a third-party alert field. Values must match an integer specified in the default version of itsi_notable_event_status.conf (or the local version if you created one). For incoming third-party alerts the triage status is generally "New".	0 - Unassigned 1 - New 2 - In Progress 3 - Pending 4 - Resolved 5 - Closed
Instructions	Specific instructions for how to handle the notable event. Static instructions aren't supported. You must use token replacement to substitute the value of a third-party alert field. When you configure aggregation policies you can specify how you want to display these instructions in Episode Review.	%itsi_instruction%
Drilldown Name	You can drill down to a specific Splunk search from the All Events tab of Episode Review. Set the name of the drilldown search link.	None
Drilldown Search	The search you drill down to.	None
Drilldown earliest offset	Defines how far back from the time of the event to start looking for related events.	Last 5 minutes
Drilldown latest offset	Defines how far ahead from the time of the event to look for related events.	Next 5 minutes
Notable Event Identifier Fields	Determine whether a notable event is unique or not, which drives the events timeline. These identifier fields form the event hash field, which is added to every notable event to help identify unique alarm types. Use the exact same fields you use to dedup on in a Splunk search. These fields must exist in the raw event, otherwise a notable event isn't created. You must provide the correct fields for the Impacted Entities section of associated episodes to populate correctly.	source
Drilldown link title	You can drill down to a specific website from the All Events tab of Episode Review. Set the name of the drilldown website link.	None
Drilldown link URI	The website you drill down to.	None

4. Click Save.

The correlation search runs as scheduled, mapping third-party alert field values to ITSI notable event fields. Third-party alerts appear as notable events in Episode Review. The events are collected in the itsi_tracked_alerts index.

Directly push events into ITSI through HEC

Rather than ingesting external alerts through a correlation search, you can also use HTTP Event Collector (HEC) to proactively push data directly into ITSI as notable events directly into the itsi_tracked_alerts index.

1. In Splunk Web, go to Settings > Data Inputs.
2. Click HTTP Event Collector.
3. Copy the token value for the Auto Generated ITSI Event Management Token.
4. Run a command similar to the following example to bring external data directly into ITSI and format it as a notable event:

   curl -k https://localhost:8088/services/collector/event -H "Authorization: Splunk <token value>" -d '{"event" : {"event_id" : "< unique GUID>", "title" : "Disk 90% Full", "status" : "4", "severity" : "6", "owner" : "unassigned", "description": "Disk is almost full", "other_field" : "more stuff"}}'