Ingest third-party alerts as ITSI notable events

ITSI lets you ingest alerts from third-party products, such as Nagios and SCOM, as notable events. You can then manage those notable events in Episode Review. The ability to ingest third-party alerts as notable events lets you use ITSI as an efficient central management console, where you can view, triage, and investigate alerts from integrated products.

Use a custom integration script

ITSI provides custom scripts with specific onboarding instructions for you to ingest alerts from common third-party systems. Each blog post provides a script with a custom correlation search and notable event aggregation policy that you can import directly into ITSI. If you use one of these custom scripts, you do not need to follow the regular onboarding steps below.

ITSI provides the following custom onboarding instructions:

• Ingest Netcool Alerts into Splunk ITSI Event Analytics
• Ingest Nagios Alarms into Splunk ITSI Event Analytics
• Ingest AppDynamics and New Relic Alerts into Splunk ITSI Event Analytics

Step 1: Ingest alerts into ITSI

There are various methods to get third-party alerts into ITSI. It is recommended that for each alert sender, you have a different source type. Use any of the following methods to ingest alerts into ITSI:

• Set up an email receiver bot. Email alerts are received, parsed, and stored to disk in a directory where a Splunk universal forwarder monitors and collects the data.
• Set up a polling mechanism that uses the third-party service's API calls to fetch new alerts every minute and store this information into Splunk.
• Use a Splunk universal forwarder to monitor the application logs on the alert-generating servers. The universal forwarder catches the alert in the log files when the tool fires alerts.

Step 2: Normalize the alert data

After you ingest alerts into ITSI, you must normalize the fields so that they display properly in Episode Review. Use an ITSI correlation search to normalize third-party alerts as ITSI notable events. A correlation search is an ad hoc Splunk search that enriches, normalizes, and deduplicates raw alerts. Each resulting row from the search becomes an ITSI notable event.

ITSI correlation searches support field substitution with tokens in the format %fieldname%. Use field substitution to map third-party alert field values to corresponding notable event fields.

Token variables cannot be the same as correlation search field names. For example, you cannot use "title," "description," "owner," "severity," "status," and so on, as variables. Instead use token variables such as %orig_description%, %orig_owner%, %orig_severity%, %orig_status%, and so on.

1. In ITSI, click Configure > Correlation Searches.
2. Click Create New Search > Create Correlation Search.
3. Configure a correlation search to ingest third party alerts.

   Search Properties

   Field name	Description	Defaults
   Search Name	A name that describes the correlation search. For example, "nagios_alert".	None
   Description	A description of the type of issue the search is intended to detect.	None
   Search Type	Enter an ad hoc search string with appropriate eval command expressions to map fields in your third-party alert data to ITSI notable event fields. For example, the following ad hoc search maps Nagios severity-level fields to ITSI severity-level fields: index="main" sourcetype="csv" | head 100 | eval orig_severity=case(severity=="OK", 3, severity=="WARNING", 4, severity=="CRITICAL", 6, severity=="UNKNOWN", 5)	Ad hoc
   Time range	The time range over which the correlation search applies. Choose Relative and run the search over the last 1 minute or last 5 minutes.	Last 15 minutes

   Association

   Field name	Description	Defaults
   Service	The ITSI services to which this correlation search applies. Leave this field blank.	None
   Entity Lookup Field	The field in the data retrieved by the correlation search that is used to look up corresponding entities. This is usually a host or node field (for example, itsiNode). If the host or node of the alert is listed as a filtered entity in one or more ITSI services, these service names are added to the notable event as "affected service(s)". The entity lookup field must be an actual entity alias field for the match to occur. For example, if the entity was created with an alias of hostname but the entity lookup field in the correlation search as a value of myhost, the entity is not matched. The entity lookup field must be hostname as well.	None

   Schedule

   Field name	Description	Defaults
   Schedule Type	Choose a Basic schedule. Configure the search interval in the Run Every menu to run every 1 minute or 5 minutes (depending on the time range above).	Basic, Every 5 minutes

   Notable Events

   Use this section to configure the notable event that is generated when search results meet a specific condition. The Splunk platform indexes the event object like any other event. You can track, manage, and update notable events in Episode Review.

   ITSI correlation searches support field substitution with tokens in the format %fieldname%. Use field substitution to map third-party alert field values to corresponding notable event fields. To see the available fields that you can use as tokens, click Run Search under Search Properties. You can use any of the fields listed in the left panel.

   Field name	Description	Defaults
   Notable Event Title	The title of the notable event in Episode Review. For example, you might enter %name% to substitute Nagios alert names as notable event titles.	None
   Notable Event Description	A brief phrase to describe the notable event. For example, you might enter %reason% to substitute Nagios descriptions for notable event descriptions.	None
   Owner	The ITSI role to which the notable event is assigned in Episode Review. Click Advanced Mode, then enter the owner using a token in the format %fieldname%. For example, %owner_field%. In advanced mode, owners must match the names of owners in the system.	Unassigned
   Severity	The level of importance of the event. Click Advanced Mode, then provide a token in the format %fieldname% to substitute the value of a third-party alert field. Values must match an integer specified in the default version of itsi_notable_event_severity.conf (or the local version if you created one).	1 - Info 2 - Normal 3 - Low 4 - Medium 5 - High 6 - Critical
   Status	The triage status of the event in Episode Review. Click Advanced Mode, then provide a token in the format %fieldname% to substitute the value of a third-party alert field. Values must match an integer specified in the default version of itsi_notable_event_status.conf (or the local version if you created one). For incoming third-party alerts the triage status is generally "New".	0 - Unassigned 1 - New 2 - In Progress 3 - Pending 4 - Resolved 5 - Closed
   Drilldown Name	You can drill down to a specific Splunk search from the All Events tab of Episode Review. Set the name of the drilldown search link.	None
   Drilldown Search	The search you drill down to.	None
   Drilldown earliest offset	Defines how far back from the time of the event to start looking for related events.	Last 5 minutes
   Drilldown latest offset	Defines how far ahead from the time of the event to look for related events.	Next 5 minutes
   Notable Event Identifier Fields	Determine whether a notable event is unique or not, which drives the events timeline. These identifier fields form the event hash field, which is added to every notable event to help identify unique alarm types.	source
   Drilldown link title	You can drill down to a specific website from the All Events tab of Episode Review. Set the name of the drilldown website link.	None
   Drilldown link URI	The website you drill down to.	None

4. Click Save.

The correlation search runs as scheduled, mapping third-party alert field values to ITSI notable event fields. Third-party alerts appear as notable events in Episode Review. The events are collected in the itsi_tracked_alerts index.

Step 3: Aggregate the events

After you ingest and normalize third-party alerts, they are collected in the itsi_tracked_alerts index as notable events and are ready to be grouped into episodes.

A notable event aggregation policy called Splunk App for Infrastructure is delivered with ITSI. This aggregation policy groups all third-party alerts (including alerts from the Splunk App for Infrastructure) that use ITSI normalized fields. Enable this aggregation policy or configure your own aggregation policy to group third-party alert data.

1. In ITSI, click Configure > Notable Event Aggregation Policies.
2. Click New Notable Event Aggregation Policy, or configure the Default Policy. See Notable event aggregation policies overview for ITSI.

For example, you might configure the Default Policy by enabling Smart Mode, then selecting several notable events to cluster on (such as itsiNode, itsiAlertId, Location, and OS_version).

After you enable an aggregation policy, episodes start appearing in Episode Review.

ITSI can also take automated actions at the beginning or end of an episode. Consider what makes the most sense for a given type of alert. Alert actions might include opening a ServiceNow ticket, sending an email, or running a custom script. For more information, see Take action on an episode in ITSI.