Splunk® IT Service Intelligence

Administer Splunk IT Service Intelligence

Download manual as PDF

Download topic as PDF

Ingest third-party alerts as ITSI notable events

ITSI lets you ingest alerts from third-party products, such as Nagios and SCOM, as notable events. You can then manage those notable events in Episode Review. The ability to ingest third-party alerts as notable events lets you use ITSI as an efficient central management console, where you can view, triage, and investigate alerts from integrated products.

Use a custom integration script

ITSI provides custom scripts with specific onboarding instructions for you to ingest alerts from common third-party systems. Each blog post provides a script with a custom correlation search and notable event aggregation policy that you can import directly into ITSI. If you use one of these custom scripts, you do not need to follow the regular onboarding steps below.

ITSI provides the following custom onboarding instructions:

Step 1: Ingest alerts into ITSI

There are various methods to get third-party alerts into ITSI. It is recommended that for each alert sender, you have a different source type. Use any of the following methods to ingest alerts into ITSI:

    • Set up an email receiver bot. Email alerts are received, parsed, and stored to disk in a directory where a Splunk universal forwarder monitors and collects the data.
    • Set up a polling mechanism that uses the third-party service's API calls to fetch new alerts every minute and store this information into Splunk.
    • Use a Splunk universal forwarder to monitor the application logs on the alert-generating servers. The universal forwarder catches the alert in the log files when the tool fires alerts.

Step 2: Normalize the alert data

After you ingest alerts into ITSI, you must normalize the fields so that they display properly in Episode Review. Use an ITSI correlation search to normalize third-party alerts as ITSI notable events. A correlation search is an ad hoc Splunk search that enriches, normalizes, and deduplicates raw alerts. Each resulting row from the search becomes an ITSI notable event.

ITSI correlation searches support field substitution with tokens in the format %fieldname%. Use field substitution to map third-party alert field values to corresponding notable event fields.

Token variables cannot be the same as correlation search field names. For example, you cannot use "title," "description," "owner," "severity," "status," and so on, as variables. Instead use token variables such as %orig_description%, %orig_owner%, %orig_severity%, %orig_status%, and so on.

  1. In ITSI, click Configure > Correlation Searches.
  2. Click Create New Search > Create Correlation Search.
  3. Configure a correlation search to ingest third party alerts.

    Search Properties

    Field name Description Defaults
    Search Name A name that describes the correlation search. For example, "nagios_alert". None
    Description A description of the type of issue the search is intended to detect. None
    Search Type Enter an ad hoc search string with appropriate eval command expressions to map fields in your third-party alert data to ITSI notable event fields. For example, the following ad hoc search maps Nagios severity-level fields to ITSI severity-level fields:

    index="main" sourcetype="csv" | head 100 | eval orig_severity=case(severity=="OK", 3, severity=="WARNING", 4, severity=="CRITICAL", 6, severity=="UNKNOWN", 5)

    Ad hoc
    Time range The time range over which the correlation search applies. Choose Relative and run the search over the last 1 minute or last 5 minutes. Last 15 minutes

    Association

    Field name Description Defaults
    Service The ITSI services to which this correlation search applies. Leave this field blank. None
    Entity Lookup Field The field in the data retrieved by the correlation search that is used to look up corresponding entities. This is usually a host or node field (for example, itsiNode). If the host or node of the alert is listed as a filtered entity in one or more ITSI services, these service names are added to the notable event as "affected service(s)".


    
The entity lookup field must be an actual entity alias field for the match to occur. For example, if the entity was created with an alias of hostname but the entity lookup field in the correlation search as a value of myhost, the entity is not matched. The entity lookup field must be hostname as well.

    None

    Schedule

    Field name Description Defaults
    Schedule Type Choose a Basic schedule. Configure the search interval in the Run Every menu to run every 1 minute or 5 minutes (depending on the time range above). Basic, Every 5 minutes

    Notable Events

    Use this section to configure the notable event that is generated when search results meet a specific condition. The Splunk platform indexes the event object like any other event. You can track, manage, and update notable events in Episode Review.

    ITSI correlation searches support field substitution with tokens in the format %fieldname%. Use field substitution to map third-party alert field values to corresponding notable event fields. To see the available fields that you can use as tokens, click Run Search under Search Properties. You can use any of the fields listed in the left panel.

    Field name Description Defaults
    Notable Event Title The title of the notable event in Episode Review. For example, you might enter %name% to substitute Nagios alert names as notable event titles. None
    Notable Event Description A brief phrase to describe the notable event. For example, you might enter %reason% to substitute Nagios descriptions for notable event descriptions. None
    Owner The ITSI role to which the notable event is assigned in Episode Review. Click Advanced Mode, then enter the owner using a token in the format %fieldname%. For example, %owner_field%. In advanced mode, owners must match the names of owners in the system. Unassigned
    Severity The level of importance of the event. Click Advanced Mode, then provide a token in the format %fieldname% to substitute the value of a third-party alert field.

    Values must match an integer specified in the default version of itsi_notable_event_severity.conf (or the local version if you created one).

    1 - Info
    2 - Normal
    3 - Low
    4 - Medium
    5 - High
    6 - Critical
    Status The triage status of the event in Episode Review. Click Advanced Mode, then provide a token in the format %fieldname% to substitute the value of a third-party alert field.

    Values must match an integer specified in the default version of itsi_notable_event_status.conf (or the local version if you created one).

    For incoming third-party alerts the triage status is generally "New".

    0 - Unassigned
    1 - New
    2 - In Progress
    3 - Pending
    4 - Resolved
    5 - Closed
    Drilldown Name You can drill down to a specific Splunk search from the All Events tab of Episode Review. Set the name of the drilldown search link. None
    Drilldown Search The search you drill down to. None
    Drilldown earliest offset Defines how far back from the time of the event to start looking for related events. Last 5 minutes
    Drilldown latest offset Defines how far ahead from the time of the event to look for related events. Next 5 minutes
    Notable Event Identifier Fields Determine whether a notable event is unique or not, which drives the events timeline. These identifier fields form the event hash field, which is added to every notable event to help identify unique alarm types. source
    Drilldown link title You can drill down to a specific website from the All Events tab of Episode Review. Set the name of the drilldown website link. None
    Drilldown link URI The website you drill down to. None
  4. Click Save.

The correlation search runs as scheduled, mapping third-party alert field values to ITSI notable event fields. Third-party alerts appear as notable events in Episode Review. The events are collected in the itsi_tracked_alerts index.

Step 3: Aggregate the events

After you ingest and normalize third-party alerts, they are collected in the itsi_tracked_alerts index as notable events and are ready to be grouped into episodes.

A notable event aggregation policy called Splunk App for Infrastructure is delivered with ITSI. This aggregation policy groups all third-party alerts (including alerts from the Splunk App for Infrastructure) that use ITSI normalized fields. Enable this aggregation policy or configure your own aggregation policy to group third-party alert data.

  1. In ITSI, click Configure > Notable Event Aggregation Policies.
  2. Click New Notable Event Aggregation Policy, or configure the Default Policy. See Notable event aggregation policies overview for ITSI.

For example, you might configure the Default Policy by enabling Smart Mode, then selecting several notable events to cluster on (such as itsiNode, itsiAlertId, Location, and OS_version).

After you enable an aggregation policy, episodes start appearing in Episode Review.

ITSI can also take automated actions at the beginning or end of an episode. Consider what makes the most sense for a given type of alert. Alert actions might include opening a ServiceNow ticket, sending an email, or running a custom script. For more information, see Take action on an episode in ITSI.

PREVIOUS
Dispatch episode actions to a remote ITSI instance
  NEXT
Ingest SNMP traps in ITSI

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.2.0, 4.2.1, 4.2.2, 4.2.3, 4.3.0, 4.3.1, 4.4.0, 4.4.1


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters