Ingest third-party alerts into ITSI
You can ingest alerts from third-party products, such as Nagios and SCOM, into IT Service Intelligence (ITSI) as notable events. You can then manage those notable events in Episode Review. The ability to ingest third-party alerts as notable events lets you use ITSI as an efficient central management console, where you can view, triage, and investigate alerts from integrated products.
Universal Alerting is part of the Content Pack for Monitoring and Alerting and provides a reusable way to ingest third party alerts into ITSI, without the need to create correlation searches or notable event aggregation policies.
For more information about Universal Alerting, see About Universal Alerting in the Content Pack for Monitoring and Alerting.
For more information about correlation searches, see Overview of correlation searches in ITSI.
Prerequisite
Requirement | Description |
---|---|
Ingest the event data into ITSI | Before you can normalize and correlate external events in ITSI, you must first ingest the event data into Splunk. There are various methods to get third-party alerts into Splunk. It is recommended that for each alert sender, you have a different source type.
Use any of the following methods to ingest alerts into ITSI:
|
ITSI role | You must have the write_itsi_correlation_search capability to ingest third-party alerts into ITSI. The itoa_admin and itoa_team_admin ITSI roles have this capabilities by default. |
Steps
Configure a correlation search to normalize third-party alerts as ITSI notable events. A correlation search is an ad hoc Splunk search that enriches, normalizes, and deduplicates raw alerts. Each resulting row from the search becomes a notable event in ITSI.
ITSI correlation searches support field substitution with tokens in the format %fieldname%
. Use field substitution to map third-party alert field values to corresponding notable event fields.
Token variables cannot be the same as correlation search field names. For example, you cannot use "title," "description," "owner," "severity," "status," and so on, as variables. Instead use token variables such as %orig_description%, %orig_owner%, %orig_severity%, %orig_status%, and so on.
- In ITSI, click Configuration > Correlation Searches.
- Click Create New Search > Create Correlation Search.
- Configure a correlation search to ingest third party alerts.
Search Properties
Field name | Description | Defaults |
---|---|---|
Search Name | A name that describes the correlation search. For example, "nagios_alert". | None |
Description | A description of the type of issue the search is intended to detect. | None |
Search Type | Enter an ad hoc search string with appropriate eval command expressions to map fields in your third-party alert data to ITSI notable event fields. For example, the following ad hoc search maps Nagios severity-level fields to ITSI severity-level fields:
Alternatively, click Predefined and choose from a library of third-party search templates. For more information, see Correlation search templates in ITSI. |
Ad hoc
|
Time range | The time range over which the correlation search applies. Choose Relative and run the search over the last 1 minute or last 5 minutes. | Last 15 minutes
|
Association
Field name | Description | Defaults |
---|---|---|
Service | The ITSI services to which this correlation search applies. Leave this field blank. | None |
Entity Lookup Field | The field in the data retrieved by the correlation search that is used to look up corresponding entities. This is usually a host or node field (for example, itsiNode ). If the host or node of the alert is listed as a filtered entity in one or more ITSI services, these service names are added to the notable event as "affected service(s)".
|
None |
Schedule
Field name | Description | Defaults |
---|---|---|
Schedule Type | Choose a Basic schedule. Configure the search interval in the Run Every menu to run every 1 minute or 5 minutes (depending on the time range above). | Basic , Every 5 minutes
|
Notable Events
Use this section to configure the notable event that is generated when search results meet a specific condition. The Splunk platform indexes the event object like any other event. The events are displayed in Episode Review. After you configure a correlation search, create an aggregation policy to determine how to individual events are grouped into episodes. For more information, see Overview of aggregation policies in ITSI.
ITSI correlation searches support field substitution with tokens in the format %fieldname%
. Use field substitution to map third-party alert field values to corresponding notable event fields. To see the available fields that you can use as tokens, click Run Search under Search Properties. You can use any of the fields listed in the left panel.
Field name | Description | Defaults |
---|---|---|
Notable Event Title | The title of the notable event in Episode Review. For example, you might enter %name% to substitute Nagios alert names as notable event titles.
|
None |
Notable Event Description | A brief phrase to describe the notable event. For example, you might enter %reason% to substitute Nagios descriptions for notable event descriptions.
|
None |
Owner | The ITSI role to which the notable event is assigned in Episode Review. Click Advanced Mode, then enter the owner using a token in the format %fieldname% . For example, %owner_field% . In advanced mode, owners must match the names of owners in the system.
|
Unassigned
|
Severity | The level of importance of the event. Click Advanced Mode, then provide a token in the format %fieldname% to substitute the value of a third-party alert field. Values must match an integer specified in the default version of |
1 - Info2 - Normal3 - Low4 - Medium5 - High6 - Critical
|
Status | The triage status of the event in Episode Review. Click Advanced Mode, then provide a token in the format %fieldname% to substitute the value of a third-party alert field. Values must match an integer specified in the default version of |
0 - Unassigned1 - New2 - In Progress3 - Pending4 - Resolved5 - Closed
|
Instructions | Specific instructions for how to handle the notable event. Static instructions aren't supported. You must use token replacement to substitute the value of a third-party alert field. When you configure aggregation policies you can specify how you want to display these instructions in Episode Review. | %itsi_instruction%
|
Drilldown Name | You can drill down to a specific Splunk search from the All Events tab of Episode Review. Set the name of the drilldown search link. | None |
Drilldown Search | The search you drill down to. | None |
Drilldown earliest offset | Defines how far back from the time of the event to start looking for related events. | Last 5 minutes
|
Drilldown latest offset | Defines how far ahead from the time of the event to look for related events. | Next 5 minutes
|
Notable Event Identifier Fields | Determine whether a notable event is unique or not, which drives the events timeline. These identifier fields form the event hash field, which is added to every notable event to help identify unique alarm types. Use the exact same fields you use to dedup on in a Splunk search. These fields must exist in the raw event, otherwise a notable event isn't created. You must provide the correct fields for the Impacted Entities section of associated episodes to populate correctly. |
source
|
Drilldown link title | You can drill down to a specific website from the All Events tab of Episode Review. Set the name of the drilldown website link. | None |
Drilldown link URI | The website you drill down to. | None |
4. Click Save.
The correlation search runs as scheduled, mapping third-party alert field values to ITSI notable event fields. Third-party alerts appear as notable events in Episode Review. The events are collected in the itsi_tracked_alerts index.
Directly push events into ITSI through HEC
Rather than ingesting external alerts through a correlation search, you can also use HTTP Event Collector (HEC) to proactively push data directly into ITSI as notable events directly into the itsi_tracked_alerts index.
- In Splunk Web, go to Settings > Data Inputs.
- Click HTTP Event Collector.
- Copy the token value for the Auto Generated ITSI Event Management Token.
- Run a command similar to the following example to bring external data directly into ITSI and format it as a notable event:
curl -k https://localhost:8088/services/collector/event -H "Authorization: Splunk <token value>" -d '{"event" : {"event_id" : "< unique GUID>", "title" : "Disk 90% Full", "status" : "4", "severity" : "6", "owner" : "unassigned", "description": "Disk is almost full", "other_field" : "more stuff"}}'
Normalize event fields in ITSI | Overview of notable events in ITSI |
This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.11.0, 4.11.1, 4.11.2, 4.11.3, 4.11.4, 4.11.5, 4.11.6, 4.12.0 Cloud only, 4.12.1 Cloud only, 4.12.2 Cloud only, 4.13.0, 4.13.1, 4.13.2, 4.13.3, 4.14.0 Cloud only, 4.14.1 Cloud only, 4.14.2 Cloud only, 4.15.0, 4.15.1, 4.15.2, 4.15.3, 4.16.0 Cloud only, 4.17.0, 4.17.1, 4.18.0, 4.18.1, 4.19.0, 4.19.1
Feedback submitted, thanks!