Splunk® IT Service Intelligence

Event Analytics Manual

Acrobat logo Download manual as PDF


Splunk IT Service Intelligence (ITSI) version 4.9.x will reach its End of Life on April 21, 2023. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see Before you upgrade IT Service Intelligence.
This documentation does not apply to the most recent version of Splunk® IT Service Intelligence. Click here for the latest version.
Acrobat logo Download topic as PDF

Best practices for implementing Event Analytics in ITSI

Consider the following best practices when configuring Event Analytics in IT Service Intelligence (ITSI). For more information about ITSI's Event Analytics functionality, see Overview of Event Analytics in ITSI.

Best practices for implementing Event Analytics for ITSI services and KPIs

For best practices around leveraging ITSI's Event Analytics functionality to translate service and KPI health into notable events and episodes, see About the Content Pack for Monitoring and Alerting. The content pack provides a set of preconfigured correlation searches and notable event aggregation policies which, when enabled, produce meaningful and actionable alerts.

Best practices for implementing Event Analytics for other data sources

The following best practices help you successfully ingest and aggregate third-party alerts in ITSI. For more information, see Ingest third-party alerts into ITSI.

To avoid duplicate events, use the same frequency and time range in correlation searches

When configuring a correlation search, consider using the same value for the search frequency and time range to avoid duplicate events. For example, a search might run every five minutes and also look back every five minutes.

If there's latency in your data and you need to look for events you might have missed, consider expanding the time range. For example, the search could run every minute but look back 5 minutes.

To reduce load on your system, don't use a time range greater than 5 minutes

Exceeding a calculation window of 5 minutes can put a lot of load on your system, especially if you have a lot of events coming in. If you want to avoid putting extra load on your system, consider reducing the time range to 5 minutes or less. One exception is if your data is coming in more sporadically. For example, if your data comes in every 15 minutes, consider using a 15-minute time range.

Normalize all the important fields in your third-party events

When you're creating correlation searches, don't only normalize on obvious fields that exist in a lot of data sources, like host, severity, event type, message, and so on. It's also important to normalize fields that you know are important in your events. For example, when you're looking at Windows event logs, what do you look at to know if something is good or bad? Normalize those fields as well and use them to build out a common information model.

Perform this normalization process for every data source you have so you can easily identify important fields when creating aggregation policies.

Create one correlation search per data source

For every third-party data source you're bringing into ITSI, create a single correlation search to normalize those fields and generate notable events. For example, one for SCOM, one for SolarWinds, and so on.

Before deleting services, unlink them from correlation searches

You can delete ITSI services that you no longer want to monitor, but to keep that action from also disabling or deleting the correlation search of which a service was part, you should manually remove the service from a correlated search before deleting that service.

Don't create too many aggregation policies

Limit the number of aggregation policies you enable in your environment. Too many aggregation policies create too many groups, which produces an overly granular view of your IT environment. By limiting the number of policies, you create more end-to-end visibility and avoid creating silos of collaboration between groups in your organization. Make sure to group events according to how those events are related, not based on how people work to resolve those issues.

Only select 5-10 fields for Smart Mode analysis

By default, when selecting fields to analyze for event similarity, Smart Mode selects any fields that have good event coverage. As a best practice, begin by unchecking all the boxes. Then select 5-10 fields that you've normalized in a correlation search.

Selecting between 5-10 fields ensures that you generate an appropriate size and quantity of episodes. If you select fewer than five fields, you only give the aggregation policy with a few things to look at when comparing similarity. For example, if the message of two events is somewhat similar and the location is similar, they might be grouped together. This can lead to a small number of very large groups. The opposite is also true. If you select too many fields, the chances of them all being similar is very low. This can lead to a large number of groups containing only one event.

Last modified on 15 July, 2022
PREVIOUS
Configure Rules Engine periodic backfill in ITSI
  NEXT
Troubleshoot the Rules Engine and event grouping in ITSI

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.9.0, 4.9.1, 4.9.2, 4.9.3, 4.9.4, 4.9.5, 4.9.6


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters