Identify event patterns with the Patterns tab

Events in search results can be grouped into event patterns. Events that belong to an event pattern share common characteristics, and usually can be returned by a specific search string. Event pattern analysis is useful for searches that return a diverse range of events because it quickly shows you the most common kinds of events in your search result dataset.

The Patterns tab simplifies event pattern identification. Click the Pattern tab to view a list of the most common patterns among the set of events returned by your search. Each of these patterns represents a set of events that share a similar structure.

Click on a pattern to:

• View the approximate number of events in your results that fit the pattern.
• View the search that returns events with this pattern.
• Save the pattern search as an event type, if possible. Not all event patterns can be saved as event types.
• Create an alert based on the pattern. For example, you can create alerts that trigger when certain patterns increase or decrease in frequency.

An event patterns example

A search that uses sourcetype=cisco:esa runs for All time and returns 112,421 events.

The patterns are based on a sample of 50,000 events.

To view all of the patterns in the list of events, click the Patterns tab. Initially the results identify 45 patterns. You can move the Smaller to Larger slider to change how narrow or broad the event patterns should be. If you drag the slider closer to the Larger side, 14 patterns are returned.

The threat level event pattern is the most common one. Some of the listed patterns are relatively rare in this dataset, and finding them in the Events tab listing might be difficult. The Patterns tab makes it easier to see these event patterns and save them as event types if necessary.

How the Patterns tab works

When you click the Patterns tab, Splunk software runs a secondary search on a subset of the search results that have been received up to that point. This search job analyzes those results and derives the most common event patterns in those results. It then lists those patterns in descending order from most prevalent to least prevalent. It may not include outlier patterns that are based on extremely small groups of events, because they are statistically unreliable.

The secondary search can take a long time to complete when the original dataset contains an extremely large variety of event patterns. For example, some searches return datasets containing over 500 patterns, where most of those patterns represent very small collections of events. The algorithm that identifies these patterns is designed to avoid doing too much work for small patterns, but it also attempts to be as accurate as possible.

The Patterns tab only accepts transforming searches when you set their search mode to Verbose. The Patterns tab cannot find patterns for real-time searches in any search mode.

Event pattern keywords

The Patterns tab defines patterns by the presence or absence of one or more keywords. If the keywords identified for a pattern are added to or excluded from the original search, the search returns events that fit that pattern. Keywords present in a pattern are identified with green text in the pattern list. Excluded keywords are not identified in the event list.

In the preceding event pattern example, the threat level event pattern has "threat" as its keyword, meaning that the search that returns events fitting the pattern would look like this:

sourcetype=cisco_esa threat

If this event pattern were also identified by the exclusion of the keyword "verified," the search that returns events fitting the pattern would look like this:

sourcetype=cisco_esa threat ( NOT verified )

To see all of the keywords associated with a pattern, click on the pattern.

Use the Patterns tab

1. From the Search view, run a search that returns more than 5000 results.

Searches returning more than 5000 results produce reliable patterns.

2. Click the Patterns tab.

You do not need to wait for the search to complete, but the pattern listing is more accurate with finalized search results.

3. (Optional) If there appear to be too many patterns or too few, or if you do not see the patterns you expect, move the slider.

Dragging the slider to Larger runs a secondary search job that consolidates some patterns together, resulting in event patterns that represent larger numbers of events, and a wider variety of events inside each pattern group.

Dragging the slider to Smaller runs a secondary search job that increases the granularity of the results. The event patterns it finds represent smaller numbers of events.

4. (Optional) Click on a pattern to view information for that pattern.

Estimated Events is the estimated count of events in the dataset returned by the original search that fit the event pattern. In this example, the original search had 112k events. This pattern accounts for an estimated 12,200 events or 10.84% of the total number of events.

Included Keywords identifies keywords that should be added to the base search to return the pattern. If the Patterns tab identifies keywords that should be excluded from the base search, they appear under an Excluded Keywords section.

You can see the search that returns events fitting the event pattern under Search.

5. (Optional) In the pattern information area, click View Events to run the search displayed under Search.

When it runs, this search uses the same time range as your original search.

6. (Optional) In the pattern information area, click Save as event type to save the search as an event type.

Save as event type is available only for event patterns based on searches that do not include pipe characters and additional search commands. See "About event types" in this manual.

7. (Optional) In the pattern information area, click Create alert to create an alert based on the pattern.

For example, create a scheduled alert that is triggered when the frequency of the event pattern rises above or drops below a threshold. If you know that events that fit an event pattern tend to appear at a steady rate of approximately 100 events per hour, set the alert to run on an hourly schedule and trigger when 150 or more events are returned. See the Alerting Manual.

Numbers in the Patterns tab

When the secondary search finishes, the Patterns tab displays a message explaining how many events it analyzed to obtain the displayed results.

The Patterns tab analyzes a subset of the total number of events returned by the original search. The maximum number of events in this subset is 50k. This maximum reduces processing times for the Pattern tab secondary search. If your original search returns less than 50k events, the secondary search analyzes up to 1000 events per timeline bar spanned by the original search. For example, if the original search spans 14 timeline bars, the secondary search analyzes 14,000 events to obtain its patterns listing.

You can control the maximum number of events analyzed by the secondary search by updating the maxevents setting, in the [findkeywords] stanza, in limits.conf. This setting defaults to 50000. Do not change this value. A number less than 50,000 reduces the accuracy of your events. A number higher than 50,000 increases the processing time required by the secondary search.

The estimated event count provided in the pattern information area does not apply to the number of events analyzed in the Patterns tab secondary search. It applies to the total number of events returned by the original search. If an event pattern is estimated to represent 7,350 events and the original search returned 265k events, the pattern accounts for 2.7% of the events returned by the search.

Restricting Patterns tab usage

All roles, including the user role, have permission to use the Patterns tab by default. To restrict usage of the Patterns tab, remove the pattern_detect capability. Roles without this capability do not see the Patterns tab option after they run a search.

For more information about capabilities, see "About defining roles with capabilities" in the Securing Splunk manual.