Detecting patterns
This section describes detecting patterns in your data. For a complete list of topics on detecting anomalies, finding and removing outliers, and time series forecasting see About advanced statistics, in this manual.
Detecting patterns in events
The cluster command is a powerful command for detecting patterns in your events. The command groups events based on how similar they are to each other. The cluster
command groups events based on the contents of the _raw
field, unless you specify another field.
When you use the cluster
command, two new fields are appended to each event.
- The
cluster_count
is the number of events that are part of the cluster. This is the cluster size. - The
cluster_label
specifies which cluster the event belongs to. For example, if the search returns 10 clusters, then the clusters are labeled from 1 to 10.
Anomalies come in small or large groups (or clusters) of events. A small group might consist of 1 or 2 login events from a user. An example of a large group of events might be a DDoS attack of thousands of similar events.
Use the cluster command parameters wisely
- Use the
labelonly=true
parameter to return all of the events. If you uselabelonly=false
, which is the default, then only one event from each cluster is returned. - Use the
showcount=true
parameter so that acluster_count
field is added to all of the events. Ifshowcount=false
, which is the default, the event count is not added to the event. - The threshold parameter
t
adjusts the cluster sensitivity. The smaller the threshold value, the fewer the number of clusters.
Other commands to use with the cluster command
- Use the dedup command on the cluster_label column to see the most recent grouped events within each cluster.
- To group the events and make the results more readable, use the sort command with the cluster columns. Sort the cluster_count column based on the number of clusters.
- For small groups of events, sort the
cluster_count
column in ascending order. - For large groups of events sort the
cluster_count
column in descending order. - Sort the
cluster_label
column in ascending order. Cluster labels are numeric. Sorting in ascending order organizes the events by label, in numerical order.
- For small groups of events, sort the
Return the 3 most recent events in each cluster
The following search uses the CustomerID in the sales_entries.log
file. Setting showcount=true
ensures that all events get a cluster_count
. The cluster threshold is set to 0.7. Setting labelonly=true
returns the incoming events. The dedup
command is used to see the 3 most recent events within each cluster. The results are sorted in descending order to group the events.
source="/opt/log/ecommsv1/sales_entries.log" CustomerID
| cluster showcount=true t=0.7 labelonly=true
| table _time, cluster_count, cluster_label, _raw
| dedup 3 cluster_label
| sort -cluster_count, cluster_label, - _time
If you do not set labelonly=true
, then only one event from each cluster is returned.
See also
- Related information
- About advanced statistics
Detecting anomalies | About time series forecasting |
This documentation applies to the following versions of Splunk® Enterprise: 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.0.13, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.1.10, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8, 7.3.9, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.0.5, 8.0.6, 8.0.7, 8.0.8, 8.0.9, 8.0.10, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.13, 8.1.14, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.0.7, 9.0.8, 9.0.9, 9.0.10, 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.1.4, 9.1.5, 9.1.6, 9.2.0, 9.2.1, 9.2.2, 9.2.3, 9.3.0, 9.3.1, 8.1.0, 8.1.10, 8.1.11, 8.1.12
Feedback submitted, thanks!