Detecting patterns
This section describes detecting patterns in your data. For a complete list of topics on detecting anomalies, finding and removing outliers, and time series forecasting see About advanced statistics, in this manual.
Detecting patterns in events
The cluster command is a powerful command for detecting patterns in your events. The command groups events based on how similar they are to each other. The cluster command groups events based on the contents of the _raw field, unless you specify another field.
When you use the cluster command, two new fields are appended to each event.
- The
cluster_countis the number of events that are part of the cluster. This is the cluster size. - The
cluster_labelspecifies which cluster the event belongs to. For example, if the search returns 10 clusters, then the clusters are labeled from 1 to 10.
Anomalies come in small or large groups (or clusters) of events. A small group might consist of 1 or 2 login events from a user. An example of a large group of events might be a DDoS attack of thousands of similar events.
Use the cluster command parameters wisely
- Use the
labelonly=trueparameter to return all of the events. If you uselabelonly=false, which is the default, then only one event from each cluster is returned. - Use the
showcount=trueparameter so that acluster_countfield is added to all of the events. Ifshowcount=false, which is the default, the event count is not added to the event. - The threshold parameter
tadjusts the cluster sensitivity. The smaller the threshold value, the fewer the number of clusters.
Other commands to use with the cluster command
- Use the dedup command on the cluster_label column to see the most recent grouped events within each cluster.
- To group the events and make the results more readable, use the sort command with the cluster columns. Sort the cluster_count column based on the number of clusters.
- For small groups of events, sort the
cluster_countcolumn in ascending order. - For large groups of events sort the
cluster_countcolumn in descending order. - Sort the
cluster_labelcolumn in ascending order. Cluster labels are numeric. Sorting in ascending order organizes the events by label, in numerical order.
- For small groups of events, sort the
Return the 3 most recent events in each cluster
The following search uses the CustomerID in the sales_entries.log file. Setting showcount=true ensures that all events get a cluster_count. The cluster threshold is set to 0.7. Setting labelonly=true returns the incoming events. The dedup command is used to see the 3 most recent events within each cluster. The results are sorted in descending order to group the events.
source="/opt/log/ecommsv1/sales_entries.log" CustomerID
| cluster showcount=true t=0.7 labelonly=true
| table _time, cluster_count, cluster_label, _raw
| dedup 3 cluster_label
| sort -cluster_count, cluster_label, - _time
If you do not set labelonly=true, then only one event from each cluster is returned.
See also
- Related information
- About advanced statistics
| Detecting anomalies | About time series forecasting |
This documentation applies to the following versions of Splunk® Enterprise: 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.0.13, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.1.10, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8, 7.3.9, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.0.5, 8.0.6, 8.0.7, 8.0.8, 8.0.9, 8.0.10, 8.1.0, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.11, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.0.7, 9.0.8, 9.0.9, 9.0.10, 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.1.4, 9.1.5, 9.1.6, 9.1.7, 9.2.0, 9.2.1, 9.2.2, 9.2.3, 9.2.4, 9.3.0, 9.3.1, 9.3.2, 8.1.10, 8.1.12, 8.1.13, 8.1.14
Download manual
Feedback submitted, thanks!