Identify spikes in data and notify using Splunk Light
You want to identify spikes in your data. Spikes can show you where you have peaks (or troughs) the indicate that some metric is rising or falling sharply. Traffic spikes, sales spikes, spikes in the number of returns, spikes in database load. Whatever type of spike you are interested in, you want to watch for it, set up alerts to notify you, and then perhaps take some action to address those spikes.
Search for spikes in your data
Use a moving trendline to help you see the spikes. Run a search followed by the trendline command using a field you want to create a trendline for.
For example, on web access data, you could chart an average of the
sourcetype=access* | timechart avg(bytes) as avg_bytes
To add another line or bar series to the chart for the simple moving average (sma) of the last 5 values of
bytes, use this command:
... | trendline sma5(avg_bytes) as moving_avg_bytes
If you want to clearly identify spikes, you might add an additional series for spikes. This is when the current value is more than twice the moving average.
... | eval spike=if(avg_bytes > 2 * moving_avg_bytes, 10000, 0)
The 10000 here is arbitrary and you should choose a value relevant to your data that makes the spike noticeable. Changing the formatting of the y-axis to Log scale also helps.
Putting this all together, the search is:
| timechart avg(bytes) as avg_bytes
| trendline sma5(avg_bytes) as moving_avg_bytes
| eval spike=if(avg_bytes > 2 * moving_avg_bytes, 10000, 0)
This search uses a simple moving average for the last 5 results (sma5). Consider a different number of values, for example sma20.
The trendline command also supports the exponential moving average (ema) and the weighted moving average (wma).
Alternatively, you can bypass the charting altogether and replace the
eval command with the
where command to filter your results.
... | where avg_bytes > 2 * moving_avg_bytes
And by looking at the table view or as an alert, you will only see the times when the
Notify when server load reaches a threshold using Splunk Light
Create an alert to monitor CPU usage using Splunk Light
This documentation applies to the following versions of Splunk® Light (Legacy): 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.5.0, 6.5.1, 6.5.2, 6.5.3, 6.5.1612 (cloud service only), 6.6.0, 6.6.1, 6.6.2, 6.6.3, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6