streamstats command: Overview, syntax, and usage

The SPL2 streamstats command adds a cumulative statistical value to each search result as each result is processed. For example, you can calculate the running total for a particular field, or compare a value in a search result with a the cumulative value, such as a running average. The streamstats command includes options for resetting the aggregates.

Use these links to quickly navigate to the main sections in this topic:

How the SPL2 streamstats command works

Suppose that you have the following data:

host	action	bytes
x	LOGON	100
y	APP_START	200
x	FILE_DOWNLOAD	400
x	REBOOT	50
y	LOGON	150
x	LOGON	100

You can use the SPL2 streamstats command to calculate and add various statistics to the search results.

Compute a moving average over a series of events

For each event, you can compute the average of the bytes field over the last 3 events, including the current event. Here's the search to use:

... | streamstats window=3 avg(bytes)

The output looks like this:

host	action	bytes	avg(bytes)
x	LOGON	100	100
y	APP_START	200	150
x	FILE_DOWNLOAD	400	233.33
x	REBOOT	50	216.66
y	LOGON	150	200
x	LOGON	100	100

For the first event, there are no previous events. The value for the bytes field is returned.
For the second event, the average is returned from the sum of first and second events.
For the remaining events, the average is returned from the sum of the current event and the two previous events.

Calculate a value until a trigger resets the calculation

Suppose you want to calculate a running total of the bytes for each host. However, when the system reboots you want the calculation for the total bytes to begin again. You can use the reset after argument to accomplish this. Here's the search to use:

...| streamstats sum(bytes) AS total_bytes BY host reset after action="REBOOT"

Because the value in the action field is a string literal, the value needs to be enclosed in double quotation marks.

The running total appears in the total_bytes field. The running total resets each time an event satisfies the action="REBOOT"criteria.

The results look like this:

host	action	bytes	total_bytes
x	LOGON	100	100
y	APP_START	200	200
x	FILE_DOWNLOAD	400	500
x	REBOOT	50	550
y	LOGON	150	150
x	LOGON	100	100

The total_bytes field accumulates a sum of the bytes so far for each host. When the reset after clause action="REBOOT" occurs in the 4th event, that event shows the sum for the x host, including the bytes for the REBOOT action. The sum of the bytes is reset for both the y and x hosts in the next events.

Applying a count to each event

You can apply a running count to your search results, which is useful when combined with other commands.

...| streamstats count()

The output looks like this:

host	action	bytes	count
x	100	LOGON	1
y	APP_START	200	2
x	FILE_DOWNLOAD	400	3
x	REBOOT	50	4
y	LOGON	150	5
x	LOGON	100	6

Syntax

The required syntax is in bold.

streamstats

[<by-clause>]

[current=<bool>]

[<reset-clause>]

[window=<int>]

<aggregation> ...

If you're going to use any of the optional arguments, they must be specified before the <aggregation>.

The BY keyword is displayed in uppercase in the syntax and examples to make the syntax easier to read. You can specify this keyword in uppercase or lowercase.

Required arguments

aggregation: Syntax: <aggregate-function> "("<field>")" [as <field>] ) ["," <aggregate-function> "("<field>")" [as <field>] ) ]...; Description: A statistical aggregation function. The function can be applied to an eval expression, or one or more fields. You can specify multiple aggregation functions. Separate each aggregation function with a comma.

By default, the name of the field added to the output is the same as your function. For example, if your search is ... | streamstats avg(bytes) the field name in the output is avg(bytes). Use the <as-clause> to place the generated result into a new field with a name that you specify, for example ... | streamstats avg(bytes) as 'avg of bytes'.

The syntax for the <aggregate-function> depends on the function that you use. See Quick Reference for SPL2 Stats and Charting Functions for information about the statistical functions.

Optional arguments

by-clause: Syntax: BY <field-list>; Description: The name of one or more fields to group the results by. The <by-clause> returns one row for each distinct value in the <by-clause> fields. Think of the <by-clause> as a grouping. You cannot use the wildcard character to specify multiple fields with similar names. You must specify each field separately.; Default: If no <by-clause> is specified, the streamstats command returns a running aggregation for each row in the incoming result set.

current: Syntax: current=<boolean>; Description: If set to true, the search includes the given, or current, event in the summary calculations. If set to false, the search uses the field value from the previous event.; Default: true

reset-clause: Syntax: reset [before <eval-expression>] [after <eval-expression>] [onchange]; Description: You can specify one or more reset condition. If multiple conditions are specified, the reset occurs when any of the conditions triggers a reset. See the Usage section.; Default: No reset is applied to the results.

window: Syntax: window=<integer>; Description: Specifies the number of events to use when computing the statistics. Must be a non-negative number.; Default: 0, which means that all previous and current events are used.

Usage

The following sections contain information to help you understand and use the SPL2 streamstats command.

Resetting the aggregations

There are several ways to reset the aggregations. You can reset before something occurs, after something occurs, and when the values in the <by clause> field changes.

Reset after and reset before

The reset after clause resets the aggregation in the next search result after the condition occurs.

The reset before clause resets the aggregation in the search result in which the condition occurs.

Suppose that you have the following data:

host	bytes	action
x	100	LOGON
y	200	APP_START
x	400	FILE_DOWNLOAD
x	50	REBOOT
y	150	LOGON
x	100	LOGON

You want to calculate the total bytes for each host. However, when the system reboots you want the calculation for the total bytes to begin again. You can use the reset after argument to accomplish this.

| streamstats reset after action="REBOOT" sum(bytes) as total_bytes BY host

Because the value in the action field is a string literal, the value needs to be enclosed in double quotation marks.

The streamstats command calculates a running total of the bytes for each host into a field called total_bytes. The running total resets each time an event satisfies the action="REBOOT"criteria. The results look like this:

host	bytes	action	total_bytes
x	100	LOGON	100
y	200	APP_START	200
x	400	FILE_DOWNLOAD	500
x	50	REBOOT	550
y	150	LOGON	150
x	100	LOGON	100

The total_bytes field accumulates a sum of the bytes so far for each host. When the reset after clause action="REBOOT" occurs in the 4th event, that event shows the sum for the x host, including the bytes for the REBOOT action. The sum of the bytes is reset for both the y and x hosts in the next events.

If the reset before clause is used instead, the results would be this:

host	bytes	action	total_bytes
x	100	LOGON	100
y	200	APP_START	200
x	400	FILE_DOWNLOAD	500
x	50	REBOOT	50
y	150	LOGON	150
x	100	LOGON	150

Reset onchange

To reset the aggregation whenever any of the fields specified in the <group-by> clause change, use the reset onchange condition.

Continuing with the previous example, you would use this syntax:

...| streamstats reset onchange sum(bytes) as total_bytes BY host

The output from this search is this:

host	bytes	action	total_bytes
x	100	LOGON	100
y	200	APP_START	200
x	400	FILE_DOWNLOAD	400
x	50	REBOOT	450
y	150	LOGON	150
x	100	LOGON	100

Because the value of the host changes between the 2nd and 3rd rows, the total_bytes is reset in the 3rd row. The reset occurs again between the 4th and 5th rows.

Combining reset clauses

You can combine the reset clauses. For example you can use this search:

...| streamstats reset after action="REBOOT" onchange sum(bytes) as total_bytes BY host

If combined, a reset occurs whenever any of the clauses triggers a reset.

Differences between SPL and SPL2

The differences between the SPL and SPL2 streamstats command are described in these sections.

Command syntax has changed

The streamstats command syntax in SPL2 is substantially different from the streamstats command in SPL. All of the reset conditions have new syntax that makes it easier to write expressions. Instead of individual reset arguments, there is one reset argument where you can specify multiple reset conditions. Additionally, you no longer have to escape quotation marks or parentheses in the reset expressions.

SPL

The following table shows the SPL syntax and an example for the streamstats command.

Syntax	Example
streamstats	streamstats
reset_on_change=<bool>	reset_on_change=true
reset_before="("<eval-expression>")"	reset_before=(reboot="COMPLETE")
reset_after="("<eval-expression>")"	reset_after=(CPUUtilization > 50)
current=<bool>	current=false
window=<int>	window=5
time_window=<span-length>
global=<bool>
allnum=<bool>
<stats-agg-term>...	avg(CPUUtilization) AS avg_cpu
<by-clause>	BY host

SPL2

The following table shows the SPL2 syntax and an example for the streamstats command.

Syntax	Example
streamstats	streamstats
<stats aggregation>	avg(CPUUtilization) AS avg_cpu
current=<bool>	current=false
window=<int>	window=5
<by-clause>	BY host
reset before <eval-expression> after <eval-expression> onchange	reset before reboot="COMPLETE" after CPUUtilization > 50 onchange

Some command options are not supported in SPL2

The following arguments from SPL do not have an equivalent argument in SPL2.

allnum
global
time_window

Related answers from Splunk Community