mstats
Description
Use the mstats
command to analyze metrics. This command performs statistics on the measurement
, metric_name
, and dimension
fields in metric indexes. You can use mstats
in historical searches and real-time searches. When you use mstats
in a real-time search with a time window, a historical search runs first to backfill the data.
The mstats
command provides the best search performance when you use it to search a single metric_name
value or a small number of metric_name
values.
Syntax
The required syntax is in bold.
- | mstats
- [prestats=<bool>]
- [append=<bool>]
- [backfill=<bool>]
- [update_period=<integer>]
- [fillnull_value=<string>]
- [chunk_size=<unsigned int>]
- <stats-metric-term>...
- WHERE [<logical-expression>]...
- [ (BY|GROUPBY) <field-list> ]
- [<span-length>]
Required arguments
- <stats-metric-term>
- Syntax: [<stats-func>|<stats-func-value>]
- Description: Provides two options for performing statistical calculations on metrics. Use
<stats-func>
to perform statistical calculations on one or more metrics that you name in the argument. Use<stats-func-value>
for cases where a wildcard can be used to represent several metrics. You cannot blend the<stats-func>
syntax and the<stats-func-value
syntax in a singlemstats
search.
- Use the
<stats-func>
syntax for most cases. You only need to use the<stats-func-value>
syntax in cases where a single metric may be represented by several different metric names, such ascpu.util
andcpu.utilization
. In these cases you can apply a wildcard to catch all of the permutations of themetric_name
.
- See Stats metric term options for details on the
<stats-func>
and<stats-func-value>
syntax options.
- The following table lists the supported functions for the
mstats
command by type of function. Use the links in the table to see descriptions and examples for each function.
Type of function Supported functions and syntax Aggregate functions avg()
count()
max()
median()
min()
perc<num>
range()
stdev()
stdevp()
sum()
sumsq()
upperperc<num>
var()
varp()
Time functions earliest()
earliest_time()
latest()
latest_time()
rate()
For an overview of using functions with commands, see Statistical and charting functions.
Optional arguments
- append
- Syntax: append=<bool>
- Description: Valid only when
prestats=true
. This argument runs themstats
command and adds the results to an existing set of results instead of generating new results. - Default: false
- backfill
- Syntax: backfill=<bool>
- Description: Valid only with real-time searches that have a time window. When
backfill=true
, themstats
command runs a search on historical data to backfill events before searching the in-memory real-time data. - Default: true
- chunk_size
- Syntax: chunk_size=<unsigned_int>
- Description: Advanced option. This argument controls how many metric time series are retrieved at a time from a single TSIDX file when the Splunk software processes searches. Lower this setting from its default only when you find a particular search is using too much memory. This can happen when a search groups by excessively high-cardinality fields (fields with very large amounts of distinct values). Setting
chunk_size
too low can causemstats
searches to have overlong runtimes. If you setchunk_size
below10000
the Splunk software will reset its value to the default of10000000
. - Default: 10000000 (10 million)
- fillnull_value
- Description: This argument sets a user-specified value that the
mstats
command substitutes for null values for any field within its group-by field list. Null values include field values that are missing from a subset of the returned events as well as field values that are missing from all of the returned events. If you do not provide afillnull_value
argument,mstats
omits rows for events with one or more null field values from its results. - Default: empty string
- <field-list>
- Syntax: <field>, ...
- Description: Specifies one or more fields to group the results by. Required when using the BY clause.
- <logical-expression>
- Syntax: <time-opts>|<search-modifier>|((NOT)? <logical-expression>)|<search-modifier>|<comparison-expression>|(<logical-expression> (OR)? <logical-expression>)
- Description: An expression describing the filters that are applied to your search. Includes time and search modifiers, and comparison expressions. See the following sections for descriptions of each of these logical expression components.
- Cannot filter on
metric_name
. Does not support CASE or TERM directives. You also cannot use the WHERE clause to search for terms or phrases.
- prestats
- Syntax: prestats=true | false
- Description: Specifies whether to use the prestats format. The prestats format is a Splunk internal format that is designed to be consumed by commands that generate aggregate calculations. When you use the prestats format, you can pipe the data into the chart, stats, or timechart commands, which are designed to accept the prestats format. When
prestats
is set totrue
, instructions with theAS
clause are not relevant. The field names for the aggregates are determined by the command that consumes the prestats format and produces the aggregate output. - Default: false
- <span-length>
- Syntax: span=<int><timescale>
- Description: The span of each time bin. If used with a <timescale>, the
span-length
is treated as a time range. If not, this is an absolute bucket length. If you do not specify a <span-length>, the default isauto
, which means that the number of time buckets adjusts to produce a reasonable number of results. For example, if seconds are used initially for the <timespan> and too many results are returned, the <timescale> is changed to a longer value, such as minutes, to return fewer time buckets.
- <timescale>
- Syntax: <sec> | <min> | <hr> | <day> | <month>
- Description: Time scale units. For the
mstats
command, the <timescale> does not support subseconds. - Default: sec
Time scale Syntax Description <sec> s | sec | secs | second | seconds Time scale in seconds. <min> m | min | mins | minute | minutes Time scale in minutes. <hr> h | hr | hrs | hour | hours Time scale in hours. <day> d | day | days Time scale in days. <month> mon | month | months Time scale in months.
- update_period
- Syntax: update_period=<integer>
- Description: Valid only with real-time searches. Specifies how frequently, in milliseconds, the real-time summary for the
mstats
command is updated. A larger number means less frequent updates to the summary and less impact on index processing. - Default: 1000 (1 second)
Stats metric term options
- <stats-func>
- Syntax: <function>"("<metric_name")" [AS <string>])...
- Description: Perform statistical calculations on one or more
metric_name
fields. You can rename the result of each function using theAS
clause, unlessprestats
is set totrue
. Themetric_name
must be enclosed in parenthesis.
- When you use the
<stats-func>
syntax, theWHERE
clause cannot filter onmetric_name
.
- <stats-func-value>
- Syntax: count(_value) | <function>(_value) [AS <string>] WHERE metric_name=<metric_name>
- Description: Specify a basic count of the
_value
field or a function on the_value
field. The_value
field uses a specific format to store the numeric value of the metric. You can specify one or more functions. You can rename the result of the function using AS unlessprestats=true
.
- When you use the
<stats-func-value>
syntax, the WHERE clause must filter on themetric_name
. Wildcards are okay.
The stats-func-value syntax does not support real-time searches. If you must run a real-time search, use the stats-func syntax instead.
Logical expression options
- <comparison-expression>
- Syntax: <field><comparison-operator><value> | <field> IN (<value-list>)
- Description: Compares a field to a literal value or provides a list of values that can appear in the field.
- <search-modifier>
- Syntax: <sourcetype-specifier> | <host-specifier> | <source-specifier> | <splunk_server-specifier>
- Description: Search for events from specified fields. For example, search for one or a combination of hosts, sources, and source types. See searching with default fields in the Knowledge Manager manual.
- <time-opts>
- Syntax: [<timeformat>] (<time-modifier>)*
- Description: Describes the format of the
<starttime>
and<endtime>
terms of the search.
Comparison expression options
- <comparison-operator>
- Syntax: = | != | < | <= | > | >=
- Description: You can use comparison operators when searching field-value pairs. Comparison expressions with the
equal ( = )
ornot equal ( != )
operator compare string values. For example, "1" does not match "1.0". Comparison expressions with greater than or less than operators< > <= >=
numerically compare two numbers and lexicographically compare other values. See Usage.
- <field>
- Syntax: <string>
- Description: The name of a field.
- <value>
- Syntax: <literal-value>
- Description: In comparison expressions, this is the literal number or string value of a field.
- <value-list>
- Syntax: (<literal-value>, <literal-value>, ...)
- Description: Used with the IN operator to specify two or more values. For example use
error IN (400, 402, 404, 406)
instead oferror=400 OR error=402 OR error=404 OR error=406
Search modifier options
- <sourcetype-specifier>
- Syntax: sourcetype=<string>
- Description: Search for events from the specified sourcetype field.
- <host-specifier>
- Syntax: host=<string>
- Description: Search for events from the specified host field.
- <source-specifier>
- Syntax: source=<string>
- Description: Search for events from the specified source field.
- <splunk_server-specifier>
- Syntax: splunk_server=<string>
- Description: Search for events from a specific server. Use "local" to refer to the search head.
Time options
- <timeformat>
- Syntax: timeformat=<string>
- Description: Set the time format for starttime and endtime terms.
- Default: timeformat=%m/%d/%Y:%H:%M:%S.
- <time-modifier>
- Syntax: starttime=<string> | endtime=<string> | earliest=<time_modifier> | latest=<time_modifier>
- Description: Specify start and end times using relative or absolute time.
You can also use the earliest and latest attributes to specify absolute and relative time ranges for your search. For more about this time modifier syntax, see About search time rangesin the ''Search Manual''.
- starttime
- Syntax: starttime=<string>
- Description: Events must be later or equal to this time. The
starttime
match thetimeformat
.
- endtime
- Syntax: endtime=<string>
- Description: All events must be earlier or equal to this time.
For a list of time modifiers, see Time modifiers.
Usage
The mstats
command is a report-generating command, except when append=true
. See Command types.
Generating commands use a leading pipe character and should be the first command in a search, except when append=true
is specified with the command.
Use the mstats
command to search metrics data. The metrics data uses a specific format for the metrics fields. See Metrics data format in Metrics.
All metrics search commands are case sensitive. This means, for example, that mstats
treats as the following as three distinct values of metric_name
: cap.gear
, CAP.GEAR
, and Cap.Gear
.
Appending mstats searches together
The mstats
command does not support subsearches. You can use the append
argument to add the results of an mstats
search to the results of a preceding mstats
search. See the topic on the tstats
command for an append
usage example.
Aggregations
If you are using the <stats-func>
syntax, numeric aggregations are only allowed on specific values of the metric_name
field. The metric name must be enclosed in parenthesis. If there is no data for the specified metric_name
in parenthesis, the search is still valid.
If you are using the <stats-func-value>
syntax, numeric aggregations are only allowed on the _value
field.
Aggregations are not allowed for values of any other field, including the _time
field.
When prestats = true
and you run an mstats
search that uses the c
and count
aggregation functions without an aggregation field, the Splunk software processes them as if they are actually count(_value)
. In addition, any statistical functions that follow in the search string must reference the _value
field. For example: | mstats count | timechart count(_value)
Wildcard characters
The mstats
command supports wildcard characters in any search filter, with the following exceptions:
- You cannot use wildcard characters in the GROUP BY clause.
- If you are using the
<stats_func_value>
syntax, you cannot use wildcard characters in the_value
field. - If you are using wildcard characters in your aggregations and you are renaming them, your rename must have matching wildcards.
- For example, this search is invalid:
| mstats sum(*.free) as FreeSum
- This search is valid:
| mstats sum(*.free) as *FreeSum
- Real-time
mstats
searches cannot utilize wildcarded metric aggregations when you use the<stats-func>
syntax.
- For example, this search is invalid, when you set it up as a real-time search:
| mstats avg(cpu.*) max(cpu.*) where index=sysmetrics
- This real-time search is valid:
| mstats avg(cpu.sys) max(cpu.usr) where index=sysmetrics
WHERE clause
Use the WHERE clause to filter by any of the supported dimensions.
If you are using the <stats-func>
syntax, the WHERE clause cannot filter by metric_name
. Filtering by metric_name
is performed based on the metric_name
fields specified with the <stats-func>
argument.
If you are using the <stats-func-value>
syntax, the WHERE clause must filter by metric_name
.
If you do not specify an index name in the WHERE clause, the mstats
command returns results from the default metrics indexes associated with your role. If you do not specify an index name and you have no default metrics indexes associated with your role, mstats
returns no results. To search against all metrics indexes use WHERE index=*
.
For more information about defining default metrics indexes for a role, see Add and edit roles with Splunk Web in Securing Splunk Enterprise.
Group results by metric name and dimension
You can group results by the metric_name
and dimension
fields.
You can also group by time. You must specify a timespan using the <span-length> argument to group by time buckets. For example, span=1hr
or span=auto
. The <span-length> argument is separate from the BY clause and can be placed at any point in the search between clauses.
Grouping by the _value
or _time
fields is not allowed.
Group by metric time series
You can group results by metric time series. A metric time series is a set of metric data points that share the same metrics and the same dimension field-value pairs. Grouping by metric time series ensures that you are not mixing up data points from different metric data sources when you perform statistical calculations on them.
Use BY _timeseries
to group by metric time series. The _timeseries
field is internal and won't display in your results. If you want to display the _timeseries
values in your search, add | rename _timeseries AS timeseries
to the search.
For a detailed overview of the _timeseries
field with examples, see Perform statistical calculations on metric time series in Metrics.
Time dimensions
The mstats
command does not recognize the following time-related dimensions.
Unsupported dimensions date_hour
date_mday
date_minute
date_month
date_second
date_wday
date_year
date_zone
metric_timestamp
time
timeendpos
timestamp
timestartpos
Memory and mstats search performance
A pair of limits.conf
settings strike a balance between the performance of mstats
searches and the amount of memory they use during the search process, in RAM and on disk. If your mstats
searches are consistently slow to complete you can adjust these settings to improve their performance, but at the cost of increased search-time memory usage, which can lead to search failures.
If you have Splunk Cloud you will need to file a Support ticket to change these settings.
For more information, see Memory and stats search performance in the Search Manual.
Lexicographical order
Lexicographical order sorts items based on the values used to encode the items in computer memory. In Splunk software, this is almost always UTF-8 encoding, which is a superset of ASCII.
- Numbers are sorted before letters. Numbers are sorted based on the first digit. For example, the numbers 10, 9, 70, 100 are sorted lexicographically as 10, 100, 70, 9.
- Uppercase letters are sorted before lowercase letters.
- Symbols are not standard. Some symbols are sorted before numeric values. Other symbols are sorted before or after letters.
You can specify a custom sort order that overrides the lexicographical order. See the blog Order Up! Custom Sort Orders.
Examples
1. Calculate a single metric grouped by time
Return the average value of the aws.ec2.CPUUtilization
metric in the mymetricdata
metric index. Bucket the results into 30 second time spans.
| mstats avg(aws.ec2.CPUUtilization) WHERE index=mymetricdata span=30s
2. Combine metrics with different metric names
Return the average value of both the aws.ec2.CPUUtilization
metric and the os.cpu.utilization
metric. Group the results by host and bucket the results into 1 minute time spans. Both metrics are combined and considered a single metric series.
| mstats avg(aws.ec2.CPUUtilization) avg(os.cpu.utilization) WHERE index=mymetricdata BY host span=1m
3: Use prestats=t mode with the timechart command
Return a timechart of the number of aws.ec2.CPUUtilization
metric data points for each day.
| mstats prestats=t count(aws.ec2.CPUUtilization) WHERE index=mymetricdata span=1d | timechart span=1d count(aws.ec2.CPUUtilization)
4. Filter the results on a dimension value and split by the values of another dimension
Return the average value of the aws.ec2.CPUUtilization
metric for all measurements with host=foo
and split the results by the values of the app
dimension.
| mstats avg(aws.ec2.CPUUtilization) WHERE host=foo BY app
5. Specify multiple aggregations of multiple metrics
Return the average and maximum of the resident set size and virtual memory size. Group the results by metric_name
and bucket them into 1 minute spans
| mstats avg(os.mem.rss) AS "AverageRSS" max(os.mem.rss) AS "MaxRSS" avg(os.mem.vsz) AS "AverageVMS" max(os.mem.vsz) AS "MaxVMS" WHERE index=mymetricdata BY metric_name span=1m
6. Aggregate a metric across all of your default metrics indexes
Find the median of the aws.ec2.CPUUtilization
metric. Do not include an index filter to search for measurements in all of the default metrics indexes associated with your role.
| mstats median(aws.ec2.CPUUtilization)
7. Get the rate of an accumulating counter metric and group the results by time series
See Perform statistical calculations on metric time series in Metrics for more information.
| mstats rate(spl.intr.resource_usage.PerProcess.data.elapsed) as data.elapsed where index=_metrics BY _timeseries | rename _timeseries AS timeseries
8. Stats-func-value example
Use the <stats-func-value>
syntax to get a count of all of the measurements for the aws.ec2.CPUUtilization
metric in the mymetricdata
index.
| mstats count(_value) WHERE metric_name=aws.ec2.CPUUtilization AND index=mymetricdata
See also
- Related information
- Overview of metrics in Metrics
msearch | multikv |
This documentation applies to the following versions of Splunk® Enterprise: 8.0.3, 8.0.4, 8.0.5, 8.0.6, 8.0.7, 8.0.8, 8.0.9, 8.0.10
Feedback submitted, thanks!