mstats command to analyze metrics. This command performs statistics on the
dimension fields in metric indexes. You can use
mstats in historical searches and real-time searches. When you use
mstats in a real-time search with a time window, a historical search runs first to backfill the data.
mstats command provides the best search performance when you use it to search a single
metric_name value or a small number of
| mstats [prestats=<bool>] [append=<bool>] [backfill=<bool>] [update_period=<integer>] <stats-metric-term>...
WHERE [<logical-expression>]... [ (BY) <field-list> ] [<span-length>]
- Syntax: (<stats-func>"("<metric_name>")" [AS <string>])...
- Description: Perform statistical calculations on one or more
metric_namefields. You can rename the result of each function using the
prestatsis set to
metric_namemust be in parenthesis.
- The following table lists the supported functions for the
mstatscommand by type of function. Use the links in the table to see descriptions and examples for each function.
Type of function Supported functions and syntax Aggregate functions
For an overview of using functions with commands, see Statistical and charting functions.
- Syntax: append=<bool>
- Description: Valid only when
prestats=true. This argument runs the
mstatscommand and adds the results to an existing set of results instead of generating new results.
- Default: false
- Syntax: backfill=<bool>
- Description: Valid only with real-time searches that have a time window. When
mstatscommand runs a search on historical data to backfill events before searching the in-memory real-time data.
- Default: true
- Syntax: <field>, ...
- Description: Specifies one or more fields to group the results by. Required when using the BY clause.
- Syntax: <time-opts>|<search-modifier>|((NOT)? <logical-expression>)|<index-expression>|<comparison-expression>|(<logical-expression> (OR)? <logical-expression>)
- Description: An expression describing the filters that are applied to your search. Includes time and search modifiers, comparison expressions, and index expressions. See the following sections for descriptions of each of these logical expression components.
- Cannot filter on
metric_name. Does not support CASE or TERM directives. You also cannot use the WHERE clause to search for terms or phrases.
- Syntax: prestats=true | false
- Description: Specifies whether to use the prestats format. The prestats format is a Splunk internal format that is designed to be consumed by commands that generate aggregate calculations. When you use the prestats format, you can pipe the data into the chart, stats, or timechart commands, which are designed to accept the prestats format. When
prestatsis set to
true, instructions with the
ASclause are not relevant. The field names for the aggregates are determined by the command that consumes the prestats format and produces the aggregate output.
- Default: false
- Syntax: span=<int><timescale>
- Description: The span of each time bin. If used with a <timescale>, the
span-lengthis treated as a time range. If not, this is an absolute bucket length. If you do not specify a <span-length>, the default is
auto, which means that the number of time buckets adjusts to produce a reasonable number of results. For example, if seconds are used initially for the <timespan> and too many results are returned, the <timescale> is changed to a longer value, such as minutes, to return fewer time buckets.
- Syntax: <sec> | <min> | <hr> | <day> | <month>
- Description: Time scale units. For the
mstatscommand, the <timescale> does not support subseconds.
- Default: sec
Time scale Syntax Description <sec> s | sec | secs | second | seconds Time scale in seconds. <min> m | min | mins | minute | minutes Time scale in minutes. <hr> h | hr | hrs | hour | hours Time scale in hours. <day> d | day | days Time scale in days. <month> mon | month | months Time scale in months.
- Syntax: update_period=<integer>
- Description: Valid only with real-time searches. Specifies how frequently, in milliseconds, the real-time summary for the
mstatscommand is updated. A larger number means less frequent updates to the summary and less impact on index processing.
- Default: 1000 (1 second)
Logical expression options
- Syntax: <field><comparison-operator><value> | <field> IN (<value-list>)
- Description: Compares a field to a literal value or provides a list of values that can appear in the field.
- Syntax: <term> | <search-modifier>
- Description: Describes the events you want to retrieve from the index using search terms and search modifiers.
- Syntax: [<timeformat>] (<time-modifier>)*
- Description: Describes the format of the
<endtime>terms of the search.
Comparison expression options
- Syntax: = | != | < | <= | > | >=
- Description: You can use comparison operators when searching field-value pairs. Comparison expressions with the
equal ( = )or
not equal ( != )operator compare string values. For example, "1" does not match "1.0". Comparison expressions with greater than or less than operators
< > <= >=numerically compare two numbers and lexicographically compare other values. See Usage.
- Syntax: <string>
- Description: The name of a field.
- Syntax: <literal-value>
- Description: In comparison expressions, this is the literal number or string value of a field.
- Syntax: (<literal-value>, <literal-value>, ...)
- Description: Used with the IN operator to specify two or more values. For example use
error IN (400, 402, 404, 406)instead of
error=400 OR error=402 OR error=404 OR error=406
Index expression options
- Syntax: "<string>"
- Description: Specify keywords or quoted phrases that the search must match. When searching for strings, quoted strings, or anything that is not a search modifier, Splunk software searches the
_rawfield for the matching events or results.
- Syntax: <sourcetype-specifier> | <host-specifier> | <source-specifier> | <splunk_server-specifier>
- Description: Search for events from specified fields. For example, search for one or a combination of hosts, sources, and source types. See searching with default fields in the Knowledge Manager manual.
- Syntax: sourcetype=<string>
- Description: Search for events from the specified sourcetype field.
- Syntax: host=<string>
- Description: Search for events from the specified host field.
- Syntax: source=<string>
- Description: Search for events from the specified source field.
- Syntax: splunk_server=<string>
- Description: Search for events from a specific server. Use "local" to refer to the search head.
- Syntax: timeformat=<string>
- Description: Set the time format for starttime and endtime terms.
- Default: timeformat=%m/%d/%Y:%H:%M:%S.
- Syntax: starttime=<string> | endtime=<string> | earliest=<time_modifier> | latest=<time_modifier>
- Description: Specify start and end times using relative or absolute time.
You can also use the earliest and latest attributes to specify absolute and relative time ranges for your search. For more about this time modifier syntax, see About search time rangesin the ''Search Manual''.
- Syntax: starttime=<string>
- Description: Events must be later or equal to this time. The
- Syntax: endtime=<string>
- Description: All events must be earlier or equal to this time.
For a list of time modifiers, see Time modifiers for search.
Generating commands use a leading pipe character and should be the first command in a search, except when
append=true is specified with the command.
mstats command to search metrics data. The metrics data uses a specific format for the metrics fields. See Metrics data format in Metrics.
mstats command supports wildcard characters in any search filter. However, you cannot use wildcard characters in the GROUP BY clause.
If you are using wildcard characters in your aggregations and you are renaming them, your rename must have matching wildcards.
For example, this search is invalid:
| mstats sum(*.free) as FreeSum
This search is valid:
| mstats sum(*.free) as *FreeSum
Numeric aggregations are only allowed on specific values of the
metric_name field. The metric name must be enclosed in parenthesis. If there is no data for the specified
metric_name in parenthesis, the search is still valid.
Aggregations are not allowed for values of any other field, including the
Use the WHERE clause to filter by any of the supported dimensions.
The WHERE clause cannot filter by
metric_name. Filtering by
metric_name is performed based on the
metric_name fields specified with the
If you do not specify an index name in the WHERE clause, the
mstats command returns results from the default metrics indexes associated with your role. If you do not specify an index name and you have no default metrics indexes associated with your role,
mstats returns no results. To search against all metrics indexes use
For more information about defining default metrics indexes for a role, see Add and edit roles with Splunk Web in Securing Splunk Enterprise.
You can group results by the
You can also group by time. You must specify a timespan using the <span-length> argument to group by time buckets. For example,
span=auto. The <span-length> argument is separate from the BY clause and can be placed at any point in the search between clauses.
Grouping by the
_time fields is not allowed.
Memory and mstats search performance
A pair of
limits.conf settings strike a balance between the performance of
mstats searches and the amount of memory they use during the search process, in RAM and on disk. If your
mstats searches are consistently slow to complete you can adjust these settings to improve their performance, but at the cost of increased search-time memory usage, which can lead to search failures.
If you have Splunk Cloud you will need to file a Support ticket to change these settings.
For more information, see Memory and stats search performance in the Search Manual.
mstats command does not recognize the following time-related dimensions.
Lexicographical order sorts items based on the values used to encode the items in computer memory. In Splunk software, this is almost always UTF-8 encoding, which is a superset of ASCII.
- Numbers are sorted before letters. Numbers are sorted based on the first digit. For example, the numbers 10, 9, 70, 100 are sorted lexicographically as 10, 100, 70, 9.
- Uppercase letters are sorted before lowercase letters.
- Symbols are not standard. Some symbols are sorted before numeric values. Other symbols are sorted before or after letters.
You can specify a custom sort order that overrides the lexicographical order. See the blog Order Up! Custom Sort Orders.
In version 7.1 of the Splunk platform, elements of the
mstats syntax were deprecated and replaced with an improved syntax. The deprecated syntax has the following requirements:
- All statistical functions must be performed on the
- A single
metric_namefilter must be specified for the
The replacement syntax makes it much easier to perform statistical aggregations across multiple metrics in a single
1. Calculate a single metric grouped by time
Return the average value of the
aws.ec2.CPUUtilization metric in the
mymetricdata metric index. Bucket the results into 30 second time spans.
| mstats avg(aws.ec2.CPUUtilization) WHERE index=mymetricdata span=30s
2. Combine metrics with different metric names
Return the average value of both the
aws.ec2.CPUUtilization metric and the
os.cpu.utilization metric. Group the results by host and bucket the results into 1 minute time spans. Both metrics are combined and considered a single metric series.
| mstats avg(aws.ec2.CPUUtilization) avg(os.cpu.utilization) WHERE index=mymetricdata BY host span=1m
3: Use prestats=t mode with the timechart command
Return a timechart of the number of
aws.ec2.CPUUtilization metric data points for each day.
| mstats prestats=t count(aws.ec2.CPUUtilization) WHERE index=mymetricdata span=1d | timechart span=1d count(aws.ec2.CPUUtilization)
4. Filter the results on a dimension value and split by the values of another dimension
Return the average value of the
aws.ec2.CPUUtilization metric for all measurements with
host=foo and split the results by the values of the
| mstats avg(aws.ec2.CPUUtilization) WHERE host=foo BY app
5. Specify multiple aggregations of multiple metrics
Return the average and maximum of the resident set size and virtual memory size. Group the results by
metric_name and bucket them into 1 minute spans
| mstats avg(os.mem.rss) AS "AverageRSS" max(os.mem.rss) AS "MaxRSS" avg(os.mem.vsz) AS "AverageVMS" max(os.mem.vsz) AS "MaxVMS" WHERE index=mymetricdata BY metric_name span=1m
6. Aggregate a metric across all of your default metrics indexes
Find the median of the
aws.ec2.CPUUtilization metric. Do not include an index filter to search for measurements in all of the default metrics indexes associated with your role.
| mstats median(aws.ec2.CPUUtilization)
7. Deprecated syntax example
Use the deprecated syntax to get a count of all of the measurements for the
aws.ec2.CPUUtilization metric in the
| mstats count(_value) WHERE metric_name=aws.ec2.CPUUtilization AND index=mymetricdata
Overview of metrics in Metrics
This documentation applies to the following versions of Splunk® Enterprise: 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8, 7.3.9