Configure special StatsD input customizations
When you set up a new StatsD metric data input using the Splunk Web interface, as discussed in Get metrics from StatsD, you may not need to perform any additional configurations for that input. However, there are some StatsD input use cases that require you to make special manual customizations to local props.conf
and transforms.conf
files.
You need to create or update configuration files to enable your Splunk deployment to:
- Convert ingested StatsD metrics data into multiple-measurement metric data points. By default, the Splunk software converts ingested StatsD metrics data into single-measurement metric data points.
- Extract dimensions from the metric names in your StatsD data.
Prerequisites The following prerequisites apply to all of the procedures discussed in this topic.
- Only users with file system access, such as system administrators, can design these StatsD input customizations using configuration files.
- Review the steps in How to edit a configuration file in the Splunk Enterprise Admin Manual.
- You can have configuration files with the same name in your default, local, and app directories. Read Where you can place (or find) your modified configuration files in the Splunk Enterprise Admin Manual.
- Learn how to Use the deployer to distribute apps and configuration updates, in Distributed Search.
Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location. Make changes to the files in the local directory.
Generate multiple-measurement metric data points from a StatsD data input
By default, Splunk software converts StatsD metric data into single-measurement metric data points. Under the single-measurement data point format, each metric data point has one key-value pair for the metric name, and another key-value pair for the corresponding metric measurement. The rest of the fields in the metric data point are dimensions.
Multiple-measurement metric data points, on the other hand, can have one or more metric measurements in the metric data point. Each measurement follows this syntax: metric_name:<metric_name>=<numeric_value>
. Each measurement in a metric data point shares the dimensions in that metric data point.
The single-measurement metric data point format is preferred for StatsD metric data ingestion because many StatsD clients embed dimension names in the metric name. When this happens, it is easier to create dimension extraction configurations for single-measurement metric data points. If you decide to have your StatsD inputs generate multiple-measurement data points, do so with the understanding that it will make dimension extraction from the metric_name
more difficult than it would otherwise be.
See Configure dimension extraction for unsupported StatsD formats in this topic for more information about configuring StatsD dimension extraction with the props.conf
and transforms.conf
files.
If you want a StatsD input to generate multiple-measurement metric data points, you need to add STATSD_EMIT_SINGLE_MEASUREMENT_FORMAT = false
to the stanza for the StatsD input source type in props.conf
. If your metrics data does not already have a source type you must create a custom source type.
Steps
- Start defining a custom source type by opening the
props.conf
configuration file for the location you want to use, such as the Search & Reporting app ($SPLUNK_HOME/etc/apps/search/local/
) or from the system ($SPLUNK_HOME/etc/system/local
). If aprops.conf
file does not exist in this location, create a text file and save it to that location with the nameprops.conf
. - Append a stanza to the
props.conf
file as follows:[<custom_metrics_source_type_name>] METRICS_PROTOCOL = STATSD STATSD_EMIT_SINGLE_MEASUREMENT_FORMAT = false
custom_metrics_source_type_name
: The name of your custom metrics source type.METRICS_PROTOCOL
: Identifies the metrics protocol used by the incoming metric data for a metrics input.STATSD
is selected here because it is a StatsD metric input.STATSD_EMIT_SINGLE_MEASUREMENT_FORMAT
: Controls whether the StatsD processor generates single-measurement or multiple-measurement metric data points. Set it tofalse
to generate multiple-measurement data points. This setting is valid only when theMETRICS_PROTOCOL
is set toSTATSD
.- Deploy the
props.conf
changes to your indexers, if you have any.
If you are employing heavy forwarders in front of your indexers, the
props.conf
processing takes place on those forwarders, not the indexers. Therefore, you must deploy theprops.conf
changes to the heavy forwarders. - Create a StatsD data input for this source type as described in Set up a data input for StatsD data, and select your custom source type.
Configure dimension extraction for unsupported StatsD formats
Many StatsD clients embed dimension names in the metric name. For example, let's say your StatsD client uses the following line metric protocol format, which is not supported natively by the Splunk platform:
<dimension>.<metric_name>:<value>|<metric_type>
Here's an example of a metric returned using this unsupported format:
10.1.1.198.cpu.percent:75|g
After the Splunk software processes this metric data and performs field extraction on it, the extracted metric name and measurement should be as follows, if you are using the single measurement metric data point format:
metric_name=cpu.percent _value=75
The extracted dimension should be:
ip=10.1.1.198
To create the correct results, you must edit Splunk configuration files or use the REST API to create a custom source type that specifies how to extract dimensions from this metrics data. This requires two procedures:
- In
transforms.conf
, define a dimension extraction configuration. - In
props.conf
, create a custom source type for the StatsD data.
The following procedures are for extracting dimensions from metric_name
fields in single-measurement metric data points. They do not apply to StatsD inputs that are set up to generate multiple-measurement metric data points.
Steps for defining a dimension extraction configuration for your StatsD metric data
- In a text editor, open the
transforms.conf
configuration file from the local directory for the location you want to use, such as the Search & Reporting app ($SPLUNK_HOME/etc/apps/search/local/
) or from the system ($SPLUNK_HOME/etc/system/local
).
If atransforms.conf
file does not exist in this location, create a text file and save it to that location. - In the
transforms.conf
file, append a stanza for each dimension extraction configuration as follows:[statsd-dims:<unique_transforms_stanza_name>] REGEX = <regular expression> REMOVE_DIMS_FROM_METRIC_NAME = <Boolean>
unique_transforms_stanza_name
: A unique name for this stanza. Stanza names for StatsD dimension extraction configurations must be prefixed withstatsd-dims:
.REGEX = <regular expression>
: A regular expression that defines how to match and extract dimensions from StatsD metrics data. The Splunk platform supports a named capturing-group extraction format(?<dim1>group)(?<dim2>group)...
to provide dimension names for the corresponding values that are extracted.REMOVE_DIMS_FROM_METRIC_NAME = <Boolean>
: Specifies whether unmatched segments of the StatsD dotted name segment are used as themetric_name
.
When set totrue
, the Splunk software removes dimension values from the measurement and the unmatched portion becomes themetric_name
. This setting defaults totrue
. When set tofalse
, this setting leaves extracted dimension values in themetric_name
.
For example, a metric measurement name isx.y.z
. The regular expression matchesy
andz
. When REMOVE_DIMS_FROM_METRIC_NAME istrue
,metric_name
isx
. Whenfalse
,metric_name
isx.y.z
.
- Save your changes to the
transforms.conf
file. - Deploy the
props.conf
andtransforms.conf
changes to your indexers, if you have any.
If you are employing heavy forwarders in front of your indexers, the
props.conf
andtransforms.conf
processing takes place on those forwarders, not the indexers. Therefore, you must deploy theprops.conf
changes to the heavy forwarders.
Steps for defining a custom source type for your StatsD metric data
- In a text editor, open the
props.conf
configuration file from the local directory for the location you want to use, such as the Search & Reporting app ($SPLUNK_HOME/etc/apps/search/local/) or from the system ($SPLUNK_HOME/etc/system/local). If aprops.conf
file does not exist in this location, create a text file and save it to that location. - Append a stanza to the props.conf file as follows:
[<metrics_sourcetype_name>] METRICS_PROTOCOL = statsd STATSD-DIM-TRANSFORMS = <statsd_dim_stanza_name1>,<statsd_dim_stanza_name2>...
metrics_sourcetype_name
: The name of your custom metrics source type.METRICS_PROTOCOL
: Identifies the metrics protocol used by the incoming metric data for a metrics input.STATSD
is selected here because it is a StatsD metric input.STATSD-DIM-TRANSFORMS
: A comma-separated list of transforms stanza names that specify how to extract dimensions. If only one stanza is used for the source type, and if the relatedtransforms.conf
stanza name is same as themetrics_sourcetype_name
, thisSTATSD-DIM-TRANSFORMS
setting can be omitted.- Save your changes to the
props.conf
file. - Create a data input for this source type as described in Set up a data input for StatsD data, and select your custom source type.
For more about editing these configuration files, see About configuration files, props.conf, and transforms.conf in the Admin Manual.
Examples of configuring dimension extraction
Let's say you have StatsD metrics data such as:
data=mem.percent.used.10.2.3.4.windows:33|g
You need to extract the ipv4
and os
dimensions.
If you defined two regular expressions, one for ipv4
and one for os
, you would append the following stanzas to your configuration files:
# transforms.conf.example [statsd-dims:regex_stanza1] REGEX = (?<ipv4>\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}) REMOVE_DIMS_FROM_METRIC_NAME = true [statsd-dims:regex_stanza2] REGEX = \S+\.(?<os>\w+): REMOVE_DIMS_FROM_METRIC_NAME = true
# props.conf.example [my_custom_metrics_sourcetype] METRICS_PROTOCOL = statsd STATSD-DIM-TRANSFORMS = regex_stanza1, regex_stanza2
Now let's say you can accomplish this same extraction using a single regular expression. In this case, you would append the following stanzas to your configuration files:
# transforms.conf.example [statsd-dims:my_custom_metrics_sourcetype] REGEX = (?<ipv4>\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\.(?<os>\w+): REMOVE_DIMS_FROM_METRIC_NAME = true
# props.conf.example [my_custom_metrics_sourcetype] METRICS_PROTOCOL = statsd
Notice that the STATSD-DIM-TRANSFORMS
setting in the props.conf
configuration file is not needed when only a single regular expression is used for a source type.
Get metrics in from StatsD | Configure StatsD dimension extraction through REST API endpoints |
This documentation applies to the following versions of Splunk Cloud Platform™: 9.2.2406 (latest FedRAMP release), 8.2.2112, 8.2.2202, 9.0.2205, 8.2.2201, 8.2.2203, 9.0.2208, 9.0.2209, 9.0.2303, 9.0.2305, 9.1.2308, 9.1.2312, 9.2.2403
Feedback submitted, thanks!