Configure special StatsD input customizations

When you set up a new StatsD metric data input using the Splunk Web interface, as discussed in Get metrics from StatsD, you may not need to perform any additional configurations for that input. However, there are some StatsD input use cases that require you to make special manual customizations to local props.conf and transforms.conf files.

You need to create or update configuration files to enable your Splunk deployment to:

• Convert ingested StatsD metrics data into multiple-measurement metric data points. By default, the Splunk software converts ingested StatsD metrics data into single-measurement metric data points.
• Extract dimensions from the metric names in your StatsD data.

Prerequisites The following prerequisites apply to all of the procedures discussed in this topic.

• Only users with file system access, such as system administrators, can design these StatsD input customizations using configuration files.
• Review the steps in How to edit a configuration file in the Splunk Enterprise Admin Manual.
• You can have configuration files with the same name in your default, local, and app directories. Read Where you can place (or find) your modified configuration files in the Splunk Enterprise Admin Manual.
• Learn how to Use the deployer to distribute apps and configuration updates, in Distributed Search.

Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location. Make changes to the files in the local directory.

Generate multiple-measurement metric data points from a StatsD data input

By default, Splunk software converts StatsD metric data into single-measurement metric data points. Under the single-measurement data point format, each metric data point has one key-value pair for the metric name, and another key-value pair for the corresponding metric measurement. The rest of the fields in the metric data point are dimensions.

Multiple-measurement metric data points, on the other hand, can have one or more metric measurements in the metric data point. Each measurement follows this syntax: metric_name:<metric_name>=<numeric_value>. Each measurement in a metric data point shares the dimensions in that metric data point.

The single-measurement metric data point format is preferred for StatsD metric data ingestion because many StatsD clients embed dimension names in the metric name. When this happens, it is easier to create dimension extraction configurations for single-measurement metric data points. If you decide to have your StatsD inputs generate multiple-measurement data points, do so with the understanding that it will make dimension extraction from the metric_name more difficult than it would otherwise be.

See Configure dimension extraction for unsupported StatsD formats in this topic for more information about configuring StatsD dimension extraction with the props.conf and transforms.conf files.

If you want a StatsD input to generate multiple-measurement metric data points, you need to add STATSD_EMIT_SINGLE_MEASUREMENT_FORMAT = false to the stanza for the StatsD input source type in props.conf. If your metrics data does not already have a source type you must create a custom source type.

Steps

1. Start defining a custom source type by opening the props.conf configuration file for the location you want to use, such as the Search & Reporting app ($SPLUNK_HOME/etc/apps/search/local/) or from the system ($SPLUNK_HOME/etc/system/local). If a props.conf file does not exist in this location, create a text file and save it to that location with the name props.conf.
2. Append a stanza to the props.conf file as follows:

   [<custom_metrics_source_type_name>]
   METRICS_PROTOCOL = STATSD
   STATSD_EMIT_SINGLE_MEASUREMENT_FORMAT = false
   

• custom_metrics_source_type_name: The name of your custom metrics source type.
• METRICS_PROTOCOL: Identifies the metrics protocol used by the incoming metric data for a metrics input. STATSD is selected here because it is a StatsD metric input.
• STATSD_EMIT_SINGLE_MEASUREMENT_FORMAT: Controls whether the StatsD processor generates single-measurement or multiple-measurement metric data points. Set it to false to generate multiple-measurement data points. This setting is valid only when the METRICS_PROTOCOL is set to STATSD.
3. Deploy the props.conf changes to your indexers, if you have any.
   

   If you are employing heavy forwarders in front of your indexers, the props.conf processing takes place on those forwarders, not the indexers. Therefore, you must deploy the props.conf changes to the heavy forwarders.

4. Create a StatsD data input for this source type as described in Set up a data input for StatsD data, and select your custom source type.

Configure dimension extraction for unsupported StatsD formats

Many StatsD clients embed dimension names in the metric name. For example, let's say your StatsD client uses the following line metric protocol format, which is not supported natively by the Splunk platform:

<dimension>.<metric_name>:<value>|<metric_type>

Here's an example of a metric returned using this unsupported format:

10.1.1.198.cpu.percent:75|g

After the Splunk software processes this metric data and performs field extraction on it, the extracted metric name and measurement should be as follows, if you are using the single measurement metric data point format:

metric_name=cpu.percent _value=75

The extracted dimension should be:

ip=10.1.1.198

To create the correct results, you must edit Splunk configuration files or use the REST API to create a custom source type that specifies how to extract dimensions from this metrics data. This requires two procedures:

• In transforms.conf, define a dimension extraction configuration.
• In props.conf, create a custom source type for the StatsD data.

The following procedures are for extracting dimensions from metric_name fields in single-measurement metric data points. They do not apply to StatsD inputs that are set up to generate multiple-measurement metric data points.

Steps for defining a dimension extraction configuration for your StatsD metric data

1. In a text editor, open the transforms.conf configuration file from the local directory for the location you want to use, such as the Search & Reporting app ($SPLUNK_HOME/etc/apps/search/local/) or from the system ($SPLUNK_HOME/etc/system/local).
   
   If a transforms.conf file does not exist in this location, create a text file and save it to that location.
2. In the transforms.conf file, append a stanza for each dimension extraction configuration as follows:

   [statsd-dims:<unique_transforms_stanza_name>]
   REGEX = <regular expression>
   REMOVE_DIMS_FROM_METRIC_NAME = <Boolean>
   

   • unique_transforms_stanza_name: A unique name for this stanza. Stanza names for StatsD dimension extraction configurations must be prefixed with statsd-dims:.
   • REGEX = <regular expression>: A regular expression that defines how to match and extract dimensions from StatsD metrics data. The Splunk platform supports a named capturing-group extraction format (?<dim1>group)(?<dim2>group)... to provide dimension names for the corresponding values that are extracted.
   • REMOVE_DIMS_FROM_METRIC_NAME = <Boolean>: Specifies whether unmatched segments of the StatsD dotted name segment are used as the metric_name.
     
     When set to true, the Splunk software removes dimension values from the measurement and the unmatched portion becomes the metric_name. This setting defaults to true. When set to false, this setting leaves extracted dimension values in the metric_name.
     
     For example, a metric measurement name is x.y.z. The regular expression matches y and z. When REMOVE_DIMS_FROM_METRIC_NAME is true, metric_name is x. When false, metric_name is x.y.z.
3. Save your changes to the transforms.conf file.
4. Deploy the props.conf and transforms.conf changes to your indexers, if you have any.
   

   If you are employing heavy forwarders in front of your indexers, the props.conf and transforms.conf processing takes place on those forwarders, not the indexers. Therefore, you must deploy the props.conf changes to the heavy forwarders.

Steps for defining a custom source type for your StatsD metric data

1. In a text editor, open the props.conf configuration file from the local directory for the location you want to use, such as the Search & Reporting app ($SPLUNK_HOME/etc/apps/search/local/) or from the system ($SPLUNK_HOME/etc/system/local). If a props.conf file does not exist in this location, create a text file and save it to that location.
2. Append a stanza to the props.conf file as follows:

   [<metrics_sourcetype_name>]
   METRICS_PROTOCOL = statsd
   STATSD-DIM-TRANSFORMS = <statsd_dim_stanza_name1>,<statsd_dim_stanza_name2>...
   

• metrics_sourcetype_name: The name of your custom metrics source type.
• METRICS_PROTOCOL: Identifies the metrics protocol used by the incoming metric data for a metrics input. STATSD is selected here because it is a StatsD metric input.
• STATSD-DIM-TRANSFORMS: A comma-separated list of transforms stanza names that specify how to extract dimensions. If only one stanza is used for the source type, and if the related transforms.conf stanza name is same as the metrics_sourcetype_name, this STATSD-DIM-TRANSFORMS setting can be omitted.
3. Save your changes to the props.conf file.
4. Create a data input for this source type as described in Set up a data input for StatsD data, and select your custom source type.

For more about editing these configuration files, see About configuration files, props.conf, and transforms.conf in the Admin Manual.

Examples of configuring dimension extraction

Let's say you have StatsD metrics data such as:

data=mem.percent.used.10.2.3.4.windows:33|g

You need to extract the ipv4 and os dimensions.

If you defined two regular expressions, one for ipv4 and one for os, you would append the following stanzas to your configuration files:

# transforms.conf.example

[statsd-dims:regex_stanza1]
REGEX = (?<ipv4>\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})
REMOVE_DIMS_FROM_METRIC_NAME = true
[statsd-dims:regex_stanza2]
REGEX = \S+\.(?<os>\w+):
REMOVE_DIMS_FROM_METRIC_NAME = true

# props.conf.example

[my_custom_metrics_sourcetype]
METRICS_PROTOCOL = statsd
STATSD-DIM-TRANSFORMS = regex_stanza1, regex_stanza2

Now let's say you can accomplish this same extraction using a single regular expression. In this case, you would append the following stanzas to your configuration files:

# transforms.conf.example

[statsd-dims:my_custom_metrics_sourcetype]
REGEX = (?<ipv4>\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\.(?<os>\w+):
REMOVE_DIMS_FROM_METRIC_NAME = true

# props.conf.example

[my_custom_metrics_sourcetype]
METRICS_PROTOCOL = statsd

Notice that the STATSD-DIM-TRANSFORMS setting in the props.conf configuration file is not needed when only a single regular expression is used for a source type.