Get metrics in from StatsD
StatsD is a network daemon that runs on the Node.js platform, sending metrics over UDP or TCP. For an overview of StatsD, see Measure Anything, Measure Everything on the Code as Craft website.
StatsD has several implementations, some of which encode dimensions in different ways. The Splunk platform supports the following formats natively:
- Basic StatsD data line metric protocol, which includes
- Expanded StatsD data line metric protocol, which adds sample rate and dimensions.
Splunk supports two
metric_type values for StatsD metric data points:
g, for gauge metrics, and
c, for counter metrics.
Basic StatsD metric protocol
The basic StatsD data line metric protocol just has three fields: the
metric_name, the metric
_value, and the
Expanded StatsD metric protocol
The expanded StatsD data line metric protocol supports dimensions and a sample rate. Sample rates only apply to counter metrics, meaning they have a
For more about formats for metric names and dimensions, see Best practices for metrics.
Example gauge metric
A gauge is a metric that represents a single numerical value that can arbitrarily go up and down. For example, you can use a gauge to represent the number of currently running search jobs, or the temperature in your server room.
Example counter metric, after processing by the Splunk platform
A counter metric counts occurrences of an event. Its value can only increase or be reset to zero. For example, you can use a counter to represent a number of requests served, tasks completed, or errors. For more information about counter metrics, see Investigate counter metrics.
Here is an example of an counter metric that has been processed by the Splunk platform.
Note that this counter metric has a sample rate of
0.5. This means that this counter metric is sampled only 50% of the time by the StatsD client. The Splunk platform adjusts for this by multiplying the metric value by 1/0.5, or 2. This means that the original metric sent from the StatsD client looked like this:
Note that the original metric event had a
_value of 3.
About the sample rate
When large numbers of data points are being produced for a particular counter metric, it can be expensive for the Splunk platform to aggregate them. The StatsD client manages this by implementing a sample rate to reduce the network traffic that it sends to the Splunk platform.
The StatsD client puts the
sample_rate value in the counter metric data point to indicate to the Splunk platform the actual downsampling percentage that it employed. The Splunk platform responds to this by multiplying the
_value of a downsampled counter metric by 1/
For example, say you have a counter metric named
event.login with a
0.1. This means that only 10% of the
event.login data points are passed from the StatsD client to your Splunk platform implementation. The Splunk platform multiplies the
event.login values by 1/0.1, or 10, to adjust for the missed data points. So if your Splunk platform implementation receives a
event.login data point with a
_value of 2, it will change that
_value to 20.
The Splunk platform passes an warning message for
sample_rate values that are not within 0 and 1. The default setting for
sample_rate is 1.
Using other StatsD formats
If you use a StatsD implementation that uses a different format for dimensions from the ones that the Splunk platform supports natively, for example, one that embeds dimensions within the metric name, you can still use metrics in the Splunk platform. However, you'll need to customize Splunk configuration files to specify how to extract dimensions from your format.
Another option is to use StatsD to gather metrics, but use collectd to send the data to the Splunk platform over HTTP. The benefit of this method is that collectd normalizes the dimension format in the metrics data. For more, see Get metrics in from collectd.
Set up a data input for StatsD data
After you configure your data source to send data in the StatsD protocol, create a UDP or TCP data input in the Splunk platform to listen for StatsD data on an open port.
- In Splunk Web, go to Settings > Data inputs.
- Under Local inputs, click Add new next to UDP or TCP, depending on the type of input you want to create.
- For Port, enter the number of the port you are using for StatsD.
- Click Next.
- Click Select Source Type, then select Metrics > statsd.
- For Index, select an existing metrics index. Or, click Create a new index to create one.
If you choose to create an index, in the New Index dialog box:
- Enter an Index Name. User-defined index names must consist of only numbers, lowercase letters, underscores, and hyphens. Index names cannot begin with an underscore or hyphen.
- For Index Data Type, click Metrics.
- Configure additional index properties as needed.
- Click Save.
- Click Review, then click Submit.
When using UDP ports to ingest metric data, you cannot use parallel ingestion or the multiple pipeline sets feature.
Get started with metrics
Extract dimensions for unsupported StatsD formats
This documentation applies to the following versions of Splunk® Enterprise: 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8, 7.3.9
Feedback submitted, thanks!