Best practices for metrics

The following are best practices when working with metrics in the Splunk platform:

Cardinality issues

Metrics search performance decreases as the cardinality of the metric time series stored in a given index and bucket increases. In other words, as the number of unique dimension sets in your metrics data increases, the speed of your metrics searches decreases. The following strategies can help you reduce the time series cardinality in your metrics indexes and buckets.

• Remove unnecessary dimensions from your data. Focus on removing dimensions that have a wide range of unique values, like user IDs or phone numbers.
• Use larger bucket sizes. This can help you reduce the overhead per metrics data point. For example, you might try sizing your buckets to 10GB.
• Split your metrics data across multiple indexes. When you do this, partition the indexes by relative search domains. Keep data that tends to be searched frequently together in the same index. For example, you may want to keep your IT Infrastructure metrics data in one index, and your Sales/Marketing metrics in another index, if those two data sets are rarely searched together.

High result row cardinality also slows down search performance. You can try to mitigate this by increasing the time bucket span to reduce the number of rows returned. You can also reduce the overall time range of your search.

StatsD Format with dimensions extension

If you are indexing data that is in StatsD format, use the StatsD format with the dimensions extension for better performance: cpu.idle:0.5|g|#host:some-hostsplunk.com,app:some-app

Use it instead of the plain StatsD format that combines dimensions with the metric name: cpu.idle.some-hostsplunk.com.some-app

Other best practices

• The _value field of a metric should be of type "Double", not type "String", to avoid causing indexing inefficiencies.

• For a faster response time for REST calls to the Metrics Catalog endpoint, use constrained time windows when applicable. By default, only the last 24 hours of data is searched. See Metrics Catalog endpoint descriptions in the REST API Reference Manual.

• Make sure dimension names do not start with an underscore ( _ ). Such dimensions will not be indexed.