Splunk® Enterprise

Metrics

Download manual as PDF

Download topic as PDF

Get metrics in from other sources

If you are gathering metrics from a source that is not natively supported, you can still add this metrics data to a metrics index.

Get metrics in from files in CSV format

There are two accepted formats for CSV files when you use them as inputs for metrics data. The format you use depends on how you want the Splunk software to index the information in the CSV file. Should it index so that each data point has multiple measurements, or so that each data point has only one measurement?

It is more efficient to use metric data points that can contain multiple measurements. When you index metrics data this way you reduce your data storage costs and can benefit from improved search performance.

Set metrics CSV source types and data inputs

If your metrics data is in CSV format, use the metrics_csv pre-trained source type. It can handle both CSV metrics formats.

Create a data input to add your CSV data to a metrics index. The input uses the pretrained metrics_csv source type. The data input should have:

  • Source type: Metrics > metrics_csv
  • Index: a metrics index

After you set up your metrics_csv input, you should have the following inputs.conf configuration on your universal forwarder. It monitors the CSV data and sends it to the metrics indexer.

#inputs.conf

[monitor:///opt/metrics_data]
index = metrics
sourcetype = metrics_csv

You should also have the following indexes.conf configuration on the metrics indexer:

#indexes.conf

[metrics]
homePath = $SPLUNK_DB/metrics/db
coldPath = $SPLUNK_DB/metrics/colddb
thawedPath = $SPLUNK_DB/metrics/thaweddb
datatype = metric
maxTotalDataSizeMB = 512000

See Monitor files and directories in the Getting Data In manual, and Create metrics indexes in the Managing Indexers and Clusters of Indexers manual.

Format a CSV file for multiple-measurement metric data points

When you format a CSV file for multiple-measurement metric data points, the first column header is _time, the metric timestamp. It is a required field.

This is followed by one or more column headers for each metric measurement. Each measurement column header follows this syntax: metric_name:<metric_name>.

The Splunk software considers additional columns that are not a timestamp or a measurement to be dimensions.

Each row of the CSV table is a separate metric data point.

Field name Required Description Example
_time Yes The metric timestamp. The format is epoch time (elapsed time since 1/1/1970), in milliseconds. 1504907933.000
metric_name:<metric_name> Yes A measurement for a specific metric, as specified by <metric_name>, such as metric_name:os.cpu.idle or metric_name:max.size.kpbs. Their values are always numeric. 13.34
dimensions No All other fields are treated as dimensions. For a dimension named ip, a value of 192.0.2.1.

Here is an example of a CSV file that is formatted for multiple-measurement metric data points. The first column is _time, the metric timestamp. The middle three columns are measurements. The last two columns are dimensions.

"_time","metric_name:cpu.usr","metric_name:cpu.sys","metric_name:cpu.idle","dc","host"
"1562020701",11.12,12.23,13.34,"east","east.splunk.com"
"1562020702",21.12,22.33,23.34,"west","west.splunk.com"

This CSV file example contains the same information as the example CSV file for single-measurement metric data points in the following section. However, because it uses two data points for this information instead of six, it will take up less space on disk when it is indexed.

Format a CSV file for single-measurement metric data points

When you format a CSV file for single-measurement metric data points, the first three columns are fields that are required for single-measurement metric data points:

  • metric_timestamp
  • metric_name
  • _value.

All additional columns are considered to be dimensions.

During the ingestion and indexing process the metric_name and _value measurements will merged into the metric_name:<metric_name>=<numeric_value> format.

Field name Required Description Example
metric_timestamp Yes The timestamp format is epoch time (elapsed time since 1/1/1970), in milliseconds. 1504907933.000
metric_name Yes The metric name using dotted-string notation. os.cpu.percent
_value Yes The numerical value associated with the metric_name. 42.12345
dimensions No All other fields are treated as dimensions. ip

Here is an example of a CSV file that is formatted for single-measurement metric data points. The first three columns of the table are the fields that are required for single-measurement metric data points. All additional columns are dimensions. This CSV file has dc and host as dimensions.

"metric_timestamp","metric_name","_value","dc","host"
"1562020701","cpu.usr",11.12,"east","east.splunk.com"
"1562020701","cpu.sys",12.23,"east","east.splunk.com"
"1562020701","cpu.idle",13.34,"east","east.splunk.com"
"1562020702","cpu.usr",21.12,"west","west.splunk.com"
"1562020702","cpu.sys",22.33,"west","west.splunk.com"
"1562020702","cpu.idle",23.34,"west","west.splunk.com"

If you compare this example to the example for multiple-measurement metric data points, you can see how the single-metric format would take up more space on disk. This table contains the same information as the multiple-measurement table. However, this table uses six data points where the multiple-measurement table only uses two.

Get metrics in from clients over TCP/UDP

You can add metrics data from a client that is not natively supported to a metrics index by manually configuring a source type for your data, then defining regular expressions to specify how the Splunk software should extract the required metrics fields. See Metrics data format.

For example, let's say you are using Graphite. The Graphite plaintext protocol format is:

<metric path> <metric value> <metric timestamp>

A sample metric might be:

510fcbb8f755.sda2.diskio.read_time 250 1487747370

To index these metrics, edit Splunk configuration files to manually specify how to extract fields.

Configure field extraction by editing configuration files

  1. Define a custom source type for your metrics data.
    1. In a text editor, open the props.conf configuration file from the local directory for the location you want to use, such as the Search & Reporting app ($SPLUNK_HOME/etc/apps/search/local/) or the system ($SPLUNK_HOME/etc/system/local). If a props.conf file does not exist in this location, create a text file and save it to that location.
    2. Append a stanza to the props.conf file as follows:
      # props.conf
      
      [<metrics_sourcetype_name>]
      TIME_PREFIX = <regular expression>
      TIME_FORMAT = <strptime-style format>
      TRANSFORMS-<class> = <transform_stanza_name>
      NO_BINARY_CHECK = true
      SHOULD_LINEMERGE = false
      pulldown_type = 1
      category = Metrics
      
      • metrics_sourcetype_name Name of your custom metrics source type.
      • TIME_PREFIX = regular expression: A regular expression that indicates where the timestamp is located.
      • TIME_FORMAT = strptime-style format: A strptime format string used to extract the date. For more about strptime, see Configure timestamp recognition in the Getting Data In manual.
      • TRANSFORMS-<class> = <transform_stanza_name>: class is a unique literal string that identifies the namespace of the field to extract. transform_stanza_name is the name of the name of your stanza in transforms.conf that indicates how to extract the field.
  2. Define a regular expression for each metrics field to extract.
    1. In a text editor, open the transforms.conf configuration file from the local directory for the location you want to use, such as the Search & Reporting app ($SPLUNK_HOME/etc/apps/search/local/) or the system ($SPLUNK_HOME/etc/system/local). If a transforms.conf file does not exist in this location, create a text file and save it to that location.
    2. Append a stanza for each regular expression as follows:
      # transforms.conf
      
      [<transform_stanza_name>]
      REGEX = <regular expression>
      FORMAT = <string>
      WRITE_META = true
      
      • transform_stanza_name: A unique name for this stanza.
      • REGEX = <regular expression>: A regular expression that defines how to match and extract metrics fields from this metrics data.
      • FORMAT = <string>: A string that specifies the format of the metrics event.
  3. Create a data input for this source type as described in Set up a data input for StatsD data, and select your custom source type.


For more about editing these configuration files, see About configuration files, props.conf, and transforms.conf in the Admin Manual.

Example of configuring field extraction

This example shows how to create a custom source type and regular expressions to extract fields from Graphite metrics data.

# props.conf.example

[graphite_plaintext]
TIME_PREFIX = \s(\d{0,10})$
TIME_FORMAT =  %s
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
pulldown_type = 1
TRANSFORMS-graphite-host = graphite_host
TRANSFORMS-graphite-metricname = graphite_metric_name
TRANSFORMS-graphite-metricvalue = graphite_metric_value
category = Metrics
# transforms.conf.example

[graphite_host]
REGEX = ^(\S[^\.]+)
FORMAT = host::$1
DEST_KEY = MetaData:Host

[graphite_metric_name]
REGEX = \.(\S+)
FORMAT = metric_name::graphite.$1
WRITE_META = true

[graphite_metric_value]
REGEX = \w+\s+(\d+.?\d+)\s+
FORMAT = _value::$1
WRITE_META = true

Get metrics in from clients over HTTP or HTTPS

If you want to send metrics data in JSON format from a client that is not natively supported to a metrics index over HTTP or HTTPS, use the HTTP Event Collector (HEC) and the /collector REST API endpoint.

Create a data input and token for HEC

  1. In Splunk Web, click Settings > Data Inputs.
  2. Under Local Inputs, click HTTP Event Collector.
  3. Verify that HEC is enabled.
    1. Click Global Settings.
    2. For All Tokens, click Enabled if this button is not already selected.
    3. Click Save.
  4. Configure an HEC token for sending data by clicking New Token.
  5. On the Select Source page, for Name, enter a token name, for example "Metrics token".
  6. Leave the other options blank or unselected.
  7. Click Next.
  8. On the Input Settings page, for Source type, click New.
  9. In Source Type, type a name for your new source type.
  10. For Source Type Category, select Metrics.
  11. Optionally, in Source Type Description type a description.
  12. Next to Default Index, select your metrics index, or click Create a new index to create one.
    If you choose to create an index, in the New Index dialog box:
    1. Enter an Index Name.
    2. For Index Data Type, click Metrics.
    3. Configure additional index properties as needed.
    4. Click Save.
  13. Click Review, and then click Submit.
  14. Copy the Token Value that is displayed. This HEC token is required for sending data.


See Getting data in with HTTP Event Collector on the Splunk Developer Portal.

Send data to a metrics index over HTTP

Use the /collector REST API endpoint and your HEC token to send data directly to a metrics index as follows:

http://<Splunk_host>:<HTTP_port>/services/collector \
-H "Authorization: Splunk <HEC_token>"                                                \
-d "<metrics_data>"

You need to provide the following values:

  • Splunk host machine (IP address, host name, or load balancer name)
  • HTTP port number
  • HEC token value
  • Metrics data, which requires an "event" field set to "metric".

For more about HEC, see Getting data in with HTTP Event Collector and Event formatting on the Splunk Developer Portal.

For more about the /collector endpoint, see /collector in the REST API Reference Manual.

Example of sending metrics using HEC

The following example shows a command that sends a metric data point to a metrics index, with the following values:

  • Splunk host machine: "localhost"
  • HTTP port number: "8088"
  • HEC token value: "b0221cd8-c4b4-465a-9a3c-273e3a75aa29"
curl -k https://localhost:8088/services/collector                     \
-H "Authorization: Splunk b0221cd8-c4b4-465a-9a3c-273e3a75aa29"       \
-d '{"time": 1486683865.000,"event":"metric","source":"disk","host":"host_1.splunk.com","fields":{"region":"us-west-1","datacenter":"dc1","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","metric_name:cpu.usr": 11.12,"metric_name:cpu.sys": 12.23, "metric_name:cpu.idle": 13.34}}'

The measurements for this metric data point appear at the end of the JSON blob. They follow a multiple-metric format that uses the "metric_name:<metric_name>":<numeric_value> syntax.

The multiple-metric JSON format

Versions of the Splunk platform previous to 8.0.0 used a JSON format that only supported one metric measurement per JSON object. This resulted in metric data points that could only contain one measurement at a time.

Version 8.0.0 of the Splunk platform supports a JSON format which allows each JSON object to contain measurements for multiple metrics. These JSON objects generate multiple-measurement metric data points. Multiple-measurement metric data points take up less space on disk and can improve search performance.

Here is an example of a JSON object in the multiple-metric format.

{
  "time": 1486683865,
  "source": "metrics",
  "sourcetype": "perflog",
  "host": "host_1.splunk.com",
  "fields": {
    "region": "us-west-1",
    "datacenter": "dc2",
    "rack": "63",
    "os": "Ubuntu16.10",
    "arch": "x64",
    "team": "LON",
    "service": "6",
    "service_version": "0",
    "service_environment": "test",
    "path": "/dev/sda1",
    "fstype": "ext3"
    "metric_name:cpu.usr": 11.12,
    "metric_name:cpu.sys": 12.23,
    "metric_name:cpu.idle": 13.34
  }
}
Last modified on 22 October, 2019
PREVIOUS
Get metrics in from collectd
  NEXT
Convert event logs to metric data points

This documentation applies to the following versions of Splunk® Enterprise: 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.0.5


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters