Splunk® Data Stream Processor

Use the Data Stream Processor

Acrobat logo Download manual as PDF


On April 3, 2023, Splunk Data Stream Processor will reach its end of sale, and will reach its end of life on February 28, 2025. If you are an existing DSP customer, please reach out to your account team for more information.
This documentation does not apply to the most recent version of Splunk® Data Stream Processor. For documentation on the most recent version, go to the latest release.
Acrobat logo Download topic as PDF

Formatting metrics data in DSP

For the Write to the Splunk platform with batching, Write to the Splunk platform, and the Write to the Splunk platform (default for environment) sink functions to properly transform your event into metrics format, your event must follow the metrics schema shown in the examples below.

Use the following table and examples as a guide for how to format your data so that it is indexed appropriately into Splunk Enterprise.

DSP field HEC metrics JSON Data Type Notes
body fields list of maps An array of one or more JSON objects following the schema defined here.

There must be at least one metric in the body field or your record is dropped.

sourcetype or source_type sourcetype string If not present, no sourcetype is included in the HEC metrics JSON.
timestamp time long integer The Data Stream Processor uses Unix epoch time in milliseconds. Your timestamp is automatically converted to Splunk epoch time format <sec>.<ms>. If blank or negative, time is set to now.
source source string If not present, no source is included in the HEC metrics JSON.
host host string If not present, no host is included in the HEC metrics JSON.
attributes fields map<string, any> Defines default dimensions used in the HEC metrics JSON. For information on the attributes schema, see the attributes row in the metrics schema. For examples on how these default dimensions are used, see example 4 and example 5.
id N/A string A DSP metrics field ignored by HEC.
kind N/A string A DSP metrics field ignored by HEC.
nanos N/A integer A DSP metrics field ignored by HEC.
any custom fields N/A any All custom fields are ignored when creating the HEC metrics JSON.
N/A index string To set the index in HEC metrics JSON, you must pass the index name as an argument in the Write Splunk Enterprise or Write Index functions. If no index is selected, your data is sent to the default index associated with your HEC token.

Example 1: The event's body is null

DSP metric event:

Event{nanos=null, kind="metric", host=null, attributes=null, source_type="test-sourcetype", id=null, source="test-source", body=null, timestamp=1000000000000}

HEC metric JSON:

null (event is dropped)

Explanation: Your event body must have at least one metric or else your event is dropped.

Example 2: The event has a body field present

DSP metric event:

Event{nanos=null,kind="metric",host=null, attributes=null, source_type="test-sourcetype", id=null, source="test-source", body=[{name:"test-metric_name", "value":2.2}],timestamp=1000000000001}

HEC metric JSON:

{"event":"metric","time":"1000000000.001","source":"test-source", "sourcetype":"test-sourcetype","index":"myindex123", "fields":{"_value":2.2,"metric_name":"test-metric_name"}}

Explanation: The body field has the correct DSP metrics schema. DSP name and value maps to HEC metrics JSON metric_name and _value respectively.

Example 3: The event's body contains two maps

DSP metric event:

Event{nanos=null, kind="metric", host=null, attributes=null, source_type="test-sourcetype", id=null, source="test-source", body=[{name:"test-metric_name", "value":2.2}, {name:"test-metric_name-2", "value":4.4}], timestamp=1000000000001}

HEC metric JSON:

{"event":"metric","time":"1000000000.001","source":"test-source","sourcetype":"test-sourcetype","index":"myindex123","fields":{"_value":2.2,"metric_name":"test-metric_name"}}

{"event":"metric","time":"1000000000.001","source":"test-source","sourcetype":"test-sourcetype","index":"myindex123","fields":{"_value":4.4,"metric_name":"test-metric_name-2"}}

Explanation: Because the body field has two maps in the array, two separate HEC metric JSON objects are created.

Example 4: The event has additional attributes

DSP metric event:

Event{nanos=null, kind="metric", host=null, 
       attributes={default_unit:"1", default_type:"g", 
                   default_dimensions={hostname:"test-hostname", 
                                       service:"test-service"}}, 
       source_type="test-sourcetype", id=null, source="test-source", 
       body=[{name:"test-metric_name", "value":2.2}], timestamp=1000000000001}

HEC metric JSON

{"event":"metric","time":"1000000000.001","source":"test-source","sourcetype":"test-sourcetype","index":"myindex123","fields":{"_value":2.2,"metric_name":"test-metric_name","service":"test-service","hostname":"test-hostname","unit":"1","type":"g"}}

Explanation: In this example, the DSP metrics event contains a valid attributes map. However, because unit, type, or default dimensions do not exist in body, the values from attributes are used in HEC metric JSON fields.

Example 5: The event has the same field, hostname, in both attributes and body

DSP metric event:

Event{nanos=null, kind="metric", host=null, 
      attributes={"default_unit":"1", "default_type":"g", 
                  "default_dimensions": {hostname:"test-hostname", 
                                      service:"test-service"}}, 
      source_type="test-sourcetype", id=null, source="test-source", 
      body=[{name:"test-metric_name", unit:"2", hostname:"overwrite-hostname", 
             "value":2.2}], 
      timestamp=1000000000001}

HEC metric JSON

{"event":"metric","time":"1000000000.001","source":"test-source","sourcetype":"test-sourcetype","index":"myindex123","fields":{"_value":2.2,"metric_name":"test-metric_name","service":"test-service","hostname":"overwrite-hostname","unit":"2","type":"g"}}

Explanation: In this example, hostname exists as a field in attributes and body and contains two different values. The value in body takes priority and is the value indexed in hostname.

Last modified on 28 September, 2020
PREVIOUS
Formatting event data in DSP
  NEXT
Performance expectations for sending data from a data pipeline to Splunk Enterprise

This documentation applies to the following versions of Splunk® Data Stream Processor: 1.1.0


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters