Formatting metrics data
For the Write to Splunk Enterprise and Write to Index sink functions to properly transform your event into metrics format, your event must follow the metrics schema.
If you are sending metrics data using the Ingest Service's /metrics
endpoint, then your metrics data is already properly formatted for Write to Splunk Enterprise and Write to Index functions.
Use the following table and examples as a guide for how to format your data so that it is indexed appropriately into Splunk Enterprise.
DSP field | HEC metrics JSON | Data Type | Notes |
---|---|---|---|
body | fields | list of maps | An array of one or more JSON objects following the schema defined here. There must be at least one metric in the |
sourcetype or source_type | sourcetype | string | If not present, no sourcetype is included in the HEC metrics JSON. |
timestamp | time | long integer | The Data Stream Processor uses Unix epoch time in milliseconds. Your timestamp is automatically converted to Splunk epoch time format <sec>.<ms> . If blank or negative, time is set to now.
|
source | source | string | If not present, no source is included in the HEC metrics JSON. |
host | host | string | If not present, no host is included in the HEC metrics JSON. |
attributes | fields | map<string, any> | Defines default dimensions used in the HEC metrics JSON. For information on the attributes schema, see the attributes row in the metrics schema. For examples on how these default dimensions are used, see example 4 and example 5. |
id | N/A | string | A DSP metrics field ignored by HEC. |
kind | N/A | string | A DSP metrics field ignored by HEC. |
nanos | N/A | integer | A DSP metrics field ignored by HEC. |
any custom fields | N/A | any | All custom fields are ignored when creating the HEC metrics JSON. |
N/A | index | string | To set the index in HEC metrics JSON, you must pass the index name as an argument in the Write Splunk Enterprise or Write Index functions. If no index is selected, your data is sent to the default index associated with your HEC token. |
Example 1: The event's body is null
DSP metric event:
Event{nanos=null, kind="metric", host=null, attributes=null, source_type="test-sourcetype", id=null, source="test-source", body=null, timestamp=1000000000000}
HEC metric JSON:
null (event is dropped)
Explanation: Your event body
must have at least one metric or else your event is dropped.
Example 2: The event has a body field present
DSP metric event:
Event{nanos=null,kind="metric",host=null, attributes=null, source_type="test-sourcetype", id=null, source="test-source", body=[{name:"test-metric_name", "value":2.2}],timestamp=1000000000001}
HEC metric JSON:
{"event":"metric","time":"1000000000.001","source":"test-source", "sourcetype":"test-sourcetype","index":"myindex123", "fields":{"_value":2.2,"metric_name":"test-metric_name"}}
Explanation: The body
field has the correct DSP metrics schema. DSP name
and value
maps to HEC metrics JSON metric_name
and _value
respectively.
Example 3: The event's body contains two maps
DSP metric event:
Event{nanos=null, kind="metric", host=null, attributes=null, source_type="test-sourcetype", id=null, source="test-source", body=[{name:"test-metric_name", "value":2.2}, {name:"test-metric_name-2", "value":4.4}], timestamp=1000000000001}
HEC metric JSON:
{"event":"metric","time":"1000000000.001","source":"test-source","sourcetype":"test-sourcetype","index":"myindex123","fields":{"_value":2.2,"metric_name":"test-metric_name"}} {"event":"metric","time":"1000000000.001","source":"test-source","sourcetype":"test-sourcetype","index":"myindex123","fields":{"_value":4.4,"metric_name":"test-metric_name-2"}}
Explanation: Because the body
field has two maps in the array, two separate HEC metric JSON objects are created.
Example 4: The event has additional attributes
DSP metric event:
Event{nanos=null, kind="metric", host=null, attributes={default_unit:"1", default_type:"g", default_dimensions={hostname:"test-hostname", service:"test-service"}}, source_type="test-sourcetype", id=null, source="test-source", body=[{name:"test-metric_name", "value":2.2}], timestamp=1000000000001}
HEC metric JSON
{"event":"metric","time":"1000000000.001","source":"test-source","sourcetype":"test-sourcetype","index":"myindex123","fields":{"_value":2.2,"metric_name":"test-metric_name","service":"test-service","hostname":"test-hostname","unit":"1","type":"g"}}
Explanation: In this example, the DSP metrics event contains a valid attributes map. However, because unit
, type
, or default dimensions
do not exist in body
, the values from attributes
are used in HEC metric JSON fields
.
Example 5: The event has the same field, hostname
, in both attributes and body
DSP metric event:
Event{nanos=null, kind="metric", host=null, attributes={"default_unit":"1", "default_type":"g", "default_dimensions": {hostname:"test-hostname", service:"test-service"}}, source_type="test-sourcetype", id=null, source="test-source", body=[{name:"test-metric_name", unit:"2", hostname:"overwrite-hostname", "value":2.2}], timestamp=1000000000001}
HEC metric JSON
{"event":"metric","time":"1000000000.001","source":"test-source","sourcetype":"test-sourcetype","index":"myindex123","fields":{"_value":2.2,"metric_name":"test-metric_name","service":"test-service","hostname":"overwrite-hostname","unit":"2","type":"g"}}
Explanation: In this example, hostname
exists as a field in attributes
and body
and contains two different values. The value in body
takes priority and is the value indexed in hostname
.
Formatting event data in DSP | Batch Events to optimize throughput to Splunk Enterprise indexes in DSP |
This documentation applies to the following versions of Splunk® Data Stream Processor: 1.0.0
Feedback submitted, thanks!