All DSP releases prior to DSP 1.4.0 use Gravity, a Kubernetes orchestrator, which has been announced end-of-life. We have replaced Gravity with an alternative component in DSP 1.4.0. Therefore, we will no longer provide support for versions of DSP prior to DSP 1.4.0 after July 1, 2023. We advise all of our customers to upgrade to DSP 1.4.0 in order to continue to receive full product support from Splunk.
About the Ingest service
You can use the Ingest service /events and /metrics endpoints to send events and metrics data to your data pipeline over HTTPS in JSON format.
The Ingest service /events endpoint
The /events endpoint accepts an array of JSON objects. Each JSON object represents a single event.
[ { "body": "Hello, World!", "attributes": { "message": "Something happened" }, "host": "dataserver.example.com", "source": "testapplication", "sourcetype": "txt", "timestamp": 1533671808138, "nanos": 0, "id": "2823738566644596", "kind": "event" }, { "body": "Hello, World2!", "attributes": { "message": "Something happened" }, "host": "dataserver.example.com", "source": "testapplication", "sourcetype": "txt", "timestamp": 1533671808138, "nanos": 0, "id": "2518594268716256", "kind": "event" },..... ]
Event schema
There are eight keys that can be included in the event schema. Including an event with a field not defined in this schema results in an "INVALID_DATA" error.
Key | Required? | Description |
---|---|---|
body | Required | The event's payload. It can be one of the basic types: a string, bytes, or a number (int32, int64, float, or double). In addition, body can be a list or a map of basic types. The default type of body is a union type of all possible types. To pass body as an argument to a downstream function that requires a specific type, you should first cast body to the appropriate more specific type. See cast or ucast for an example.
|
attributes | Optional | Specifies a JSON object that contains explicit custom fields. |
host | Optional | The host value to assign to the event data. This is typically the hostname of the client from which you are sending data. |
source | Optional | The source value to assign to the event data. For example, if you are sending data from an app you are developing, you can set this key to the name of the app. |
source_type | Optional | The sourcetype value to assign to the event data. |
timestamp | Optional | The event time in epoch time format in milliseconds. If this key is missing, a timestamp with the current time is assigned when the event reaches the Ingest service. |
nanos | Optional | Nanoseconds part of the timestamp. |
id | Optional | Unique ID of the event. If it is not specified, the system generates an ID. |
kind | Optional | The value event , to indicate that the record is an event.
|
The Ingest service /metrics endpoint
The /metrics endpoint accepts a list of JSON metric objects. Each JSON object represents a single metric event.
Payload schema = [<JsonMetricObject>, <JsonMetricObject>, ...] # a list of JsonMetricObject JsonMetricObject = { "body": [<Metric>, <Metric>, ...], "timestamp": int64, "nanos": int32, "source": string, "sourcetype": string, "host": string, "id": string, "kind": string, "attributes": { "defaultDimensions": map[string]string, "defaultType": string, "defaultUnit": string } }
Metrics schema
There are eight keys that can be included in the metrics schema. Including a metric with a field not defined in this schema results in an "INVALID_DATA" error.
Key | Required? | Description |
---|---|---|
body | Required | An array of one or more JSON objects following the defined schema. Each object represents a measurement of a given metric at the time denoted by the parent object's timestamp.
// Metric = { // "name": "cpu.util", // "value": 45.0, // "dimensions": {"Server":"nginx", "Region":"us-west-1"}, // "type": "g", // "unit": "percent"} Metric = map[string] object { "name": string, // required. metric name "value": numeric, // required. double | float | int | long } |
attributes | Optional | JSON objects that follow the defined schema. For example: {"Server":"nginx", "Region":"us-west-1", ...} . If set, individual metrics inherit these dimensions. If there is a dimension also given in the body field of the individual metric, the body field dimension takes precedent.
"attributes": { "defaultDimensions": map[string]string, // optional. String map. // For example: {"Server":"nginx", "Region":"us-west-1", ...}, "defaultType": string, // optional. metric type, by default it is "g" for "gauge" "defaultUnit": string // optional. metric unit, by default it is "none" } |
host | Optional | The host value to assign to the event data. This is typically the host name of the client from which you are sending data. |
source | Optional | The source value to assign to the event data. For example, if you are sending data from an app you are developing, you can set this key to the name of the app. |
source_type | Optional | The source type value to assign to the event data. |
timestamp | Optional | The event time in epoch time format in milliseconds. If this key is missing, a timestamp with the current time is assigned when the event reaches the Ingest service. |
nanos | Optional | Nanoseconds part of the timestamp. |
id | Optional | Unique ID of the event. If it is not specified, the system generates an ID. |
kind | Optional | The value metric , to indicate that the record is a metric event.
|
Metrics examples using SCloud command
To send data using the ./scloud ingest post-metrics
command, format your data in a streaming JSON format where each line is an array of metrics. SCloud sends the payload of metrics event to the body
field so you don't need to include body
in the command.
The ./scloud ingest post-metrics
command works with SCloud 4.0.0 but not with SCloud 1.0.0.
The following example shows an SCloud command that sends an array of CPU and memory utilization metrics records to the .
echo '[{"name": "cpu.util", "value": 45.0, "unit": "percent", "sourcetype": "aws:cloudwatch", "dimensions": {"InstanceId": "i-065d598370ac25b90", "Region": "us-west-1"}, "type": "g"}, {"name": "mem.util", "value": 20, "unit": "gigabytes", "dimensions": {"InstanceId": "i-065d598370ac25b90", "Region": "us-west-1"}, "type": "g"}, {"name": "net.in", "value": 3000, "unit": "bytes/second", "sourcetype": "aws:cloudwatch", "dimensions": {"InstanceId": "i-065d598370ac25b90", "Region": "us-west-1"}, "type": "g"}]' | ./scloud ingest post-metrics --sourcetype "aws:cloudwatch" --timestamp 1526627123013 --default-type "g"
In this example, we use flags to assign values to some fields such as sourcetype
and timestamp
instead of ingesting them through the body of the records. When included as part of the body, these fields get dropped from the record and don't show up in the preview. For the full list of flags you can set, use the ./scloud ingest post-metrics --help
command.
Interpreting pipeline statuses | Use the Ingest service to send test events to your pipeline |
This documentation applies to the following versions of Splunk® Data Stream Processor: 1.2.0, 1.2.1-patch02, 1.2.1, 1.2.2-patch02, 1.2.4, 1.2.5, 1.3.0, 1.3.1, 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.4.5
Feedback submitted, thanks!