On October 30, 2022, all 1.2.x versions of the Splunk Data Stream Processor will reach its end of support date. See the Splunk Software Support Policy for details.
Get data from HTTP clients into DSP using the Splunk HEC API
You can update an existing Splunk HTTP Event Collector (HEC) workflow to use DSP HEC so that your HTTP clients send data to DSP data pipelines instead of Splunk indexes. See HTTP Event Collector and the for more information about the differences between Splunk HEC and DSP HEC.
To redirect data from your Splunk HEC workflow to the , start by creating a DSP HEC token and configuring your HTTP client to use it. Then, use the Splunk DSP Firehose source function in your pipeline to receive data from your HTTP client. Any data that your HTTP client sends to the Splunk HEC API will go to your DSP pipeline instead of the Splunk platform.
Prerequisites
Before you can update a Splunk HEC workflow to use DSP HEC, you must have the following:
- An existing Splunk HEC configuration and workflow. See Set up and use HTTP Event Collector in Splunk Web in the Splunk Enterprise Getting Data In manual for more information about setting up and using Splunk HEC.
- A DSP HEC token. See Create DSP HEC tokens through the Ingest Service or Create DSP HEC tokens through the Splunk Cloud Services CLI. Make sure to copy the value from the
token
field in the response after the token is created.
Steps
- Update the HTTP client used in your current Splunk HEC workflow so that it starts sending data to DSP:
- Set the base URL to
https://<DSP_HOST>:31000
, where<DSP_HOST>
is the IP address of the master node. - Set the token to
Authorization: Splunk <dsp-hec-token>
, where<dsp-hec-token>
is thetoken
value that was returned when you created your DSP HEC token.
- Set the base URL to
- Restart your Splunk HEC workflow.
- In DSP, add the Splunk DSP Firehose source function to the start of your pipeline.
When you activate the pipeline, the source function starts receiving data from the HTTP client. For information about sending your data to the Splunk platform after processing it in DSP, see Connecting your DSP pipeline to the Splunk platform.
See the following sections for examples of how events and metrics can be sent to DSP through Splunk HEC API endpoints. See Set the default field values in DSP HEC for information on how you can specify default values for certain fields in the outgoing event or metric.
Example: Use the Splunk HEC API to send events to a DSP pipeline
The following example demonstrates how to use the Splunk HEC API /services/collector
endpoint to send events to a DSP pipeline. In this example, <DSP_HOST>
and <dsp-hec-token>
are placeholders for the IP address of your master node and your DSP HEC token, respectively.
curl -X POST "https://<DSP_HOST>:31000/services/collector" \ -H "Authorization: Splunk <dsp-hec-token>" \ -d '{ "event": "event1", "time": 1505333071.247, "index": "index1", "source": "source1", "sourcetype": "sourcetype1", "host": "example.host.com", "fields": { "region": "northeast", "datacenter": "dc", "rack": "62", "os": "Ubuntu18.04", "arch": "x64" }, }'
Example: Use the Splunk HEC API to send metrics to a DSP pipeline
See Get started with metrics in the Splunk Enterprise Metrics manual for more information about using metrics with the Splunk platform.
The following example demonstrates how to use the Splunk HEC API /services/collector
endpoint to send metrics to a DSP pipeline. In this example, <DSP_HOST>
and <dsp-hec-token>
are placeholders for the IP address of your master node and your DSP HEC token, respectively.
curl -X POST "https://<DSP_HOST>:31000/services/collector" \ -H "Authorization: Splunk <dsp-hec-token>" \ -d '{ "time": 1505333071.247, "index": "index1", "source": "source1", "sourcetype": "sourcetype1", "host": "example.host.com", "fields": { "region": "northeast", "datacenter": "dc", "rack": "62", "os": "Ubuntu18.04", "arch": "x64", "metric_name": "cpu.user", "_value": 0.5 }, }
Example: Use the Splunk HEC API to send multiple metrics to a DSP pipeline
Version 8.0.0 and higher of the Splunk platform supports a JSON format which allows each JSON object to contain measurements for multiple metrics. You can use this multiple-metric JSON format with your DSP pipelines. See The multiple-metric JSON format in the Splunk Enterprise Metrics manual for more information.
The following example demonstrates how to use the Splunk HEC API /services/collector
endpoint to send multiple metrics to a DSP pipeline. In this example, <DSP_HOST>
and <dsp-hec-token>
are placeholders for the IP address of your master node and your DSP HEC token, respectively.
curl -X POST "https://<DSP_HOST>:31000/services/collector" \ -H "Authorization: Splunk <dsp-hec-token>" \ -d '{ "time": 1505333071.247, "index": "index1", "source": "source1", "sourcetype": "sourcetype1", "host": "example.host.com", "fields": { "region": "northeast, "datacenter": "dc", "rack": "62", "os": "Ubuntu18.04", "arch": "x64", "metric_name:cpu.idle": 28.7261266542177580, "metric_name:cpu.usr": 28.7261266542177580, "metric_name:cpu.sys": 28.7261266542177580 }, }
Example: Use basic authentication with the Splunk HEC API to send events to a DSP pipeline
You can use basic authentication, where the username is any alphanumeric string and the password is your DSP HEC token, to send events to a DSP pipeline through the Splunk HEC API.
The following example demonstrates how to use basic authentication with the Splunk HEC API /services/collector/event
endpoint to send events to a DSP pipeline. In this example, <username>
and <dsp-hec-token>
are placeholders for your username and your DSP HEC token, respectively. <DSP_HOST>
is a placeholder for the IP address of your master node.
curl -k -u "<username>:<dsp-hec-token>" "https://<DSP_HOST>:31000/services/collector/event" \ -d { "event": "event1", "time": 1505333071.247, "index": "index1", "source": "source1", "sourcetype": "sourcetype1", "host": "example.host.com", "fields": { "region": "northeast", "datacenter": "dc", "rack": "62", "os": "Ubuntu18.04", "arch": "x64" }, }'
HTTP Event Collector and the | Create and manage DSP HEC tokens through the Ingest Service |
This documentation applies to the following versions of Splunk® Data Stream Processor: 1.2.0, 1.2.1-patch02, 1.2.1, 1.2.2-patch02, 1.2.4, 1.2.5
Feedback submitted, thanks!