Splunk® Data Stream Processor

Connect to Data Sources and Destinations with DSP

Acrobat logo Download manual as PDF


On October 30, 2022, all 1.2.x versions of the Splunk Data Stream Processor will reach its end of support date. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see the Upgrade the Splunk Data Stream Processor topic.
This documentation does not apply to the most recent version of Splunk® Data Stream Processor. For documentation on the most recent version, go to the latest release.
Acrobat logo Download topic as PDF

HTTP Event Collector and the

If you already use Splunk HTTP Event Collector (HEC) to ingest data in the Splunk platform, you can update your data ingestion workflow to use DSP HEC and send your data to DSP instead. Send the events and metrics from your HTTP client to DSP so that you can process your data before sending it to the Splunk platform for indexing.

DSP HEC doesn't support the same tokens as Splunk HEC, so you'll need to create a new token and configure your HTTP client to use it. You can then use the Splunk DSP Firehose source function in a data pipeline to receive data from the HTTP client. DSP HEC uses the DSP API Gateway port to connect to the Splunk Data Stream Processor. See Get data from HTTP clients into DSP using the Splunk HEC API for instructions on how to redirect your Splunk HEC workflow to DSP.

DSP HEC supports the following Splunk HEC endpoints:

  • /services/collector
  • /services/collector/event
  • /services/collector/event/1.0

You can send metrics and events to any of these endpoints. See HTTP Event Collector REST API endpoints and Format events for HTTP Event Collector in the Splunk Enterprise Getting Data In manual to learn more about the differences between these endpoints and when to use them. The maximum supported metric or event payload size for all /services/collector endpoints is 5 MB. If you send a metric or event payload that is larger than 5 MB, you will receive an HTTP 413 Request entity exceeded size limit error message in the response body.

Differences between Splunk HEC and DSP HEC

Splunk HEC DSP HEC
Allows events and metrics to be written directly to the Splunk platform. Allows events and metrics to be written to DSP. See Connecting your DSP pipeline to a Splunk index if the final destination for the ingested data is the Splunk platform.
Splunk Indexer error codes can be returned directly to the HTTP client. Splunk Indexer error codes return an Invalid Data Format error in DSP HEC.
Each HEC token is associated with a set of authorized indexes. An error is returned if an event refers to another index. DSP HEC can't directly control which index an event is written to. You can set default values for the index fields in the DSP HEC tokens, but you'll also need to configure your pipeline to route the data according to the index value. See Connecting your DSP pipeline to a Splunk index for more information on configuring index routing.
A typical Splunk HEC token looks like this: ef976ef0-dc7b-46b9-aa2e-c407cad884e2 The DSP HEC token format is dsphec:sha256:UUID. A typical DSP HEC token looks like this:

dsphec:e9da86d351cf9a7642d8c50195c3f466220911a15c177809bd1161a51e8c5f24:14c813f1-33ab-426b-8350-1b3e7f1e83f8

Asynchronous event acknowledgment is supported via the /services/collector/ack API endpoint. DSP HEC does not support the asynchronous ACK protocol or the /services/collector/ack endpoint. If an HTTP 200 response is received from DSP HEC, the events in the request have been delivered to the Splunk DSP Firehose and are available for processing in your pipeline. No ACK is necessary.
Raw events are supported via the /services/collector/raw API endpoint. Raw events are not supported.
MINT formatted data is supported via the /services/collector/mint API endpoint. MINT formatted data is not supported.
Uses port 8088 to connect to Splunk Enterprise. Uses port 31000 to connect to the Splunk Data Stream Processor API services.
The maximum payload size (max_content_length) defaults to 2 GB. The maximum supported event or metric payload size for all /services/collector endpoints is 5 MB.

Learn more

To learn more about Splunk HEC, see the following pages from the Splunk Enterprise Getting Data In manual:

Last modified on 13 January, 2023
PREVIOUS
Formatting data into the Splunk Infrastructure Monitoring metrics schema
  NEXT
Get data from HTTP clients into DSP using the Splunk HEC API

This documentation applies to the following versions of Splunk® Data Stream Processor: 1.2.0, 1.2.1-patch02, 1.2.1, 1.2.2-patch02, 1.2.4, 1.2.5, 1.3.0, 1.3.1


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters