Get data from HTTP clients into DSP using the Splunk HEC API

You can update an existing Splunk HTTP Event Collector (HEC) workflow to use DSP HEC so that your HTTP clients send data to DSP data pipelines instead of Splunk indexes. See HTTP Event Collector and the for more information about the differences between Splunk HEC and DSP HEC.

To redirect data from your Splunk HEC workflow to the , start by creating a DSP HEC token and configuring your HTTP client to use it. Then, use the Splunk DSP Firehose source function in your pipeline to receive data from your HTTP client. Any data that your HTTP client sends to the Splunk HEC API will go to your DSP pipeline instead of the Splunk platform.

Prerequisites

Before you can update a Splunk HEC workflow to use DSP HEC, you must have the following:

An existing Splunk HEC configuration and workflow. See Set up and use HTTP Event Collector in Splunk Web in the Splunk Enterprise Getting Data In manual for more information about setting up and using Splunk HEC.
A DSP HEC token. See Create DSP HEC tokens through the Ingest Service or Create DSP HEC tokens through the Splunk Cloud Services CLI. Make sure to copy the value from the token field in the response after the token is created.

Steps

Update the HTTP client used in your current Splunk HEC workflow so that it starts sending data to DSP:
1. Set the base URL to https://<DSP_HOST>, where <DSP_HOST> is the IP address of the controller node.
2. Set the token to Authorization: Splunk <dsp-hec-token>, where <dsp-hec-token> is the token value that was returned when you created your DSP HEC token.
If you are sending GZIP or ZSTD compressed data to your DSP pipeline, set the content encoding to "Content-Encoding: gzip" for GZIP or "Content-Encoding: zstd" for ZSTD.
Restart your Splunk HEC workflow.
In DSP, add the Splunk DSP Firehose source function to the start of your pipeline.

When you activate the pipeline, the source function starts receiving data from the HTTP client. For information about sending your data to the Splunk platform after processing it in DSP, see Connecting your DSP pipeline to the Splunk platform.

See the following sections for examples of how events and metrics can be sent to DSP through Splunk HEC API endpoints. See Set the default field values in DSP HEC for information on how you can specify default values for certain fields in the outgoing event or metric.

Example: Use the Splunk HEC API to send events to a DSP pipeline

The following example demonstrates how to use the Splunk HEC API /services/collector endpoint to send events to a DSP pipeline. In this example, <DSP_HOST> and <dsp-hec-token> are placeholders for the IP address of your controller node and your DSP HEC token, respectively.

curl -X POST "https://<DSP_HOST>/services/collector" \
   -H "Authorization: Splunk <dsp-hec-token>" \
   -d '{ 
         "event": "event1", 
         "time": 1505333071.247, 
         "index": "index1", 
         "source": "source1", 
         "sourcetype": "sourcetype1", 
         "host": "example.host.com", 
         "fields": {
                  "region": "northeast",
                  "datacenter": "dc",
                  "rack": "62",
                  "os": "Ubuntu18.04",
                  "arch": "x64"
                  }, 
         }'

Example: Use the Splunk HEC API to send metrics to a DSP pipeline

See Get started with metrics in the Splunk Enterprise Metrics manual for more information about using metrics with the Splunk platform.

The following example demonstrates how to use the Splunk HEC API /services/collector endpoint to send metrics to a DSP pipeline. In this example, <DSP_HOST> and <dsp-hec-token> are placeholders for the IP address of your controller node and your DSP HEC token, respectively.

curl -X POST "https://<DSP_HOST>/services/collector" \
   -H "Authorization: Splunk <dsp-hec-token>" \
   -d '{
         "time": 1505333071.247, 
         "index": "index1",
         "source": "source1", 
         "sourcetype": "sourcetype1", 
         "host": "example.host.com", 
         "fields": {
                  "region": "northeast",
                  "datacenter": "dc",
                  "rack": "62",
                  "os": "Ubuntu18.04",
                  "arch": "x64",        
                  "metric_name": "cpu.user",
                  "_value": 0.5
                  },
         }

Example: Use the Splunk HEC API to send multiple metrics to a DSP pipeline

Version 8.0.0 and higher of the Splunk platform supports a JSON format which allows each JSON object to contain measurements for multiple metrics. You can use this multiple-metric JSON format with your DSP pipelines. See The multiple-metric JSON format in the Splunk Enterprise Metrics manual for more information.

The following example demonstrates how to use the Splunk HEC API /services/collector endpoint to send multiple metrics to a DSP pipeline. In this example, <DSP_HOST> and <dsp-hec-token> are placeholders for the IP address of your controller node and your DSP HEC token, respectively.

curl -X POST "https://<DSP_HOST>/services/collector" \
   -H "Authorization: Splunk <dsp-hec-token>" \
   -d '{
         "time": 1505333071.247, 
         "index": "index1",
         "source": "source1", 
         "sourcetype": "sourcetype1", 
         "host": "example.host.com", 
         "fields": {
                  "region": "northeast,
                  "datacenter": "dc",
                  "rack": "62",
                  "os": "Ubuntu18.04",
                  "arch": "x64",
                  "metric_name:cpu.idle": 28.7261266542177580,
                  "metric_name:cpu.usr": 28.7261266542177580,
                  "metric_name:cpu.sys": 28.7261266542177580
                  },
         }

Example: Use basic authentication with the Splunk HEC API to send events to a DSP pipeline

You can use basic authentication, where the username is any alphanumeric string and the password is your DSP HEC token, to send events to a DSP pipeline through the Splunk HEC API.

The following example demonstrates how to use basic authentication with the Splunk HEC API /services/collector/event endpoint to send events to a DSP pipeline. In this example, <username> and <dsp-hec-token> are placeholders for your username and your DSP HEC token, respectively. <DSP_HOST> is a placeholder for the IP address of your controller node.

curl -k -u  "<username>:<dsp-hec-token>" "https://<DSP_HOST>/services/collector/event" \
    -d { 
        "event": "event1", 
         "time": 1505333071.247, 
         "index": "index1", 
         "source": "source1", 
         "sourcetype": "sourcetype1", 
         "host": "example.host.com", 
         "fields": {
                  "region": "northeast",
                  "datacenter": "dc",
                  "rack": "62",
                  "os": "Ubuntu18.04",
                  "arch": "x64"
                  }, 
         }'

Example: Use the Splunk HEC API to send compressed data to a DSP pipeline

The following example demonstrates how to use the Splunk HEC API /services/collector endpoint to send data compressed with GZIP to a DSP pipeline. This example assumes that your events data is already compressed with GZIP and the file name is events.gz.

curl -X POST "https://<DSP_HOST>/services/collector" \
   -H "Authorization: Splunk <dsphec-token>" \
   -H "Content-Encoding: gzip" \
   -d @events.gz

The following example demonstrates how to use the Splunk HEC API /services/collector endpoint to send data compressed with ZSTD to a DSP pipeline. This example assumes that your events data is already compressed with ZSTD and the file name is events.zstd.

curl -X POST "https://<DSP_HOST>/services/collector" \
   -H "Authorization: Splunk <dsphec-token>" \
   -H "Content-Encoding: zstd" \
   -d @events.zstd

Related answers from Splunk Community