Splunk® Data Stream Processor

Connect to Data Sources and Destinations with DSP

Acrobat logo Download manual as PDF


Acrobat logo Download topic as PDF

Get data from HTTP clients into DSP using the Splunk HEC API

You can update an existing Splunk HTTP Event Collector (HEC) workflow to use DSP HEC so that your HTTP clients send data to DSP data pipelines instead of Splunk indexes. See HTTP Event Collector and the for more information about the differences between Splunk HEC and DSP HEC.

To redirect data from your Splunk HEC workflow to the , start by creating a DSP HEC token and configuring your HTTP client to use it. Then, use the Splunk DSP Firehose source function in your pipeline to receive data from your HTTP client. Any data that your HTTP client sends to the Splunk HEC API will go to your DSP pipeline instead of the Splunk platform.

Prerequisites

Before you can update a Splunk HEC workflow to use DSP HEC, you must have the following:

Steps

  1. Update the HTTP client used in your current Splunk HEC workflow so that it starts sending data to DSP:
    1. Set the base URL to https://<DSP_HOST>:31000, where <DSP_HOST> is the IP address of the master node.
    2. Set the token to Authorization: Splunk <dsp-hec-token>, where <dsp-hec-token> is the token value that was returned when you created your DSP HEC token.
  2. Restart your Splunk HEC workflow.
  3. In DSP, add the Splunk DSP Firehose source function to the start of your pipeline.

When you activate the pipeline, the source function starts receiving data from the HTTP client. For information about sending your data to the Splunk platform after processing it in DSP, see Connecting your DSP pipeline to the Splunk platform.

See the following sections for examples of how events and metrics can be sent to DSP through Splunk HEC API endpoints. See Set the default field values in DSP HEC for information on how you can specify default values for certain fields in the outgoing event or metric.

Example: Use the Splunk HEC API to send events to a DSP pipeline

The following example demonstrates how to use the Splunk HEC API /services/collector endpoint to send events to a DSP pipeline. In this example, <DSP_HOST> and <dsp-hec-token> are placeholders for the IP address of your master node and your DSP HEC token, respectively.

curl -X POST "https://<DSP_HOST>:31000/services/collector" \
   -H "Authorization: Splunk <dsp-hec-token>" \
   -d '{ 
         "event": "event1", 
         "time": 1505333071.247, 
         "index": "index1", 
         "source": "source1", 
         "sourcetype": "sourcetype1", 
         "host": "example.host.com", 
         "fields": {
                  "region": "northeast",
                  "datacenter": "dc",
                  "rack": "62",
                  "os": "Ubuntu18.04",
                  "arch": "x64"
                  }, 
         }'

Example: Use the Splunk HEC API to send metrics to a DSP pipeline

See Get started with metrics in the Splunk Enterprise Metrics manual for more information about using metrics with the Splunk platform.

The following example demonstrates how to use the Splunk HEC API /services/collector endpoint to send metrics to a DSP pipeline. In this example, <DSP_HOST> and <dsp-hec-token> are placeholders for the IP address of your master node and your DSP HEC token, respectively.

curl -X POST "https://<DSP_HOST>:31000/services/collector" \
   -H "Authorization: Splunk <dsp-hec-token>" \
   -d '{
         "time": 1505333071.247, 
         "index": "index1",
         "source": "source1", 
         "sourcetype": "sourcetype1", 
         "host": "example.host.com", 
         "fields": {
                  "region": "northeast",
                  "datacenter": "dc",
                  "rack": "62",
                  "os": "Ubuntu18.04",
                  "arch": "x64",        
                  "metric_name": "cpu.user",
                  "_value": 0.5
                  },
         }

Example: Use the Splunk HEC API to send multiple metrics to a DSP pipeline

Version 8.0.0 and higher of the Splunk platform supports a JSON format which allows each JSON object to contain measurements for multiple metrics. You can use this multiple-metric JSON format with your DSP pipelines. See The multiple-metric JSON format in the Splunk Enterprise Metrics manual for more information.

The following example demonstrates how to use the Splunk HEC API /services/collector endpoint to send multiple metrics to a DSP pipeline. In this example, <DSP_HOST> and <dsp-hec-token> are placeholders for the IP address of your master node and your DSP HEC token, respectively.

curl -X POST "https://<DSP_HOST>:31000/services/collector" \
   -H "Authorization: Splunk <dsp-hec-token>" \
   -d '{
         "time": 1505333071.247, 
         "index": "index1",
         "source": "source1", 
         "sourcetype": "sourcetype1", 
         "host": "example.host.com", 
         "fields": {
                  "region": "northeast,
                  "datacenter": "dc",
                  "rack": "62",
                  "os": "Ubuntu18.04",
                  "arch": "x64",
                  "metric_name:cpu.idle": 28.7261266542177580,
                  "metric_name:cpu.usr": 28.7261266542177580,
                  "metric_name:cpu.sys": 28.7261266542177580
                  },
         }

Example: Use basic authentication with the Splunk HEC API to send events to a DSP pipeline

You can use basic authentication, where the username is any alphanumeric string and the password is your DSP HEC token, to send events to a DSP pipeline through the Splunk HEC API.

The following example demonstrates how to use basic authentication with the Splunk HEC API /services/collector/event endpoint to send events to a DSP pipeline. In this example, <username> and <dsp-hec-token> are placeholders for your username and your DSP HEC token, respectively. <DSP_HOST> is a placeholder for the IP address of your master node.

curl -k -u  "<username>:<dsp-hec-token>" "https://<DSP_HOST>:31000/services/collector/event" \
    -d { 
        "event": "event1", 
         "time": 1505333071.247, 
         "index": "index1", 
         "source": "source1", 
         "sourcetype": "sourcetype1", 
         "host": "example.host.com", 
         "fields": {
                  "region": "northeast",
                  "datacenter": "dc",
                  "rack": "62",
                  "os": "Ubuntu18.04",
                  "arch": "x64"
                  }, 
         }'
Last modified on 22 June, 2021
PREVIOUS
HTTP Event Collector and the
  NEXT
Create and manage DSP HEC tokens through the Ingest Service

This documentation applies to the following versions of Splunk® Data Stream Processor: 1.2.0, 1.2.1


Was this documentation topic helpful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters