Splunk® Data Stream Processor

Connect to Data Sources and Destinations with DSP

On April 3, 2023, Splunk Data Stream Processor reached its end of sale, and will reach its end of life on February 28, 2025. If you are an existing DSP customer, please reach out to your account team for more information.

All DSP releases prior to DSP 1.4.0 use Gravity, a Kubernetes orchestrator, which has been announced end-of-life. We have replaced Gravity with an alternative component in DSP 1.4.0. Therefore, we will no longer provide support for versions of DSP prior to DSP 1.4.0 after July 1, 2023. We advise all of our customers to upgrade to DSP 1.4.0 in order to continue to receive full product support from Splunk.

Create and manage DSP HEC tokens through the Ingest Service

DSP HEC tokens are entities that let logging agents and HTTP clients connect to the . When the clients connect, they present this token. If DSP HEC receives a valid token, it accepts the connection, and the client delivers its payload of application events in JSON format.

Splunk HEC tokens do not work with DSP HEC. You must create a new token for DSP HEC.

You can create DSP HEC tokens using either the Ingest Service or the Splunk Cloud Services CLI. The following examples demonstrate how to create, retrieve, update, and delete DSP HEC tokens using the Ingest Service. See Create and manage DSP HEC tokens through the Splunk Cloud Services CLI for information about using the CLI.

Prerequisites

Before you can create and manage DSP HEC tokens using the Ingest Service, you must have your access token for authenticating to DSP. This access token is distinct from DSP HEC tokens.

To retrieve your DSP access token, do the following:

  1. If you haven't done so already, set up the Splunk Cloud Services CLI to work with DSP. For detailed in structions, see Configure the Splunk Cloud Services CLI in the Install and administer the Data Stream Processor manual.
  2. From the DSP installer directory, log in to Splunk Cloud Services CLI with the --verbose flag. 
    ./scloud login --verbose
  3. When prompted, enter your DSP password.
  4. Copy the access_token value that is returned.

You'll need to use this copied value in your requests to the Ingest Service.

Create a DSP HEC token with the Ingest Service

The following example demonstrates how to use the Ingest Service to create a DSP HEC token. The placeholders in the example are defined as follows:

  • <DSP_HOST> is the IP address of the controller node.
  • <access-token> is your DSP access token.
  • <dsp-hec-token-name> is the name that you want to assign to the DSP HEC token. This name must be 4-36 characters in length, and cannot contain uppercase letters or special characters.

The "indexes" line is an optional setting that specifies the indexes that the token is permitted to send data to. You can omit this line if you don't want to restrict your token to a specific set of indexes. 

curl -X POST "https://<DSP_HOST>/default/ingest/v1beta2/collector/tokens" \
    -H "Authorization: Bearer <access-token>" \
    -H "Content-Type: application/json" \
    -d '{
          "name": "<dsp-hec-token-name>",
          "description": "An example token",
          "indexes": ["index1", "index2"],
          "disabled": false
         }'

The generated DSP HEC token is only returned once. You can't retrieve the token at a later date. Make sure to keep track of the returned token value.

A typical response when you create a DSP HEC token using a POST request looks like this: 

{
  "tenant": "default",
  "name": "<dsp-hec-token-name>",
  "token": "<dsp-hec-token>",
  "description": "An example token",
  "source":"",
  "sourcetype":"",
  "index": "",
  "indexes":["index1","index2"],
  "created_by": "name@example.com",
  "created_at": "2020-01-09T06:46:50.129Z",
  "last_modified_by": "name@example.com",
  "last_modified_at": "2020-01-09T06:46:50.129Z",
  "disabled": false,
  "allow_query_string_auth":false,
  "ack_enabled":false
}

You can also create tokens that specify default values for source, sourcetype, and index. See Use the metadata of the DSP HEC token to specify default values for more information.

Verify that the DSP HEC token is working in your pipeline

To verify that the DSP HEC token is working, start a preview session in DSP and send a test event to your pipeline using the token.

  1. In DSP, create a pipeline that starts with the Splunk DSP Firehose source function.
  2. Click the Start Preview icon (Start Preview button).
  3. Use your HTTP client to send the following HTTP request: 
    curl -X POST "https://<DSP_HOST>/services/collector/event" \
   -H "Authorization: Splunk <dsp-hec-token>" \
   -d '{
   "event": "Hello, world!",
   "sourcetype": "manual"
}'

You can see the test event in the Preview Results pane.

Update a DSP HEC token by name with the Ingest Service

The following example demonstrates how to use the Ingest Service to update the description of a DSP HEC token. The placeholders in the example are defined as follows:

  • <DSP_HOST> is the IP address of the controller node.
  • <name> is the name of the DSP HEC token that you want to update.
  • <access-token> is your DSP access token.
curl -X PUT "https://<DSP_HOST>/default/ingest/v1beta2/collector/tokens/<name>" \
    -H "Authorization: Bearer <access-token>" \
    -H "Content-Type: application/json" \
    -d '{
          "description": "A token for the clients in us-west datacenter 1"
         }'

A typical response when you update the description for a DSP HEC token using a PUT request looks like this: 

{
    "tenant": "default",
    "name": "<dsp-hec-token-name>",
    "description": "A token for the clients in us-west datacenter 1",
    "source":"",
    "sourcetype":"",
    "index":"",
    "indexes": ["index1", "index2"],
    "created_by": "name@example.com",
    "created_at": "2020-03-27T23:45:54.072714Z",
    "last_modified_by": "name@example.com",
    "last_modified_at": "2020-03-28T18:42:03.921962Z",
    "disabled": false,
    "allow_query_string_auth":false,
    "ack_enabled":false
}

You can also update the default values for source, sourcetype, and index in the token. See Use the metadata of the DSP HEC token to specify default values for more information.

Disable a DSP HEC token by name with the Ingest Service

The following example demonstrates how to use the Ingest Service to disable a DSP HEC token. The placeholders in the example are defined as follows:

  • <DSP_HOST> is the IP address of the controller node.
  • <name> is the name of the DSP HEC token that you want to disable.
  • <access-token> is your DSP access token.
curl -X PUT "https://<DSP_HOST>/default/ingest/v1beta2/collector/tokens/<name>" \
    -H "Authorization: Bearer <access-token>" \
    -H "Content-Type: application/json" \
    -d '{
          "disabled": true
         }'

A typical response when you disable a DSP HEC token using a PUT request looks like this: 

{
    "tenant": "defaults",
    "name": "<dsp-hec-token-name>",
    "description": "An example token",
    "source":"",
    "sourcetype":"",
    "index": "",
    "indexes": ["index1", "index2"],
    "created_by": "name@example.com",
    "created_at": "2020-03-27T23:45:54.072714Z",
    "last_modified_by": "name@example.com",
    "last_modified_at": "2020-03-28T18:42:03.921962Z",
    "disabled": true,
    "allow_query_string_auth":false,
    "ack_enabled":false
}

Enable a DSP HEC token by name with the Ingest Service

The following example demonstrates how to use the Ingest Service to enable a DSP HEC token that was previously disabled. The placeholders in the example are defined as follows:

  • <DSP_HOST> is the IP address of the controller node.
  • <name> is the name of the DSP HEC token that you want to enable.
  • <access-token> is your DSP access token.
curl -X PUT "https://<DSP_HOST>/default/ingest/v1beta2/collector/tokens/<name>" \
    -H "Authorization: Bearer <access-token>" \
    -H "Content-Type: application/json" \
    -d '{
          "disabled": false
         }'

A typical response when you disable a DSP HEC token using a PUT request looks like this: 

{
    "tenant": "default",
    "name": "<dsp-hec-token-name>",
    "description": "An example token",
    "source":"",
    "sourcetype":"",
    "index": "",
    "indexes": ["index1", "index2"],
    "created_by": "name@example.com",
    "created_at": "2020-03-27T23:45:54.072714Z",
    "last_modified_by": "name@example.com",
    "last_modified_at": "2020-03-28T18:42:03.921962Z",
    "disabled": false,
    "allow_query_string_auth":false,
    "ack_enabled":false
}

Get the metadata for a DSP HEC token by name with the Ingest Service

The following example demonstrates how to use the Ingest Service to get the metadata from a DSP HEC token. The placeholders in the example are defined as follows:

  • <DSP_HOST> is the IP address of the controller node.
  • <name> is the name of the DSP HEC token that you want to return metadata for.
  • <access-token> is your DSP access token.
curl -X GET "https://<DSP_HOST>/default/ingest/v1beta2/collector/tokens/<name>" \
    -H "Authorization: Bearer <access-token>" \
    -H "Content-Type: application/json"

A typical response when you request the metadata for a DSP HEC token using a GET request looks like this: 

{
    "tenant": "default",
    "name": "<dsp-hec-token-name>",
    "description": "An example token",
    "source":"",
    "sourcetype":"",
    "index": "",
    "indexes": ["index1", "index2"],
    "created_by": "name@example.com",
    "created_at": "2020-03-27T23:45:54.072714Z",
    "last_modified_by": "name@example.com",
    "last_modified_at": "2020-03-27T23:45:54.072714Z",
    "disabled": false,
    "allow_query_string_auth":false,
    "ack_enabled":false
}

List all DSP HEC tokens with the Ingest Service

The following example demonstrates how to use the Ingest Service to list all DSP HEC tokens. The placeholders in the example are defined as follows:

  • <DSP_HOST> is the IP address of the controller node.
  • <access-token> is your DSP access token.
curl -X GET "https://<DSP_HOST>/default/ingest/v1beta2/collector/tokens" \
    -H "Authorization: Bearer <access-token>" \
    -H "Content-Type: application/json"

Optionally, you can limit the number of returned tokens by including the limit and offset options. Using the following example, set <limit> to the number of DSP HEC tokens to return, and set <offset> to the number of tokens that you want to skip ahead. For instance, setting <limit> to 2 and <offset> to 3 returns the 4th and 5th tokens in your tenant. 

curl -X GET "https://<DSP_HOST>/default/ingest/v1beta2/collector/tokens?limit=<limit>&offset=<offset>" \
    -H "Authorization: Bearer <access-token>" \
    -H "Content-Type: application/json"

A typical response when you list all DSP HEC tokens using a GET request looks like this: 

{
  "tokens": [
    {
      "tenant": "default",
      "name": "<dsp-hec-token-name-01>",
      "description": "An example token",
      "source":"",
      "sourcetype":"",
      "index": "",
      "indexes": ["index1", "index2"],
      "createdAt": "2020-01-09T06:46:50.129Z",
      "createUserID": "user@example.com",
      "last_modified_by": "name@example.com",
      "last_modified_at": "2020-03-28T18:42:03.921962Z",
      "disabled": false,
      "allow_query_string_auth":false,
      "ack_enabled":false
    },
    {
      "tenant": "default",
      "name": "<dsp-hec-token-name-02>",
      "description": "A second example token",
      "index": "index1",
      "indexes": ["index1", "index2"],
      "createdAt": "2020-01-09T06:49:23.104Z",
      "createUserID": "user@example.com",
      "last_modified_by": "name@example.com",
      "last_modified_at": "2020-03-28T18:42:03.921962Z",
      "disabled": false,
      "allow_query_string_auth":false,
      "ack_enabled":false
      "disabled": false
    }
  ]
}

Delete a DSP HEC token by name with the Ingest Service

The following example demonstrates how to use the Ingest Service to delete a specific DSP HEC token. The placeholders in the example are defined as follows:

  • <DSP_HOST> is the IP address of the controller node.
  • <name> is the name of the DSP HEC token that you want to delete.
  • <access-token> is your DSP access token.
curl -X DELETE "https://<DSP_HOST>/default/ingest/v1beta2/collector/tokens/<name>" \
    -H "Authorization: Bearer <access-token>" \
    -H "Content-Type: application/json"

Delete all DSP HEC tokens with the Ingest Service

The following example demonstrates how to use the Ingest Service to delete all of the DSP HEC tokens in your DSP tenant. The placeholders in the example are defined as follows:

  • <DSP_HOST> is the IP address of the controller node.
  • <access-token> is your DSP access token.
curl -X DELETE "https://<DSP_HOST>/default/ingest/v1beta2/collector/tokens" \
    -H "Authorization: Bearer <access-token>" \
    -H "Content-Type: application/json"
Last modified on 13 January, 2023
Get data from HTTP clients into using the Splunk HEC API   Create and manage HEC tokens through the Splunk Cloud Services CLI

This documentation applies to the following versions of Splunk® Data Stream Processor: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.4.5, 1.4.6

Acrobat logo Download manual
Acrobat logo Download this page

Was this topic useful?







You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters