Splunk® Data Stream Processor

Function Reference

On April 3, 2023, Splunk Data Stream Processor reached its end of sale, and will reach its end of life on February 28, 2025. If you are an existing DSP customer, please reach out to your account team for more information.

All DSP releases prior to DSP 1.4.0 use Gravity, a Kubernetes orchestrator, which has been announced end-of-life. We have replaced Gravity with an alternative component in DSP 1.4.0. Therefore, we will no longer provide support for versions of DSP prior to DSP 1.4.0 after July 1, 2023. We advise all of our customers to upgrade to DSP 1.4.0 in order to continue to receive full product support from Splunk.

Send data to a Splunk index

Use the Send to a Splunk Index sink function to send data to an external Splunk Enterprise system.

This function sends data to a Splunk Enterprise index using the Splunk HTTP Event Collector (HEC). For more information, see the Get data with HTTP Event Collector chapter in the Splunk Enterprise Getting Data In manual.

Prerequisites

Before you can use this function, you must do the following:

Function input schema

See Connecting Splunk indexes to your DSP pipeline.

Required arguments

connection_id
Syntax: string
Description: The ID of the Splunk Enterprise Connection.
Example in Canvas View: "576205b3-f6f5-4ab7-8ffc-a4089a95d0c4"
index
Syntax: expression<string>
Description: The index to send your data to.
Example in Canvas View: "main"
payload
Syntax: expression<bytes>
Description: The name of the DSP record field (for example, "bytes") that has the byte payload to be written directly to the HEC endpoint. Set to null if your records are not in bytes.
Example in Canvas View: bytes

Optional arguments

parameters
Syntax: map<string, string>
Description: The optional parameters you can enter in this function. See the following table for a description of each parameter.
Parameter Input Description Example
parameters map<string, string> The following rows list the optional parameters you can enter in this function. See the "Parameters" table for available options. Defaults to empty { }.
hec-token-validation boolean Set to true to enable HEC token validation. Defaults to true. "hec-token-validation": "true"
hec-enable-ack boolean Set to true for the function to wait for an acknowledgement for every single event. Set to false if acknowledgements in your Splunk platform are disabled or to increase throughput. Defaults to true. "hec-enable-ack": "true"
hec-gzip-compression boolean Set to true to compress HEC JSON data and increase throughput at the expense of increasing pipeline CPU utilization. Defaults to false. "hec-gzip-compression": "false"
http-connection-timeout-ms integer The HTTP connection timeout, in milliseconds. If the HTTP server does not reply within the timeout period, the connection attempt is retried. "http-connection-timeout-ms": "10000"
http-connect-timeout-ms integer The HTTP connect timeout in milliseconds. If the HTTP connection is not established within the timeout period, the connection attempt is retried. "http-connect-timeout-ms": "10000"
http-socket-timeout-ms integer The HTTP socket timeout in milliseconds. If data is not received from the HTTP server within the timeout period, the socket is closed. "http-socket-timeout-ms": "10000"
async boolean Set to true to send data asynchronously. In async mode, send operations from DSP do not wait for a response to return therefore increasing performance. See Performance expectations for sending data from DSP pipelines to Splunk Enterprise. Defaults to false.

Best practices are to enable this for performance optimization. When async is enabled, the DSP HEC client attempts to write a HEC JSON payload to the Splunk HEC endpoint a maximum of three times. Each attempt has a 10 second timeout, and a maximum of 100 async I/O operations can happen concurrently across all indexers. If you require additional optimizations and you have a support contract, contact Splunk Customer Support.

"async": "true"

SPL2 example

In this example, your data comes out of batch_bytes as batched byte payloads with a max size of 2MB and is passed into the splunk_enterprise sink function. This data is then sent to the Splunk Enterprise endpoint for indexing.

When working in the SPL View, you can write the function by providing all arguments in this exact order.

...| batch_bytes bytes=to_bytes(host) size="2MB" millis=5000
| into splunk_enterprise(
    "b5c57cbd-1470-4639-9938-deb3509cbbc8",
    "events_idx",
    bytes,
    {"hec-enable-ack": "false", "hec-token-validation": "true"}
   );

Alternatively, you can use named arguments to declare the arguments in any order and leave out optional arguments you don't want to declare. All unprovided arguments use their default values. The following example provides the arguments in an arbitrary order.

...| batch_bytes bytes=to_bytes(host) size="2MB" millis=5000
| into splunk_enterprise(
    index: "events_idx",
    connection_id: "b5c57cbd-1470-4639-9938-deb3509cbbc8",
    parameters: {"hec-enable-ack": "false", "hec-token-validation": "true"},
    payload : bytes
   );

If you want to use a mix of unnamed and named arguments in your functions, you need to list all unnamed arguments in the correct order before providing the named arguments.

Last modified on 04 November, 2021
Send data to Splunk HTTP Event Collector   Send data to a Splunk index (Default for Environment)

This documentation applies to the following versions of Splunk® Data Stream Processor: 1.2.0, 1.2.1-patch02, 1.2.1, 1.2.2-patch02, 1.2.4, 1.2.5, 1.3.0, 1.3.1, 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.4.5


Was this topic useful?







You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters