Splunk® Data Stream Processor

Function Reference

Acrobat logo Download manual as PDF

Acrobat logo Download topic as PDF

Send data to a Splunk index

Send data to an external Splunk Enterprise system. You must create a Splunk Enterprise connection to use this function, see Create a DSP connection to a Splunk index. This function sends data to a Splunk Enterprise index using the Splunk HTTP Event Collector (HEC), which requires events to be formatted in a specific way. See Connecting Splunk indexes to your DSP pipeline for more information.

Function input schema

See Connecting Splunk indexes to your DSP pipeline.

Required arguments

connection_id
Syntax: string
Description: The ID of the Splunk Enterprise Connection.
Example: "576205b3-f6f5-4ab7-8ffc-a4089a95d0c4"
index
Syntax: expression<string>
Description: The index to send your data to.
Example:"main"
payload
Syntax: expression<bytes>
Description: The name of the DSP record field (for example, "bytes") that has the byte payload to be written directly to the HEC endpoint. Set to null if your records are not in bytes.
Example: bytes

Optional arguments

parameters
Syntax: map<string, string>
Description: The optional parameters you can enter in this function. See the following table for a description of each parameter.
Parameter Input Description Example
parameters map<string, string> The following rows list the optional parameters you can enter in this function. See the "Parameters" table for available options. Defaults to empty { }.
hec-token-validation boolean Set to true to enable HEC token validation. Defaults to true. hec-token-validation: true
hec-enable-ack boolean Set to true for the function to wait for an acknowledgement for every single event. Set to false if acknowledgments in your Splunk platform are disabled or to increase throughput. Defaults to true. hec-enable-ack: true
hec-gzip-compression boolean Set to true to compress HEC JSON data and increase throughput at the expense of increasing pipeline CPU utilization. Defaults to false. hec-gzip-compression: false
async boolean Set to true to send data asynchronously. In async mode, send operations from DSP do not wait for a response to return therefore increasing performance. Best practices are to enable this for performance optimization. See Performance expectations for sending data from DSP pipelines to Splunk Enterprise. Defaults to false. async: false
async-max-retries number Ignored unless async is set to True. The maximum number of times the DSP HEC client will attempt to write a HEC JSON payload to the Splunk Enterprise HEC endpoint. Defaults to 3. async-max-retries: 4
async-timeout-ms number Ignored unless async is set to True. The maximum time, in milliseconds, to wait for an asynchronous write to the Splunk Enterprise HEC endpoint to succeed. When an async I/O request times out, an exception is thrown and the pipeline is restarted. Defaults to 10000 (10 seconds). async-timeout-ms: 10000
async-capacity number Ignored unless async is set to True. The maximum number of async I/O operations that can be happening concurrently. This number is total across all indexers, not per indexer. Defaults to 100. async-capacity: 100

SPL2 example

In this example, your data comes out of batch_bytes as batched byte payloads with a max size of 2MB and is passed into the splunk_enterprise sink function. This data is then sent to the Splunk Enterprise endpoint for indexing.

You can write the function by providing all arguments in this exact order.

...| batch_bytes bytes=to_bytes(host) size="2MB" millis=5000
| into splunk_enterprise(
    "b5c57cbd-1470-4639-9938-deb3509cbbc8",
    "events_idx",
    bytes,
    {"async": "true", "hec-enable-ack": "false", "hec-token-validation": "true"}
   );

Alternatively, you can use named arguments to declare the arguments in any order and leave out optional arguments you don't want to declare. All unprovided arguments use their default values. See SPL2 syntax for more details. The following example provides the arguments in an arbitrary order.

...| batch_bytes bytes=to_bytes(host) size="2MB" millis=5000
| into splunk_enterprise(
    index: "events_idx",
    connection_id: "b5c57cbd-1470-4639-9938-deb3509cbbc8",
    parameters: {"async": "true", "hec-enable-ack": "false", "hec-token-validation": "true"},
    payload : bytes
   );
Last modified on 05 November, 2020
PREVIOUS
Send data to a Splunk index with batching
  NEXT
Send data to a Splunk index (Default for Environment)

This documentation applies to the following versions of Splunk® Data Stream Processor: 1.2.0


Was this documentation topic helpful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters