Splunk® Data Stream Processor

Connect to Data Sources and Destinations with DSP

Acrobat logo Download manual as PDF


Acrobat logo Download topic as PDF

Process data from a universal forwarder in DSP

The Splunk universal forwarder sends unparsed data, which means that the data is sent into the in 64-kilobyte blocks. As a result, events that are too long might get truncated, and multiple events might be grouped together as one event. In addition, the timestamp associated with the event reflects the time when the event was ingested into DSP instead of the time when the event was originally generated.

Use the Apply Line Break and the Apply Timestamp Extraction functions to perform the following operations on your data:

  • Split the incoming stream of data into separate lines based on the location of a timestamp in the event body.
  • Merge the separated lines into events.
  • Extract the timestamp from the event body and use the extracted value as the timestamp of the event itself.

Prerequisites

Before you can process universal forwarder data in DSP, you must configure the universal forwarder to send data to DSP, and add either the Splunk DSP Firehose source function or the Forwarders Service source function to the start of your pipeline. See Create a connection between a Splunk forwarder and the Forwarders service.

The following instructions assume that you already have a pipeline that is ingesting data from a universal forwarder.

Steps

  1. On the Canvas View, click the + icon on your chosen source function to add a new function after it, and select the Apply Line Break function.
  2. On the View Configurations tab, confirm that Break type is set to Auto (Default). When Auto is selected, the function breaks events based on the location of timestamps in the event body. See Apply Line Break for information about other supported break types.
  3. Click the + icon on the Apply Line Break function to add a new function, and select the Apply Timestamp Extraction function.
  4. On the View Configurations tab, confirm that Extraction is set to Auto. When Auto is selected, the function uses built-in timestamp rules and the Splunk-provided datetime.xml file to detect and extract timestamps. See Apply Timestamp Extraction for information about other supported extraction types.
  5. (Optional) To confirm that your data is being processed successfully, do the following:
    1. Click the pipeline options icon (DSP Ellipses icon) located beside the Activate button, and select Validate.
    2. Click the Start Preview icon (Start Preview button) , and wait until the Preview Results tab displays the message Polling for preview data.
    3. Click each function and check the Preview Results tab to see how your data is being transformed by each function in your pipeline.
  6. Once you've confirmed that your universal forwarder events are being processed in the DSP as desired, click Save.

You now have a pipeline that performs the necessary transformations on universal forwarder events for DSP.

Last modified on 24 March, 2022
PREVIOUS
Allow DSP users to use the Forwarders service
  NEXT
Connecting your DSP pipeline to a Splunk index

This documentation applies to the following versions of Splunk® Data Stream Processor: 1.3.0, 1.3.1


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters