Process data from a universal forwarder in DSP

The Splunk universal forwarder sends unparsed data, which means that the data is sent into the in 64-kilobyte blocks. As a result, events that are too long might get truncated, and multiple events might be grouped together as one event. In addition, the timestamp associated with the event reflects the time when the event was ingested into DSP instead of the time when the event was originally generated.

Use the Apply Line Break and the Apply Timestamp Extraction functions to perform the following operations on your data:

• Split the incoming stream of data into separate lines based on the location of a timestamp in the event body.
• Merge the separated lines into events.
• Extract the timestamp from the event body and use the extracted value as the timestamp of the event itself.

Prerequisites

Before you can process universal forwarder data in DSP, you must configure the universal forwarder to send data to DSP, and add either the Splunk DSP Firehose source function or the Forwarders Service source function to the start of your pipeline. See Create a connection between a Splunk forwarder and the Forwarders service.

The following instructions assume that you already have a pipeline that is ingesting data from a universal forwarder.

Steps

1. On the Canvas View, click the + icon on your chosen source function to add a new function after it, and select the Apply Line Break function.
2. On the View Configurations tab, confirm that Break type is set to Auto (Default). When Auto is selected, the function breaks events based on the location of timestamps in the event body. See Apply Line Break for information about other supported break types.
3. Click the + icon on the Apply Line Break function to add a new function, and select the Apply Timestamp Extraction function.
4. On the View Configurations tab, confirm that Extraction is set to Auto. When Auto is selected, the function uses built-in timestamp rules and the Splunk-provided datetime.xml file to detect and extract timestamps. See Apply Timestamp Extraction for information about other supported extraction types.
5. (Optional) To confirm that your data is being processed successfully, do the following:
   1. Click the pipeline options icon () located beside the Activate button, and select Validate.
   2. Click the Start Preview icon () , and wait until the Preview Results tab displays the message Polling for preview data.
   3. Click each function and check the Preview Results tab to see how your data is being transformed by each function in your pipeline.
6. Once you've confirmed that your universal forwarder events are being processed in the DSP as desired, click Save.

You now have a pipeline that performs the necessary transformations on universal forwarder events for DSP.