Splunk® Data Stream Processor

Use the Data Stream Processor

On April 3, 2023, Splunk Data Stream Processor reached its end of sale, and will reach its end of life on February 28, 2025. If you are an existing DSP customer, please reach out to your account team for more information.

All DSP releases prior to DSP 1.4.0 use Gravity, a Kubernetes orchestrator, which has been announced end-of-life. We have replaced Gravity with an alternative component in DSP 1.4.0. Therefore, we will no longer provide support for versions of DSP prior to DSP 1.4.0 after July 1, 2023. We advise all of our customers to upgrade to DSP 1.4.0 in order to continue to receive full product support from Splunk.

About lookups

Lookups enrich your data by adding field-value combinations from lookup datasets. The uses lookups to match field-value combinations in your data with field-value combinations in external lookup files. If those field-value combinations are found in your lookup file, the corresponding field-value combinations from the file are appended to your data.

Types of lookups

There are two types of lookups:

  • CSV lookups
  • KV Store lookups
Lookup type Data source Description
CSV A CSV file Populates your events with fields pulled from CSV files. Each column in a CSV table is interpreted as the potential values of a field.

Use CSV lookups when you have small sets of data that are relatively static. The maximum file size is 50MB.

KV Store A Splunk Enterprise KV Store collection Matches fields in your events to fields in a KV Store collection and outputs corresponding fields in that collection to your events. In order to use the KV Store lookup, you must first create a KV Store collection. See Use configuration files to create a KV Store collection in the Splunk>Dev documentation.

Use a KV Store lookup when you have a large lookup table (over 50MB) or a table that is updated often. Modifications to the KV Store collection do not typically require a pipeline restart unless you are changing the schema of the KV Store collection. Changes to the lookup connection, such as changes to the username, password, KV Store URL, or collection name, will require a pipeline restart.

The currently supports lookups to KV Store collections up to 10GB in size or 6.5 million records, depending on whichever is lower. If you are performing lookups to a distributed Splunk Enterprise environment, make sure you have an appropriately sized Splunk Enterprise environment capable of handling many requests per second. See Troubleshoot lookups to the Splunk Enterprise KV Store for more information.

Last modified on 03 March, 2022
Summarize records with the stats function   Upload a CSV file to the to enrich data with a lookup

This documentation applies to the following versions of Splunk® Data Stream Processor: 1.3.0, 1.3.1, 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4

Was this topic useful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters