Splunk® Data Stream Processor

Use the Data Stream Processor

Acrobat logo Download manual as PDF

Acrobat logo Download topic as PDF

About lookups

Lookups enrich your data by adding field-value combinations from lookup datasets. The uses lookups to match field-value combinations in your data with field-value combinations in external lookup files. If those field-value combinations are found in your lookup file, the corresponding field-value combinations from the file are appended to your data.

Types of lookups

There are two types of lookups:

  • CSV lookups
  • KV Store lookups
Lookup type Data source Description
CSV A CSV file Populates your events with fields pulled from CSV files. Each column in a CSV table is interpreted as the potential values of a field.

Use CSV lookups when you have small sets of data that are relatively static. If you want to modify the CSV file used by a CSV lookup, you'll need to restart the associated pipelines. The maximum file size is 50MB.

KV Store A Splunk Enterprise KV Store collection Matches fields in your events to fields in a KV Store collection and outputs corresponding fields in that collection to your events. In order to use the KV Store lookup, you must first create a KV Store collection. See Use configuration files to create a KV Store collection in the Splunk>Dev documentation.

Use a KV Store lookup when you have a large lookup table (over 50MB) or a table that is updated often. Modifications to the KV Store collection do not typically require a pipeline restart unless you are changing the schema of the KV Store collection. Changes to the lookup connection, such as changes to the username, password, KV Store URL, or collection name, will require a pipeline restart.

The currently supports lookups to KV Store collections up to 10GB in size or 6.5 million records, depending on whichever is lower. If you are performing lookups to a distributed Splunk Enterprise environment, make sure you have an appropriately sized Splunk Enterprise environment capable of handling many requests per second. See Troubleshoot lookups to the Splunk Enterprise KV Store for more information.

Last modified on 19 February, 2021
Summarize records with the stats function
Upload a CSV file to the to enrich data with a lookup

This documentation applies to the following versions of Splunk® Data Stream Processor: 1.2.0, 1.2.1

Was this documentation topic helpful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters