Define a KV Store lookup in Splunk Web

KV Store lookups populate your events with fields pulled from your App Key Value Store (KV Store) collections. Invoke KV Store lookups through REST endpoints or by using the search commands lookup, inputlookup, and outputlookup. Use a KV Store lookup when you have a large lookup table or a table that is updated often

KV Store vs. CSV files

The KV Store adds a lookup type to use with your apps. Before the KV Store feature was added, you might have used CSV-based lookups to augment data within your apps. Consider the following tradeoffs when deciding whether a KV Store lookup or a CSV-based lookup is best for your scenario:

Lookup type	Pros	Cons
KV Store lookup	Enables per-record insert and updates. Allows optional data type enforcement on write operations. Allows you to define field accelerations to improve search performance. Provides REST API access to the data collection.	Does not support case-insensitive field lookups.
CSV lookup	Performs well for files that are small or rarely modified. CSV files are easier to modify manually. Integrating with other applications such as Microsoft Excel is easier because CSV is a standard format. Supports case-sensitive field lookups.	Does not provide multiuser access locking. Requires a full rewrite of the file for edit operations (outputlookup). Does not support REST API access.

KV Store collections

Before you create a KV Store lookup, your Splunk deployment must have at least one KV Store collection. Certain apps, such as Enterprise Security, include KV Store collections with their installation.

Splunk Web currently does not support the creation of KV Store collections. If you use Splunk Cloud Platform, you need to use the Splunk App for Lookup File Editing to add a unique KV Store collection to your Splunk deployment. To download the Splunk App for Lookup File Editing, see Splunk App for Lookup File Editing on Splunkbase.

If you have access to the configuration files for your Splunk deployment, you can create a KV Store collection yourself. See Use configuration files to create a KV Store collection on the Splunk Developer Portal.

KV Store collections are databases. They store your data as key/value pairs. When you create a KV Store lookup, the collection should have at least two fields. One of those fields should have a set of values that match with the values of a field in your event data, so that lookup matching can take place.

When you invoke the lookup in a search with the lookup command, you designate a field in your search data to match with the field in your KV Store collection. When a value of this field in an event matches a value of the designated field in your KV Store collection, the corresponding value(s) for the other field(s) in your KV Store collection can be added to that event.

The KV Store field does not have to have the same name as the field in your events. Each KV Store field can be multivalued.

KV Store collections live on the search head, while CSV files are replicated to indexers. If your lookup data changes frequently you may find that KV Store lookups offer better performance than an equivalent CSV lookup.

Special KV Store collection configuration for federated search

If you plan to run standard mode federated searches that include KV Store lookups, ensure that the lookup definition and the KV Store collection are defined on both the local federated search head and the remote search heads of the standard mode federated providers in the search. See Custom knowledge object coordination for standard mode federated providers in the Search Manual.

In addition, you must ensure that replicate=true is set in collections.conf for the KV Store collection on the remote search head of the standard mode federated provider. This setting enables the lookup to run on the remote search head. If replicate=true is not set for KV Store collections on your standard mode federated providers, your federated searches may return incorrect results.

If you use the Splunk App for Lookup File Editing to set up your KV Store collections, select Replicate when you define a KV Store lookup on your standard mode federated provider. Selecting Replicate sets replicate=true for the KV Store collection that backs the KV Store lookup.

For more information about federated search see About federated search in the Search Manual.

Define a KV Store lookup

Prerequisites

• A KV Store collection. If you have Splunk Cloud Platform, file a Support ticket if you need a new KV Store collection. Otherwise, see Use configuration files to create a KV Store collection in the Splunk Developer Portal.
• About lookups
• Configure a time-bounded lookup
• Make your lookup automatic

Steps

1. Select Settings > Lookups.
2. Click Lookup definitions.
3. Click Add new.
4. Change the Type to KV Store.
5. Enter the collection name to use.
6. List all of the fields that are supported by the KV Store lookup. The fields must be delimited by a comma followed by a space. A field can be any combination of key and value that you have in your KV store collection.
7. (Optional) Configure time-based lookup.

   Time-based options	Description	Default value
   Name of time field	Specify the name of the field in the lookup table that represents the timestamp.	No value.
   Time format	Specify the strptime format of the timestamp field.	%s.%Q - This is the UTC strptime format.
   Minimum offset	The minimum time in seconds that the event time may be ahead of the lookup entry time for a match to occur.	0
   Maximum offset	The maximum time in seconds that the event time may be ahead of the lookup entry time for a match to occur.	2000000000

8. (Optional) To define advanced options for your lookup, select the Advanced options check box.

   Advanced options	Description	Default value
   Minimum matches	The minimum number of matches for each input lookup value.	0
   Maximum matches	Enter a number from 1-1000 to specify the maximum number of matches for each lookup value.	If time-based, the default value is 1; otherwise, the default value is 1000.
   Default matches	When fewer than the minimum number of matches are present for an input, the Splunk software provides this value one or more times until the minimum is reached. Splunk software treats NULL values as matching values and does not replace them with the Default matches value.	No value.
   Maximum external batch	The maximum size of the external batch. The range is 1 to 1000. Do not change this value unless you know what you are doing.	300
   Match type	Optionally set up non-exact matching of a comma-and-space-delimited field list. The format is <match_type>(<field_name1><field_name2>,...<field_nameN>). Available values for match type are WILDCARD, CIDR.	EXACT
   Filter lookup	Filter results from the lookup table before returning data. Create this filter as a search query with Boolean expressions and comparison operators. To improve performance, KV store lookups filter their results when they first retrieve data.	No value.

9. Click Save.

Your lookup is now defined as a KV Store lookup and will show up in the list of Lookup definitions.

Share the lookup definition

Now that you have created a KV store lookup definition, you need share the definition with other users. You can share it with users of a specific app, or you can share it globally to users of all apps.

1. In the Lookup definitions list, for the lookup definition you created, click Permissions.
2. In the Permissions dialog box, under Object should appear in, select All apps to share globally or the app that you want to share it with.
3. Click Save.
   In the Lookup definitions page, your lookup now has the permissions you have set.

Permissions for lookup table files must be at the same level or higher than those of the lookup definitions that use those files.

Make the lookup automatic

Instead of using the lookup command in your search when you want to apply a KV store lookup to your events, you can set the lookup to run automatically. When your lookup is automatic, the Splunk software applies it to all searches at search time.

See Define an automatic lookup in Splunk Web for more information.

Prefilter large KV Store collections

When your KV Store collection is extremely large, performance can suffer when your lookups must search through the entire collection to retrieve matching field values. If you know that you only need results from a subset of records in the lookup table, improve search performance by using the filter attribute to filter out all of the records that do not need to be looked at.

The filter attribute requires a string containing a search query with Boolean expressions and/or comparison operators (==, !=, >, <, <=, >=, OR , AND, and NOT). This query runs whenever you run a search that invokes this lookup.

For example, if your lookup configuration has filter = (CustID>500) AND (CustName="P*"), it tries to retrieve values only from those records in the KV Store collection that have a CustID value that greater than 500 and a CustName value that begins with the letter P.

If you do not want to install a filter in the lookup definition you can get a similar effect when you use the where clause in conjunction with the inputlookup command.

Configure KV Store lookups with .conf files

KV Store lookups can also be configured using .conf files. See Configure KV store lookups for more information.

For developer-focused KV Store lookup configuration instructions, see Use lookups with KV Store data in the Splunk Developer Portal.