Define a KV Store lookup in Splunk Web
KV Store lookups populate your events with fields pulled from your App Key Value Store (KV Store) collections. KV Store lookups can be invoked through REST endpoints or by using the search commands
outputlookup. Use a KV Store lookup when you have a large lookup table or a table that is updated often.
KV Store vs. CSV files
The KV Store adds a lookup type to use with your apps named KV Store lookups. Before the KV Store feature was added, you might have used CSV-based lookups to augment data within your apps. Consider the following tradeoffs when deciding whether a KV Store lookup or a CSV-based lookup is best for your scenario:
|KV Store lookup||
||Does not support case-insensitive field lookups.|
KV Store collections
Before you create a KV Store lookup, your Splunk deployment must have at least one KV Store collection defined in
collections.conf. See Use configuration files to create a KV Store collection on the Splunk Developer Portal.
Certain apps, such as Enterprise Security, also include KV Store collections with their installation. If you have Splunk Cloud and want to define KV Store lookups, use one of the default KV Store collections or file a Support ticket to add a unique KV Store collection.
KV Store collections are databases. They store your data as key/value pairs. When you create a KV Store lookup, the collection should have at least two fields. One of those fields should have a set of values that that match with the values of a field in your event data, so that lookup matching can take place.
When you invoke the lookup in a search with the
lookup command, you designate a field in your search data to match with the field in your KV Store collection. When a value of this field in an event matches a value of the designated field in your KV Store collection, the corresponding value(s) for the other field(s) in your KV Store collection can be added to that event.
The KV Store field does not have to have the same name as the field in your events. Each KV Store field can be multivalued.
KV Store collections live on the search head, while CSV files are replicated to indexers. If your lookup data changes frequently you may find that KV Store lookups offer better performance than an equivalent CSV lookup.
Define a KV Store lookup
- You must be an admin user with .conf and file directory access to create a KV Store collection. If you have Splunk Cloud and want to define KV Store lookups, file a Support ticket in order to add a collection.
- About lookups
- Use configuration files to create a KV Store collection store
- Configure a time-bounded lookup
- Make your lookup automatic
- Select Settings > Lookups.
- Click Lookup definitions.
- Click Add new.
- Change the Type to KV Store.
- Enter the collection name to use as defined in
- List all of the fields that are supported by the KV Store lookup. The fields must be delimited by a comma followed by a space. A field can be any combination of key and value that you have in your KV store collection.
- (Optional) Configure time-based lookup.
Time-based options Description Default value Name of time field Specify the name of the field in the lookup table that represents the timestamp. No value. Time format Specify the strptime format of the timestamp field.
%s.%Q- This is the UTC strptime format.
Minimum offset The minimum time in seconds that the event time may be ahead of the lookup entry time for a match to occur. 0 Maximum offset The maximum time in seconds that the event time may be ahead of the lookup entry time for a match to occur. 2000000000
- (Optional) To define advanced options for your lookup, select the Advanced options check box.
Advanced options Description Default value Minimum matches The minimum number of matches for each input lookup value.
Maximum matches Enter a number from 1-1000 to specify the maximum number of matches for each lookup value. If time-based, the default value is
1; otherwise, the default value is
Default matches When fewer than the minimum number of matches are present for an input, the Splunk software provides this value one or more times until the minimum is reached. No value. Maximum external batch The maximum size of the external batch. The range is 1 to 1000. Do not change this value unless you know what you are doing.
Match type Optionally set up non-exact matching of a comma-and-space-delimited field list. The format is
<match_type>(<field_name1><field_name2>,...<field_nameN>). Available values for match type are
Filter lookup Filter results from the lookup table before returning data. Create this filter as a search query with Boolean expressions and comparison operators. To improve performance, KV store lookups filter their results when they first retrieve data. No value.
- Click Save.
Your lookup is now defined as a KV Store lookup and will show up in the list of Lookup definitions.
Now that you have created a KV store lookup definition, you need share the definition with other users. You can share it with users of a specific app, or you can share it globally to users of all apps.
- In the Lookup definitions list, for the lookup definition you created, click Permissions.
- In the Permissions dialog box, under Object should appear in, select All apps to share globally or the app that you want to share it with.
- Click Save.
In the Lookup definitions page, your lookup now has the permissions you have set.
Permissions for lookup table files must be at the same level or higher than those of the lookup definitions that use those files.
Make the lookup automatic
Instead of using the
lookup command in your search when you want to apply a KV store lookup to your events, you can set the lookup to run automatically. When your lookup is automatic, the Splunk software applies it to all searches at search time.
See Define an automatic lookup in Splunk Web for more information.
Prefilter large KV Store collections
When your KV Store collection is extremely large, performance can suffer when your lookups must search through the entire collection to retrieve matching field values. If you know that you only need results from a subset of records in the lookup table, improve search performance by using the
filter attribute to filter out all of the records that do not need to be looked at.
filter attribute requires a string containing a search query with Boolean expressions and/or comparison operators (==, !=, >, <, <=, >=, OR , AND, and NOT). This query runs whenever you run a search that invokes this lookup.
For example, if your lookup configuration has
filter = (CustID>500) AND (CustName="P*"), it tries to retrieve values only from those records in the KV Store collection that have a
CustID value that greater than 500 and a
CustName value that begins with the letter P.
If you do not want to install a filter in the lookup definition you can get a similar effect when you use the
where clause in conjunction with the
Configure KV Store lookups with .conf files
KV Store lookups can also be configured using .conf files. See Configure KV store lookups for more information.
For developer-focused KV Store lookup configuration instructions, see Use lookups with KV Store data in the Splunk Developer Portal.
Define an external lookup in Splunk Web
Define a geospatial lookup in Splunk Web
This documentation applies to the following versions of Splunk® Enterprise: 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.0.13, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.1.10, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.10, 7.2.8, 7.2.9, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 8.0.0, 8.0.1, 7.2.7, 8.0.2, 8.0.3