Define a KV Store lookup in Splunk Web
KV Store lookups populate your events with fields pulled from your App Key Value Store (KV Store) collections. Invoke KV Store lookups through REST endpoints or by using the search commands lookup
, inputlookup
, and outputlookup
. Use a KV Store lookup when you have a large lookup table or a table that is updated often
KV Store vs. CSV files
The KV Store adds a lookup type to use with your apps. Before the KV Store feature was added, you might have used CSV-based lookups to augment data within your apps. Consider the following tradeoffs when deciding whether a KV Store lookup or a CSV-based lookup is best for your scenario:
Lookup type | Pros | Cons |
---|---|---|
KV Store lookup |
|
Does not support case-insensitive field lookups. |
CSV lookup |
|
|
KV Store collections
Before you create a KV Store lookup, your Splunk deployment must have at least one KV Store collection. Certain apps, such as Enterprise Security, include KV Store collections with their installation.
Splunk Web currently does not support the creation of KV Store collections. If you use Splunk Cloud Platform, you need to use the Splunk App for Lookup File Editing to add a unique KV Store collection to your Splunk deployment. To download the Splunk App for Lookup File Editing, see Splunk App for Lookup File Editing on Splunkbase.
If you have access to the configuration files for your Splunk deployment, you can create a KV Store collection yourself. See Use configuration files to create a KV Store collection on the Splunk Developer Portal.
KV Store collections are databases. They store your data as key/value pairs. When you create a KV Store lookup, the collection should have at least two fields. One of those fields should have a set of values that match with the values of a field in your event data, so that lookup matching can take place.
When you invoke the lookup in a search with the lookup
command, you designate a field in your search data to match with the field in your KV Store collection. When a value of this field in an event matches a value of the designated field in your KV Store collection, the corresponding value(s) for the other field(s) in your KV Store collection can be added to that event.
The KV Store field does not have to have the same name as the field in your events. Each KV Store field can be multivalued.
KV Store collections live on the search head, while CSV files are replicated to indexers. If your lookup data changes frequently you may find that KV Store lookups offer better performance than an equivalent CSV lookup.
Special KV Store collection configuration for federated search
If you plan to run standard mode federated searches that include KV Store lookups, ensure that the lookup definition and the KV Store collection are defined on both the local federated search head and the remote search heads of the standard mode federated providers in the search. See Custom knowledge object coordination for standard mode federated providers in the Search Manual.
In addition, you must ensure that replicate=true
is set in collections.conf
for the KV Store collection on the remote search head of the standard mode federated provider. This setting enables the lookup to run on the remote search head. If replicate=true
is not set for KV Store collections on your standard mode federated providers, your federated searches may return incorrect results.
If you use the Splunk App for Lookup File Editing to set up your KV Store collections, select Replicate when you define a KV Store lookup on your standard mode federated provider. Selecting Replicate sets replicate=true
for the KV Store collection that backs the KV Store lookup.
For more information about federated search see About federated search in the Search Manual.
Define a KV Store lookup
Prerequisites
- A KV Store collection. If you have Splunk Cloud Platform, file a Support ticket if you need a new KV Store collection. Otherwise, see Use configuration files to create a KV Store collection in the Splunk Developer Portal.
- About lookups
- Configure a time-bounded lookup
- Make your lookup automatic
Steps
- Select Settings > Lookups.
- Click Lookup definitions.
- Click Add new.
- Change the Type to KV Store.
- Enter the collection name to use.
- List all of the fields that are supported by the KV Store lookup. The fields must be delimited by a comma followed by a space. A field can be any combination of key and value that you have in your KV store collection.
- (Optional) Configure time-based lookup.
Time-based options Description Default value Name of time field Specify the name of the field in the lookup table that represents the timestamp. No value. Time format Specify the strptime format of the timestamp field. %s.%Q
- This is the UTC strptime format.Minimum offset The minimum time in seconds that the event time may be ahead of the lookup entry time for a match to occur. 0 Maximum offset The maximum time in seconds that the event time may be ahead of the lookup entry time for a match to occur. 2000000000 - (Optional) To define advanced options for your lookup, select the Advanced options check box.
Advanced options Description Default value Minimum matches The minimum number of matches for each input lookup value. 0
Maximum matches Enter a number from 1-1000 to specify the maximum number of matches for each lookup value. If time-based, the default value is 1
; otherwise, the default value is1000
.Default matches When fewer than the minimum number of matches are present for an input, the Splunk software provides this value one or more times until the minimum is reached.
Splunk software treats NULL values as matching values and does not replace them with the Default matches value.
No value. Maximum external batch The maximum size of the external batch. The range is 1 to 1000. Do not change this value unless you know what you are doing. 300
Match type Optionally set up non-exact matching of a comma-and-space-delimited field list. The format is <match_type>(<field_name1><field_name2>,...<field_nameN>)
. Available values for match type areWILDCARD
,CIDR
.EXACT
Filter lookup Filter results from the lookup table before returning data. Create this filter as a search query with Boolean expressions and comparison operators. To improve performance, KV store lookups filter their results when they first retrieve data. No value. - Click Save.
Your lookup is now defined as a KV Store lookup and will show up in the list of Lookup definitions.
Now that you have created a KV store lookup definition, you need share the definition with other users. You can share it with users of a specific app, or you can share it globally to users of all apps.
- In the Lookup definitions list, for the lookup definition you created, click Permissions.
- In the Permissions dialog box, under Object should appear in, select All apps to share globally or the app that you want to share it with.
- Click Save.
In the Lookup definitions page, your lookup now has the permissions you have set.
Permissions for lookup table files must be at the same level or higher than those of the lookup definitions that use those files.
Make the lookup automatic
Instead of using the lookup
command in your search when you want to apply a KV store lookup to your events, you can set the lookup to run automatically. When your lookup is automatic, the Splunk software applies it to all searches at search time.
See Define an automatic lookup in Splunk Web for more information.
Prefilter large KV Store collections
When your KV Store collection is extremely large, performance can suffer when your lookups must search through the entire collection to retrieve matching field values. If you know that you only need results from a subset of records in the lookup table, improve search performance by using the filter
attribute to filter out all of the records that do not need to be looked at.
The filter
attribute requires a string containing a search query with Boolean expressions and/or comparison operators (==, !=, >, <, <=, >=, OR , AND, and NOT). This query runs whenever you run a search that invokes this lookup.
For example, if your lookup configuration has filter = (CustID>500) AND (CustName="P*")
, it tries to retrieve values only from those records in the KV Store collection that have a CustID
value that greater than 500 and a CustName
value that begins with the letter P.
If you do not want to install a filter in the lookup definition you can get a similar effect when you use the where
clause in conjunction with the inputlookup
command.
Configure KV Store lookups with .conf files
KV Store lookups can also be configured using .conf files. See Configure KV store lookups for more information.
For developer-focused KV Store lookup configuration instructions, see Use lookups with KV Store data in the Splunk Developer Portal.
Define an external lookup in Splunk Web | Define a geospatial lookup in Splunk Web |
This documentation applies to the following versions of Splunk® Enterprise: 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.0.7, 9.0.8, 9.0.9, 9.0.10, 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.1.4, 9.1.5, 9.1.6, 9.2.0, 9.2.1, 9.2.2, 9.2.3, 9.3.0, 9.3.1
Feedback submitted, thanks!