Knowledge Manager Manual

Define a KV Store lookup in Splunk Web

KV Store lookups populate your events with fields pulled from your App Key Value Store (KV Store) collections. KV Store lookups can be invoked through REST endpoints or by using the search commands lookup, inputlookup, and outputlookup. Use a KV Store lookup when you have a large lookup table or a table that is updated often.

KV Store vs. CSV files

The KV Store adds a lookup type to use with your apps named KV Store lookups. Before the KV Store feature was added, you might have used CSV-based lookups to augment data within your apps. Consider the following tradeoffs when deciding whether a KV Store lookup or a CSV-based lookup is best for your scenario:

Lookup type	Pros	Cons
KV Store lookup	Enables per-record insert and updates. Allows optional data type enforcement on write operations. Allows you to define field accelerations to improve search performance. Provides REST API access to the data collection.	Does not support case-insensitive field lookups.
CSV lookup	Performs well for files that are small or rarely modified. CSV files are easier to modify manually. Integrating with other applications such as Microsoft Excel is easier because CSV is a standard format. Supports case-sensitive field lookups.	Does not provide multiuser access locking. Requires a full rewrite of the file for edit operations (outputlookup). Does not support REST API access.

KV Store collections

Before you create a KV Store lookup, your Splunk deployment must have at least one KV Store collection defined in collections.conf. See Use configuration files to create a KV Store collection on the Splunk Developer Portal.

Certain apps, such as Enterprise Security, also include KV Store collections with their installation. If you have Splunk Cloud and want to define KV Store lookups, use one of the default KV Store collections or file a Support ticket to add a unique KV Store collection.

KV Store collections are databases. They store your data as key/value pairs. When you create a KV Store lookup, the collection should have at least two fields. One of those fields should have a set of values that match with the values of a field in your event data, so that lookup matching can take place.

When you invoke the lookup in a search with the lookup command, you designate a field in your search data to match with the field in your KV Store collection. When a value of this field in an event matches a value of the designated field in your KV Store collection, the corresponding value(s) for the other field(s) in your KV Store collection can be added to that event.

The KV Store field does not have to have the same name as the field in your events. Each KV Store field can be multivalued.

KV Store collections live on the search head, while CSV files are replicated to indexers. If your lookup data changes frequently you may find that KV Store lookups offer better performance than an equivalent CSV lookup.

Define a KV Store lookup

Prerequisites

• You must be an admin user with .conf and file directory access to create a KV Store collection. If you have Splunk Cloud and want to define KV Store lookups, file a Support ticket in order to add a collection.

Review

• About lookups
• Use configuration files to create a KV Store collection store
• Configure a time-bounded lookup
• Make your lookup automatic

Steps

1. Select Settings > Lookups.
2. Click Lookup definitions.
3. Click Add new.
4. Change the Type to KV Store.
5. Enter the collection name to use as defined in collections.conf.
6. List all of the fields that are supported by the KV Store lookup. The fields must be delimited by a comma followed by a space. A field can be any combination of key and value that you have in your KV store collection.
7. (Optional) Configure time-based lookup.

   Time-based options	Description
   Name of time field	The minimum number of matches for each input lookup value. The default is 0.
   Time format	Enter a number from 1-1000 to specify the maximum number of matches for each lookup value. If time-based, the default is 1; otherwise, the default is 1000.
   Minimum offset	When fewer than the minimum number of matches are present for an given input, the Splunk software provides this value one or more times until the minimum is reached.
   Maximum offset	If the check box is selected, case-sensitive matching is performed for all fields in a lookup table. The default value is true.

8. (Optional) To define advanced options for your lookup, select the Advanced options check box.

   Advanced options	Description
   Minimum matches	The minimum number of matches for each input lookup value. The default value is 0.
   Maximum matches	Enter a number from 1-1000 to specify the maximum number of matches for each lookup value. If time-based, the default value is 1; otherwise, the default value is 1000.
   Default matches	When fewer than the minimum number of matches are present for an input, the Splunk software provides this value one or more times until the minimum is reached.
   Maximum external batch	The maximum size of the external batch. The range is 1 to 1000. The default is 300. Do not change this value unless you know what you are doing.
   Match type	Optionally set up non-exact matching of a comma-and-space-delimited field list. The format is <match_type>(<field_name1><field_name2>,...<field_nameN>). Available values for match_type are WILDCARD and CIDR.
   Filter lookup	Filter results from the lookup table before returning data. Create this filter as a search query with Boolean expressions and comparison operators.

9. Click Save.

Your lookup is now defined as a KV Store lookup and will show up in the list of Lookup definitions.

Share the lookup definition

Now that you have created a KV store lookup definition, you need share the definition with other users. You can share it with users of a specific app, or you can share it globally to users of all apps.

1. In the Lookup definitions list, for the lookup definition you created, click Permissions.
2. In the Permissions dialog box, under Object should appear in, select All apps to share globally or the app that you want to share it with.
3. Click Save.
   In the Lookup definitions page, your lookup now has the permissions you have set.

Permissions for lookup table files must be at the same level or higher than those of the lookup definitions that use those files.

Make the lookup automatic

Instead of using the lookup command in your search when you want to apply a KV store lookup to your events, you can set the lookup to run automatically. When your lookup is automatic, the Splunk software applies it to all searches at search time.

See Define an automatic lookup in Splunk Web for more information.

Prefilter large KV Store collections

When your KV Store collection is extremely large, performance can suffer when your lookups must search through the entire collection to retrieve matching field values. If you know that you only need results from a subset of records in the lookup table, improve search performance by using the filter attribute to filter out all of the records that do not need to be looked at.

The filter attribute requires a string containing a search query with Boolean expressions and/or comparison operators (==, !=, >, <, <=, >=, OR , AND, and NOT). This query runs whenever you run a search that invokes this lookup.

For example, if your lookup configuration has filter = (CustID>500) AND (CustName="P*"), it tries to retrieve values only from those records in the KV Store collection that have a CustID value that greater than 500 and a CustName value that begins with the letter P.

If you do not want to install a filter in the lookup definition you can get a similar effect when you use the where clause in conjunction with the inputlookup command.

Configure KV Store lookups with .conf files

KV Store lookups can also be configured using .conf files. See Configure KV store lookups for more information.

For developer-focused KV Store lookup configuration instructions, see Use lookups with KV Store data in the Splunk Developer Portal.