Splunk® Enterprise

Knowledge Manager Manual

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Configure KV Store lookups

KV Store lookups populate your events with fields pulled from your App Key Value Store (KV Store) collections. KV Store lookups can be invoked through REST endpoints or by using the following search commands: lookup, inputlookup, and outputlookup. Use KV Store lookups when you have a significantly large lookup table or a table that is updated often.

You can also set up KV Store lookups as automatic lookups. Automatic lookups run in the background at search time and automatically add output fields to events that have the correct match fields. You do not need to invoke automatic lookups with the lookup command. See Make your lookup automatic.

This topic shows you how to set up and manage KV Store lookups by configuring lookup stanzas in props.conf. Configuration files give you a greater degree of control over lookup design and behavior than you get when you set up lookup files using Splunk Web. However, if you do not have access to the .conf files, or if you prefer to maintain lookups through Splunk Web whenever possible, you can configure KV Store lookups using the pages at Settings > Lookups. See Use lookups to add information to your events in this manual.

Splunk Cloud users: You must use Splunk Web to define lookups. If your Splunk Cloud deployment is a managed deployment, you must request a restart from Splunk Support after uploading lookup files, to make newly uploaded files appear in the list of files available for defining lookups.

You can also define lookups that:

  • Populate your events with fields pulled from CSV files.
  • Use Python scripts or binary executables to populate your events with field values from an external source.

See Configure CSV and external lookups in this manual.

For developer-focused KV Store lookup configuration instructions, see Use lookups with KV Store data in the Splunk Developer Portal.

About KV Store collections

Before you create a KV Store lookup, your Splunk deployment must have at least one KV Store collection defined in collections.conf. See Use configuration files to create a KV Store collection on the Splunk Developer Portal for information on working with JSON.

KV Store collections are containers of data similar to a database. They store your data as key/value pairs. When you create a KV Store lookup, the collection should have at least two fields. One of those fields should have a set of values that that match with the values of a field in your event data, so that lookup matching can take place.

When you invoke the lookup in a search with the lookup command, you designate a field in your search data to match with the field in your KV Store collection. When a value of this field in an event matches a value of the designated field in your KV Store collection, the corresponding value(s) for the other field(s) in your KV Store collection can be added to that event.

The KV Store field does not have to have the same name as the field in your events. Each KV Store field can be multivalued.

KV Store collections live on the search head, while CSV files are replicated to indexers. If your lookup data changes frequently you may find that KV Store lookups offer better performance than an equivalent CSV lookup.

Define a KV Store lookup stanza in transforms.conf

A transforms.conf KV Store lookup stanza provides the location of the KV Store collection that is to be used as a lookup table. It can optionally include field matching rules and rules for time-bounded lookups.

If you want a KV Store lookup to be available globally, add its lookup stanza to the version of transforms.conf in $SPLUNK_HOME/etc/system/local/. If you want the lookup to be specific to a particular app, add its stanza to the version of transforms.conf in $SPLUNK_HOME/etc/apps/<app_name>/local/.

Caution: Do not edit configuration files in $SPLUNK_HOME/etc/system/default.

The KV Store lookup stanza format

When you add a KV Store lookup stanza to transforms.conf it should follow this format.

[<lookup_name>]
external_type = kvstore
collection = <string>
fields_list = <string>
filter = <string>
  • [<lookup_name>] is the name of the lookup.
  • external_type should be set to kvstore if you are defining a KV store lookup.
  • collection is the name of the KV Store collection associated with the lookup.
  • fields_list is a list of all fields that are supported by the KV Store lookup. The fields must be delimited by a comma followed by a space. A field can be any combination of key and value that you have in your KV store collection.

    By default, each KV Store record has a unique key ID, which is stored in the internal _key field. Add _key to the list of fields in fields_list if you want to be able to modify specific records through your KV Store lookup. You can then specify the key ID value in your lookup operations.

    When you use the outputlookup command to write to the KV Store without specifying a key ID, a key ID is generated for you.

Configure a KV Store lookup

Prerequisities

Steps
If you have Splunk Cloud and want to define KV store lookups, file a Support ticket. If you have Splunk Enterprise, perform the following steps.

  1. Define a KV Store collection in collections.conf.
  2. Create a KV Store lookup stanza in transforms.conf, following the stanza format described above.
    If you want the lookup to be available globally, add its lookup stanza to the version of transforms.conf in $SPLUNK_HOME/etc/system/local/. If you want the lookup to be specific to a particular app, add its stanza to the version of transforms.conf in $SPLUNK_HOME/etc/apps/<app_name>/local/.
    Caution: Do not edit configuration files in $SPLUNK_HOME/etc/system/default.
  3. (Optional) Use the filter attribute to prefilter significantly large KV Store lookup tables.
    You can speed up lookup searches against significantly large KV Store collections by using the filter attribute to restrict the searches.
  4. (Optional) Set up field/value matching rules for the KV Store lookup.
  5. (Optional) If the KV Store collection contains time fields, make the KV Store lookup time-bounded.
  6. (Optional) Make the KV Store lookup an automatic lookup by adding a configuration to props.conf.
    If you want the automatic lookup to be available globally, add its lookup stanza to the version of props.conf in $SPLUNK_HOME/etc/system/local/. If you want the lookup to be specific to a particular app, add its stanza to the version of props.conf in $SPLUNK_HOME/etc/apps/<app_name>/local/.
    Caution: Do not edit configuration files in $SPLUNK_HOME/etc/system/default.
  7. Save your .conf file changes.
  8. Restart Splunk Enterprise to implement your changes.
    If you have set up an automatic lookup, after restart you should see the output fields from your lookup table listed in the fields sidebar. From there, you can select the fields to display in each of the matching search results.

Prefilter large KV Store collections

When your KV Store collection is extremely large, performance can suffer when your lookups must search through the entire collection to retrieve matching field values. If you know that you only need results from a subset of records in the lookup table, improve search performance by using the filter attribute to filter out all of the records that do not need to be looked at.

The filter attribute requires a string containing a search query with Boolean expressions and/or comparison operators (==, !=, >, <, <=, >=, OR , AND, and NOT). This query runs whenever you run a search that invokes this lookup.

For example, if your lookup configuration has filter = (CustID>500) AND (CustName="P*"), it tries to retrieve values only from those records in the KV Store collection that have a CustID value that greater than 500 and a CustName value that begins with the letter P.

Note: If you do not want to install a filter in the lookup definition you can get a similar effect when you use the where clause in conjunction with the inputlookup command.

KV store lookup example

Here is a KV Store lookup called employee_info. It is located in your app's $SPLUNK_HOME/etc/system/local/ directory.

[employee_info]
external_type = kvstore
collection = kvstorecoll
fields_list = _key, CustID, CustName, CustStreet, CustCity, CustZip
filter = (CustID>500) AND (CustName="P*")

The employee_info lookup takes an employee ID in an event and outputs corresponding employee information to that event such as the employee name, street address, city, and zip code. The lookup works with a KV Store collection called kvstorecoll. The filter restricts the lookup query to records with a customer ID greater than 500 and a customer name that begins with the letter "P".

To see how to make this KV Store lookup "automatic" by adding a configuration to props.conf, see "Make your lookup automatic," in this manual.

Search commands and KV Store lookups

After you save a KV Store lookup stanza and restart Splunk Enterprise, you can interact with the new KV store lookup through search commands.

Use lookup to match values in a KV Store collection with field values in the search results and then output corresponding field values to those results. This search uses the employee_info lookup defined in the preceding use case example.

... | lookup employee_info CustID AS ID OUTPUT CustName AS Name | ...

It matches employee id values in kvstorecoll with employee id values in your events and outputs the corresponding employee name values to your events.

You can use the inputlookup search command to search on the contents of a KV Store collection. See the Search Reference topic on inputlookup for examples.

You can use the outputlookup search command to write search results from the search pipeline into a KV store collection. See the Search Reference topic on outputlookup for examples.

You can also find several examples of KV Store lookup searches in "Use lookups with KV Store data" in the Splunk Developer Portal.

PREVIOUS
Configure external lookups
  NEXT
Configure geospatial lookups

This documentation applies to the following versions of Splunk® Enterprise: 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.5.0, 6.5.1, 6.5.2, 6.5.3, 6.5.4


Comments

The examples using the following transforms.conf stanza
[employee_info]
external_type = kvstore
collection = kvstorecoll
fields_list = _key, CustID, CustName, CustStreet, CustCity, CustZip
filter = (CustID>500) AND (CustName="P*")

should be revised to reflect Employee info such as EmpID, EmpName, EmpStreet, EmpCity, EmpZip, etc. instead of CustID, CustName, CustStreet, CustCity, CustZip

As well as the example searches such as ... | lookup employee_info CustID AS ID OUTPUT CustName AS Name | ...

A nit but it is inconsistent and caused me to look at it twice to infer that it was an error in content style.

KSharp

Ksharp splunk, Splunker
November 14, 2016

Wpreston: Apologies for the confusion. The fact that KV Store lookups can be configured to be automatic wasn't documented in time for the 6.3 release. We have been in the process of updating the lookups documentation to make this clear and also make the lookups documentation easier to understand as a whole. We have corrected the topic.

Mness, Splunker
November 18, 2015

It also says that you can't set up a KV store lookup as an automatic lookup in the "Define a KV Store lookup stanza in transforms.conf" section.

Wpreston
November 18, 2015

I think the first paragraph needs to be corrected. In the first paragraph, it states that "You cannot set up KV Store lookups as automated lookups." However, it step 6 of the Configure a KV Store Lookup says that you can set up an automatic lookup using a KV store.

Wpreston
November 18, 2015

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters