Splunk® Enterprise

Knowledge Manager Manual

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Configure KV Store lookups

Lookups add fields from an external source to your events based on the values of fields currently present in those events. This topic discusses KV Store lookups, which populate your events with fields pulled from your App Key Value Store (KV Store) collections. KV Store lookups can only be invoked through REST endpoints or by using the following search commands: lookup, inputlookup, and outputlookup. You cannot set up KV Store lookups as automated lookups.

This topic shows you how to set up and manage KV Store lookups by configuring lookup stanzas in props.conf. Configuration files give you a greater degree of control over lookup design and behavior than you get when you set up lookup files using Splunk Web.

If you do not have access to the .conf files, or if you prefer to maintain lookups through Splunk Web whenever possible, you can configure KV Store lookups using the pages at Settings > Lookups. See "Use lookups to add information to your events" in this manual.

You can also define lookups that:

  • Populate your events with fields pulled from CSV files.
  • Use Python scripts or binary executables to populate your events with field values from an external source.

See "Configure CSV and external lookups" in this manual.

For developer-focused KV Store lookup configuration instructions, see "Use lookups with KV Store data" in the Splunk Developer Portal.

About KV Store collections

Before you create a KV Store lookup, your Splunk Enterprise implementation must have at least one KV Store collection defined in collections.conf. See "Use configuration files to create a KV Store collection store" on the Splunk Developer Portal.

KV Store collections are containers of data similar to a database. They store your data as key/value pairs. When you create a KV Store lookup, the collection should have at least two fields. One of those fields should have a set of values that that match with the values of a field in your event data, so that lookup matching can take place.

When you invoke the lookup in a search with the lookup command, you designate a field in your search data to match with the field in your KV Store collection. When a value of this field in an event matches a value of the designated field in your KV Store collection, the corresponding value(s) for the other field(s) in your KV Store collection can be added to that event.

The KV Store field does not have to have the same name as the field in your events. Each KV Store field can be multivalued.

Note: KV Store collections live on the search head, while CSV files are replicated to indexers. If your lookup data changes frequently you may find that KV Store lookups offer better performance than an equivalent CSV lookup.

Define a KV Store lookup stanza in transforms.conf

A KV Store lookup matches fields in your events with fields that Splunk Enterprise has stored in KV store collections. KV store lookups can only be invoked using search commands: lookup, inputlookup, and outputlookup. You cannot set up KV Store lookups as automated lookups.

A KV Store lookup stanza provides the location of the KV Store collection that is to be used as a lookup table. It can optionally include field matching rules and rules for time-bounded lookups.

If you want a KV Store lookup to be available globally, add its lookup stanza to the version of transforms.conf in $SPLUNK_HOME/etc/system/local/. If you want the lookup to be specific to a particular app, add its stanza to the version of transforms.conf in $SPLUNK_HOME/etc/apps/<app_name>/local/.

Caution: Do not edit configuration files in $SPLUNK_HOME/etc/system/default.

The KV Store lookup stanza format

When you add a KV Store lookup stanza to transforms.conf it should follow this format.

[<lookup_name>]
external_type = kvstore
collection = <string>
fields_list = <string>
  • [<lookup_name>] is the name of the lookup.
  • external_type should be set to kvstore if you are defining a KV store lookup.
  • collection is the name of the KV Store collection associated with the lookup.
  • fields_list is a list of all fields that are supported by the KV Store lookup. The fields must be delimited by a comma followed by a space. A field can be any combination of key and value that you have in your KV store collection.
By default, each KV Store record has a unique key ID, which is stored in the internal "_key" field. Add _key to the list of fields in fields_list if you want to be able to modify specific records through your KV Store lookup. You can then specify the key ID value in your lookup operations.
When you use the outputlookup command to write to the KV Store without specifying a key ID, Splunk Enterprise generates a key ID for you.

The transforms.conf KV store lookup stanza can optionally include attributes that:

Configure a KV Store lookup

1. Define a KV Store collection in collections.conf. See "Use configuration files to create a KV Store collection store" on the Splunk Developer Portal.

2. Create a KV Store lookup stanza in transforms.conf, following the stanza format described above.

3. Save your .conf file changes.

4. Restart Splunk Enterprise to add the lookup to your system.

KV store lookup example

Here is a KV Store lookup called employee_info. It is located in your app's $SPLUNK_HOME/etc/system/local directory.

[employee_info]
external_type = kvstore
collection = kvstorecoll
fields_list = _key, CustID, CustName, CustStreet, CustCity, CustZip

The employee_info lookup takes an employee ID in an event and outputs corresponding employee information to that event such as the employee name, street address, city, and zip code. The lookup works with a KV Store collection called kvstorecoll.

Search commands and KV Store lookups

After you save a KV Store lookup stanza and restart Splunk, you can interact with the new KV store lookup through search commands.

Use lookup to match values in a KV Store collection with field values in the search results and then output corresponding field values to those results. This search uses the employee_info lookup defined in the preceding use case example.

... | lookup employee_info CustID AS ID OUTPUT CustName AS Name | ...

It matches employee id values in kvstorecoll with employee id values in your events and outputs the corresponding employee name values to your events.

You can use the inputlookup search command to search on the contents of a KV Store collection. See the Search Reference topic on inputlookup for examples.

You can use the outputlookup search command to write search results from the search pipeline into a KV store collection. See the Search Reference topic on outputlookup for examples.

You can also find several examples of KV Store lookup searches in "Use lookups with KV Store data" in the Splunk Developer Portal.

PREVIOUS
Configure CSV and external lookups
  NEXT
About workflow actions in Splunk Web

This documentation applies to the following versions of Splunk® Enterprise: 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.2.14, 6.2.15


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters