Make your lookup automatic

When you create a lookup configuration in transforms.conf, you invoke it by running searches that reference it. However, you can optionally create an additional props.conf configuration that makes the lookup "automatic." This means that it runs in the background at search time and automatically adds output fields to events that have the correct match fields.

You can make all lookup types automatic. However, KV Store lookups have an additional setup step that you must complete before you configure them as automatic lookups in props.conf. See Enable replication for a KV Store collection.

Each automatic lookup configuration you create is limited to events that belong to a specific host, source, or source type. Automatic lookups can access any data in a lookup table that belongs to you or which you have shared.

When your lookup is automatic you do not need to invoke its transforms.conf configuration with the lookup command.

Splunk software does not support nested automatic lookups.

The automatic lookup format in props.conf

An automatic lookup configuration in props.conf:

• References the lookup table you configured in transforms.conf.
• Specifies the fields in your events that the lookup should match in the lookup table.
• Specifies the corresponding fields that the lookup should output from the lookup table to your events.

At search time, the LOOKUP-<class> configuration identifies a lookup and describes how that lookup should be applied to your events. To create an automatic lookup, follow this syntax:

[<spec>]
LOOKUP-<class> = $TRANSFORM <match_field_in_lookup_table> OUTPUT|OUTPUTNEW <output_field_in_lookup_table> 

• The stanza header is [<spec>]. <spec> can be:
  • <sourcetype>, the source type of an event.
  • host::<host>, where <host> is the host, or host-matching pattern, for an event.
  • source::<source>, where <source> is the source, or source-matching pattern, for an event.
• <spec> cannot use regular expression syntax.
• $TRANSFORM: References the transforms.conf stanza that defines the lookup table.
• match_field_in_lookup_table: This variable is the field in your lookup table that matches a field in your events with the same host, source, or source type as this props.conf stanza. If the match field in your events has a different name from the match field in the lookup table, use the AS clause as specified in Step 3, below.
• output_field_from_lookup_table: The corresponding field in the lookup table that you want to add to your events. If the output field in your events should have a different name from the output field in the lookup table, use the AS clause as specified in Step 3, below.

You can have multiple fields on either side of the lookup. For example, you can have:

$TRANSFORM <match_field_in_lookup_table1>, <match_field_in_lookup_table>OUTPUT|OUTPUTNEW <output_field_from_lookup_table1>, <output_field_from_lookup_table2>

You can also have one matching field return two output fields, three matching fields return one output field, and so on.

If you do not include an OUTPUT|OUTPUTNEW clause, Splunk software adds all the field names and values from the lookup table to your events. When you use OUTPUTNEW, Splunk software can add only the output fields that are "new" to the event. If you use OUTPUT, output fields that already exist in the event are overwritten.

If the "match" field names in the lookup table and your events are not identical, or if you want to "rename" the output field or fields that get added to your events, use the AS clause:

[<stanza name>]
LOOKUP-<class> = $TRANSFORM <match_field_in_lookup_table> AS <match_field_in_event> OUTPUT|OUTPUTNEW <output_field_from_lookup_table> AS <output_field_in_event>

For example, if the lookup table has a field named dept and you want the automatic lookup to add it to your events as department_name, set department_name as the value of <output_field_in_event>.

When you set up your match fields and output fields, avoid creating automatic lookup reference cycles. A reference cycle occurs when match and output fields are reused, either within the same lookup or among related lookups. For example, you do not want to set up a lookup where the <match_field_in_event> and the <output_field_in_event> values are the same. This can easily happen if you do not explicitly set an OUTPUT|OUTPUTNEW clause in your automatic lookup configuration.

For more information about automatic lookup reference cycles see Define an automatic lookup in Splunk Web.

Multiple lookup configurations in a single props.conf stanza

You can have multiple LOOKUP-<class> configurations in a single props.conf stanza. Each lookup should have its own unique lookup name. For example, if you have multiple lookups, you can name them LOOKUP-table1, LOOKUP-table2, and so on.

You can also have different props.conf automatic lookup stanzas that each reference the same lookup stanza in transforms.conf.

Create an automatic lookup stanza in props.conf

1. Create a stanza header that references the host, source, or source type that you are associating the lookup with.
2. Add a LOOKUP-<class> configuration to the stanza that you have identified or created.

   As described in the preceding section this configuration specifies:

   • What fields in your events it should match to fields in the lookup table.
   • What corresponding output fields it should add to your events from the lookup table.

   Be sure to make the <class> value unique. You can run into trouble if two or more automatic lookup configurations have the same <class> name. See "Do not use identical names in automatic lookup configurations."

3. (Optional) Include the AS clause in the configuration when the "match" field names in the lookup table and your events are not identical, or when you want to "rename" the output field or fields that get added to your events, use the AS clause.
4. Restart Splunk Enterprise to apply your changes.

   If you have set up an automatic lookup, after restart you should see the output fields from your lookup table listed in the fields sidebar. From there, you can select the fields to display in each of the matching search results.

Enable replication for a KV store collection

In Splunk Enterprise, KV Store collections are not bundle-replicated to indexers by default, and lookups run locally on the search head rather than on remote peers. If you enable replication for a KV Store collection, you can run the lookups on your indexers which let you use automatic lookups with your KV Store collections.

To enable replication for a KV Store collection and allow lookups against that collection to be automatic:

1. Open collections.conf.
2. Set replicate to true in the stanza for the collection.

   This parameter is set to false by default.

3. Restart Splunk Enterprise to apply your changes.

If your indexers are running a version of Splunk Enterprise that is older than 6.3, attempts to run an automatic lookup fail with a "lookup does not exist" error. You must upgrade your indexers to 6.3 or later to use this functionality.

For more information, see Use configuration files to create a KV Store collection at the Splunk Developer Portal.

Example configuration of an automatic KV Store lookup

This configuration references the example KV Store lookup configuration in Configure KV Store lookups, in this manual. The KV Store lookup is defined in transforms.conf, in a stanza named employee_info.

[access_combined]
LOOKUP-http = employee_info CustID AS cust_ID OUTPUT CustName AS cust_name, CustCity AS cust_city

This configuration uses the employee_info lookup in transforms.conf to add fields to your events. Specifically it adds cust_name and cust_city fields to any access_combined event with a cust_ID value that matches a CustID value in the kvstorecoll KV Store collection. It also uses the AS clause to:

• Find matching fields in the KV Store collection.
• Rename output fields when they are added to events.