Splunk® Enterprise

Knowledge Manager Manual

Download manual as PDF

Download topic as PDF

Configure a time-based lookup

If your lookup table has a field that represents time, you can use it to create a time-based lookup. This is also referred to as a temporal lookup. You can configure all four lookup types as time-based lookups.

Simple time-based lookups attempt to match the event timestamp with the timestamp of a record in the lookup table, and then perform operations like adding one or more fields to the event from the matched record.

You can also define time-bounded lookups, which use the event time to define a range of time within which to match lookup records. For example, you could create a time-bounded lookup that matches the first lookup table record with a timestamp that falls within 10 seconds before the event timestamp.

Defining time-based lookups

To create a simple time-based lookup, add the following lines to your lookup stanza in transforms.conf:

time_field = <field_name>
time_format = <string>

Here are the definitions of these settings.

Setting Description Default
time_field Identifies the field in the lookup table that represents the timestamp. The search processor applies the first matching entry in descending order.

When time_field is present in a saved search stanza, max_matches = 1 by default. For more information about max_matches see Add field matching rules to your lookup configuration.
Defaults to an empty string, because lookups are not time-based by default.
time_format Specifies the strptime() format of the time_field attribute. You can use some nonstandard date-time strptime() formats. See the material about enhanced strptime() support in Configure timestamp recognition in the Getting Data In Manual. %s.%Q

This is the Unix epoch time value in seconds (%s), with optional milliseconds (%Q).

Defining time-bounded lookups

To create a time bounded lookup, add these optional settings to your time-based lookup configuration:

max_offset_secs = <integer>
min_offset_secs = <integer>

Here are the definitions of these settings:

Setting Description Default
max_offset_secs The maximum amount of time in seconds that an event timestamp can be later than the lookup record timestamp, for a match to occur. 2000000000 (effectively no default)
min_offset_secs The minimum amount of time in seconds that an event timestamp can be later than the lookup record timestamp, for a match to occur. 0

The max_offset_secs and min_offset_secs settings define the earliest and latest times within which the search processor can search for matching records in the lookup table. The search processor calculates the earliest and latest time values from the event time like this:

earliest = event timestamp - max_offset_secs
latest = event timestamp - min_offset_secs

Within this window of time, the search processor applies a match in descending order of time up to the point where we get max_matches number of matches for that event. If max_matches is not set, it defaults to 1. For more information about max_matches see Add field matching rules to your lookup configuration.

Time-based lookup example

Here's an example of a CSV lookup that uses DHCP logs to identify users on a network based on their IP address and the timestamp. The DHCP logs are in a file, dhcp.csv, which contains the timestamp, IP address, and the user's name and MAC address.

Prerequisities

Steps

  1. In a transforms.conf file, put:
    [dhcpLookup]
    filename = dhcp.csv
    time_field = timestamp
    time_format = %d/%m/%y %H:%M:%S
    
  2. In a props.conf file, make the lookup automatic:
    [dhcp]
    LOOKUP-table = dhcpLookup ip mac OUTPUT user
    
  3. Save your file changes.

If you wanted to turn this into a time-bounded lookup, you could add the following settings to the [dhcpLookup] stanza in transforms.conf:

max_offset_secs = 10
min_offset_secs = 0

This would cause the lookup to match events to the first lookup table record with a timestamp that falls within a range of time bounded by the event timestamp and ten seconds before the event timestamp.

PREVIOUS
Add field matching rules to your lookup configuration
  NEXT
Make your lookup automatic

This documentation applies to the following versions of Splunk® Enterprise: 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.0.13, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 8.0.0


Comments

@Woodcock

As you can see, we've updated the topic to better explain the matching logic for time-bounded lookups. It's also been restructured to do a better job of differentiating between simple "time-based" lookups and slightly more complex "time-bounded" lookups. Hope this helps.

Mness, Splunker
January 10, 2018

This does not describe the algorithm for matching and really, that is the most important part. Does it use the min/max offsets only in 1 direction? Does it match "closest time that is not bigger" or "closest time that is not smaller" or what?

Woodcock
December 29, 2017

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters