Configure a time-based lookup

If your lookup table has a field that represents time, you can use it to create a time-based lookup. This is also referred to as a temporal lookup. You can configure all four lookup types as time-based lookups.

Simple time-based lookups attempt to match the event timestamp with the timestamp of a record in the lookup table, and then perform operations like adding one or more fields to the event from the matched record.

You can also define time-bound lookups, which use the event time to define a range of time within which to match lookup records. For example, you could create a time-bound lookup that matches the first lookup table record with a timestamp that falls within 10 seconds before the event timestamp.

Defining time-based lookups

To create a simple time-based lookup, add the following lines to your lookup stanza in transforms.conf:

time_field = <field_name>
time_format = <string>

Here are the definitions of these settings.

Setting	Description	Default
time_field	Identifies the field in the lookup table that represents the timestamp. The search processor applies the first matching entry in descending order. When time_field is present in a saved search stanza, max_matches = 1 by default. For more information about max_matches see Add field matching rules to your lookup configuration.	Defaults to an empty string, because lookups are not time-based by default.
time_format	Specifies the strptime() format of the time_field attribute. You can use some nonstandard date-time strptime() formats. See the material about enhanced strptime() support in Configure timestamp recognition in the Getting Data In Manual.	%s.%Q This is the Unix epoch time value in seconds (%s), with optional milliseconds (%Q).

Defining time-bound lookups

To create a time-bound lookup, add these optional settings to your time-based lookup configuration:

max_offset_secs = <integer>
min_offset_secs = <integer>

Here are the definitions of these settings:

Setting	Description	Default
max_offset_secs	The maximum amount of time in seconds that an event timestamp can be later than the lookup record timestamp, for a match to occur.	2000000000 (effectively no default)
min_offset_secs	The minimum amount of time in seconds that an event timestamp can be later than the lookup record timestamp, for a match to occur.	0

The max_offset_secs and min_offset_secs settings define the earliest and latest times within which the search processor can search for matching records in the lookup table. The search processor calculates the earliest and latest time values from the event time like this:

earliest = event timestamp - max_offset_secs
latest = event timestamp - min_offset_secs

Within this window of time, the search processor applies a match in descending order of time up to the point where we get max_matches number of matches for that event. If max_matches is not set, it defaults to 1. For more information about max_matches see Add field matching rules to your lookup configuration.

Time-based lookup example

Here's an example of a CSV lookup that uses DHCP logs to identify users on a network based on their IP address and the timestamp. The DHCP logs are in a file, dhcp.csv, which contains the timestamp, IP address, and the user's name and MAC address.

Prerequisities

• See about lookups and field actions for more information on lookups.
• See Make your lookup automatic for information on configuring an automatic lookup.

Steps

1. In a transforms.conf file, put:

   [dhcpLookup]
   filename = dhcp.csv
   time_field = timestamp
   time_format = %d/%m/%y %H:%M:%S
   

2. In a props.conf file, make the lookup automatic:

   [dhcp]
   LOOKUP-table = dhcpLookup ip mac OUTPUT user
   

3. Save your file changes.

If you wanted to turn this into a time-bound lookup, you could add the following settings to the [dhcpLookup] stanza in transforms.conf:

max_offset_secs = 10
min_offset_secs = 0

This would cause the lookup to match events to the first lookup table record with a timestamp that falls within a range of time bound by the event timestamp and ten seconds before the event timestamp.