Configure timestamp recognition

Configure timestamp recognition

Splunk uses timestamps to correlate events by time, create the histogram in Splunk Web and to set time ranges for searches. Timestamps are assigned to events at index time.

Splunk assigns a timestamp to most events based on information in the raw event data. If an event doesn't contain timestamp information, Splunk attempts to assign a timestamp value to the event as it's indexed. Splunk stores timestamp values in the _time field (in UTC time format). con Most events don't require any special timestamp handling; you can just let Splunk handle it without any configuration.

Precedence rules for timestamp assignment

Splunk uses the following precedence to assign timestamps to events:

1. Look for a time or date in the event itself using an explicit TIME_FORMAT if provided.

Use positional timestamp extraction for events that have more than one timestamp value in the raw data.

2. If no TIME_FORMAT is provided, or no match is found, attempt to automatically identify a time or date in the event itself.

3. If an event doesn't have a time or date, use the timestamp from the most recent previous event of the same source.

4. If no events in a source have a time or date, look in the source (or file) name.

5. For file sources, if no time or date can be identified in the file name, use the modification time on the file.

6. If no other timestamp is found, set the timestamp to the current system time (the time at which the event is indexed by Splunk).

Configure timestamps

Most events don't require any special timestamp handling; you can just let Splunk handle it without any configuration.

For some sources and distributed deployments, you may have to configure timestamp formatting to extract timestamps from events. Configure Splunk's timestamp extraction processor by editing props.conf.

Configure how Splunk recognizes timestamps by editing props.conf. Splunk uses strptime() formatting to identify timestamp values in your events. Specify what Splunk recognizes as a timestamp by setting a strptime() format in the TIME_FORMAT= key.

Note: If your event has more than one timestamp, set Splunk to recognize the correct timestamp with positional timestamp extraction.

Use $SPLUNK_HOME/etc/system/README/props.conf.example as an example, or create your own props.conf. Make any configuration changes to a copy of props.conf in $SPLUNK_HOME/etc/system/local/, or your own custom application directory in $SPLUNK_HOME/etc/apps/.

Configure any of the following attributes in props.conf to set Splunk's timestamp recognition. Refer to $SPLUNK_HOME/etc/system/README/props.conf.spec for full specification of the keys.

[<spec>]
DATETIME_CONFIG = <filename relative to $SPLUNK_HOME>
MAX_TIMESTAMP_LOOKAHEAD = <integer>
TIME_PREFIX = <regular expression>
TIME_FORMAT = <strptime-style format>
TZ = <posix timezone string>
MAX_DAYS_AGO = <integer>
MAX_DAYS_HENCE = <integer>

[<spec>]

• <spec> indicates what to apply timestamp extraction to. This can be one of the following:
  • <sourcetype>, the sourcetype of an event.
  • host::<host>, where <host> is the host of an event.
  • source::<source>, where <source> is the source of an event.
• If an event contains data that matches the value of <spec>, then the timestamp rules specified in the stanza apply to that event.
• Add additional stanzas to customize timestamp recognition for any type of event.

DATETIME_CONFIG = <filename relative to $SPLUNK_HOME>

• Specify a file to use to configure Splunk's timestamp processor (by default Splunk uses $SPLUNK_HOME/etc/datetime.xml).
• To use a custom datetime.xml, specify the correct path to your custom file in all keys that refer to datetime.xml.
• Set DATETIME_CONFIG = NONE to prevent the timestamp processor from running. When timestamp processing is off, Splunk does not look at the text of the event for the timestamp--it instead uses the event's "time of receipt"; in other words, the time the event is received via its input. For file-based inputs, this means that Splunk derives the event timestamp from the modification time of the input file.
• Set DATETIME_CONFIG = CURRENT to assign the current system time to each event as it's indexed.

TIME_PREFIX = <regular expression>

• When set, Splunk looks for a match for this regex in the event text before attempting to extract a timestamp. The timestamp algorithm only looks for a timestamp in the event text that follows the end of the first regex match.
• Ideally you should use a regular expression that points to the space exactly before your event's timestamp.
  • For example, if the timestamp follows the phrase abc123 in your events, you should set TIME_PREFIX to abc123.
• If the TIME_PREFIX cannot be found in the event text, timestamp extraction does not take place.
• Defaults to empty string.

MAX_TIMESTAMP_LOOKAHEAD = <integer>

• Specify how far (how many characters) into an event Splunk should look for a timestamp.
• This constraint to timestamp extraction is applied from the point of the TIME_PREFIX-set location.
  • For example, if TIME_PREFIX positions a location 11 characters into the event, and MAX_TIMESTAMP_LOOKAHEAD is set to 10, timestamp extraction will be constrained to characters 11 through 20.
• If set to 0 or -1, the length constraint for timestamp recognition is effectively disabled. This can have negative performance implications which scale with the length of input lines (or with event size when LINE_BREAKER is redefined for event splitting).
• Default is 150 characters.

TIME_FORMAT = <strptime-style format>

• Specifies a strptime() format string to extract the date.
• Unix strptime() is an industry standard for designating time formats. For more information, see the "Enhanced strptime() support" subtopic.
• This method of date extraction does not support in-event timezones.
• For best results, the <strptime-style format> should describe the day of the year and the time of day.
• TIME_FORMAT starts reading after the TIME_PREFIX.
• If <strptime-style format> contains an hour component, but no minute component, TIME_FORMAT ignores the hour component. It treats the format as an anomaly and considers the precision to be date-only.
• Default is empty.

TZ = <timezone identifier>

• Splunk's logic for determining a particular event's time zone is as follows:
  • If the event has a timezone in its raw text (such as UTC or -08:00, use that).
  • Otherwise, if TZ is set to a valid timezone string, use that. Specify a timezone setting using a value from the zoneinfo TZID database.
  • Otherwise, use the timezone of the system that is running splunkd.
• For more details and examples, see "Specify timezones of timestamps", in this manual.
• Defaults to empty.

MAX_DAYS_AGO = <integer>

• Specifies the maximum number of days in the past, from the current date, that an extracted date can be valid.
• For example, if MAX_DAYS_AGO = 10 then Splunk ignores dates older than 10 days from the current date.
• Default is 2000.
• Note: If you have data that is more than 2000 days old, increase this setting.

MAX_DAYS_HENCE = <integer>

• Specifies the maximum number of days in the future from the current date that an extracted date can be valid.
• For example, if MAX_DAYS_HENCE = 3, dates that are more than 3 days in the future are ignored.
• False positives are less likely with a tighter window.
• If your servers have the wrong date set or are in a timezone that is one day ahead, set this value to at least 3.
• Defaults to 2. This allows timestamp extractions that are up to a day in the future.

Enhanced strptime() support

Configure timestamp parsing in props.conf with the TIME_FORMAT= key. Splunk implements an enhanced version of Unix strptime() that supports additional formats (allowing for microsecond, millisecond, any time width format, and some additional time formats for compatibility). See the table below for a list of the additionally supported strptime() formats.

In previous versions, Splunk parsed timestamps using only the standard Linux strptime() conversion specifications. Now, in addition to standard Unix strptime() formats, Splunk's strptime() implementation supports recognition of the following date-time formats:

%N	For GNU date-time nanoseconds. Specify any sub-second parsing by providing the width: %3N = milliseconds, %6N = microseconds, %9N = nanoseconds.
%Q,%q	For milliseconds, microseconds for Apache Tomcat. %Q and %q can format any time resolution if the width is specified.
%I	For hours on a 12-hour clock format. If %I appears after %S or %s (like "%H:%M:%S.%l") it takes on the log4cpp meaning of milliseconds.
%+	For standard UNIX date format timestamps.
%v	For BSD and OSX standard date format.
%z, %::z, %:::z	GNU libc support.
%o	For AIX timestamp support (%o used as an alias for %Y).
%p	The locale's equivalent of AM or PM. (Note: there may be none.)

strptime() format expression examples

Below are some sample date formats with strptime() expressions that handle them.

1998-12-31	%Y-%m-%d
98-12-31	%y-%m-%d
1998 years, 312 days	%Y years, %j days
Jan 24, 2003	%b %d, %Y
January 24, 2003	%B %d, %Y
q|25 Feb '03 = 2003-02-25|	q|%d %b '%y = %Y-%m-%d|

Examples

Your data might contain an easily recognizable timestamp to extract such as:

...FOR: 04/24/07 PAGE 01...

The entry in props.conf is:

[host::foo]
TIME_PREFIX = FOR: 
TIME_FORMAT = %m/%d/%y

Your data might contain other information that Splunk parses as timestamps, for example:

...1989/12/31 16:00:00 ed May 23 15:40:21 2007...

Splunk extracts the date as Dec 31, 1989, which is not useful. In this case, configure props.conf to extract the correct timestamp from events from host::foo:

[host::foo]
TIME_PREFIX = \d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} \w+\s
TIME_FORMAT = %b %d %H:%M:%S %Y

This configuration assumes that all timestamps from host::foo are in the same format. Configure your props.conf stanza to be as granular as possible to avoid potential timestamping errors.

You can also configure Splunk's timestamp extraction processor to:

• Apply timezone offsets.
• Recognize European date format.
• Pull the correct timestamp from events with more than one timestamp.
• Improve indexing performance.

Finally, train Splunk to recognize new timestamp formats.

• Was this topic useful?
• Post a comment