Configure timestamp recognition

Most events do not require special timestamp handling. The Splunk platform recognizes and extracts timestamps correctly. However, with some sources and distributed deployments, you might need to configure how the Splunk platform extracts timestamps to ensure that the timestamps have the proper format.

You can configure timestamp extraction in these ways:

• Use the Set Source Type page in Splunk Web to interactively adjust timestamps on sample data. Once you're happy with the results, you can save the changes to a new source type and then apply that source type to your data inputs. This option is available on Splunk Cloud Platform only if you upload a file directly to the instance. See Assign the correct source types to your data.
• Use the props.conf configuration file. This option is available on both Splunk Cloud Platform and Splunk Enterprise under the following conditions:
  • If you have Splunk Cloud Platform and need to modify timestamp extraction, use a heavy forwarder to ingest the data and send it to the Splunk Cloud Platform instance. The heavy forwarder lets you specify configurations that extract the timestamps. You cannot make these configurations with the universal forwarder or with Splunk Cloud Platform directly.
  • On Splunk Enterprise instances, if you need to modify timestamp extraction, specify the configuration on the indexers. In cases where you have to forward data, you must configure a heavy forwarder to handle these changes.

Since Splunk uses 32-bit epoch time, events after 2038 cannot be indexed.

The timestamp processor

On a Splunk Enterprise instance, you can find the timestamp processor at $SPLUNK_HOME/etc/datetime.xml by default. You do not need to edit this file normally, unless you work with unusual custom timestamps. You cannot edit this file on a Splunk Cloud Platform instance because you do not have access to the Splunk Cloud Platform file system.

If you need to configure timestamp recognition, you can make changes by editing timestamp settings in the props.conf configuration file, as described in this topic.

If you have a custom timestamp that can't be handled by configuring the props.conf file, substitute your own timestamp processor with the DATETIME_CONFIG setting. This setting specifies the file that the Splunk platform uses for timestamp processing.

Edit timestamp properties in the props.conf configuration file

You can edit timestamp properties on a heavy forwarder to ensure that Splunk Cloud Platform sees and uses the proper timestamps, or you can edit them directly on a Splunk Enterprise instance.

To configure how the Splunk platform recognizes timestamps, edit the props.conf configuration file. With this file, you can control how the Splunk platform processes the timestamps that it sees in incoming events. See the props.conf specification file in the Admin Manual.

The props.conf file has several settings for timestamp processing. For example, you can use the TIME_FORMAT setting to specify a format for the timestamp that is based on the strptime() string to time-structure conversion function.

You can also configure where to look in an event for a timestamp, what time zone to use for events, or how to deal with timestamps of varying currency.

Follow these steps to configure timestamp recognition:

1. For Splunk Cloud Platform instances or on Splunk Enterprise instances that receive data from forwarders, install a new Splunk Enterprise instance and configure it as a heavy forwarder.
2. If you use Splunk Cloud Platform, install the Splunk Cloud Platform universal forwarder credentials package on the forwarder.
3. On the forwarder, use a text editor to edit the props.conf file in the $SPLUNK_HOME/etc/system/local/ directory or in your own custom application directory in $SPLUNK_HOME/etc/apps/.
4. Specify timestamp recognition settings that suit your needs, fit your incoming data, and comply with the syntax. See Syntax overview in this topic for more information.
5. Save the changes and restart the forwarder.
   The forwarder begins processing timestamps based on the new configuration that you provided.

If you haven't worked with configuration files before, see About configuration files in the Admin Manual.

Syntax overview

When you configure timestamp recognition, it is very important that your configuration file and its settings follow the appropriate syntax. Syntax errors can result in the incorrect timestamping and ingestion of the events.

See the following overview of the syntax:

[<spec>]
DATETIME_CONFIG = <filename relative to the $SPLUNK_HOME directory>
TIME_PREFIX = <regular expression>
MAX_TIMESTAMP_LOOKAHEAD = <integer>
TIME_FORMAT = <strptime-style format>
TZ = <POSIX time zone string>
MAX_DAYS_AGO = <integer>
MAX_DAYS_HENCE = <integer>
MAX_DIFF_SECS_AGO = <integer>
MAX_DIFF_SECS_HENCE = <integer>

In this syntax, <spec> can have any of the following meanings:

• <sourcetype>, the source type of an event.
• host::<host>, where <host> is the host value for an event.
• source::<source>, where <source> is the source value for an event.

If an event contains data that matches the value of <spec>, then the timestamp rules specified in the stanza apply to that event. You can have multiple stanzas to handle different <spec> values.

DATETIME_CONFIG values of NONE or CURRENT must be listed in all caps. Otherwise, a parsing error may occur that could cause data loss.

Timestamp validity settings and their impact on events

By default, indexes all events unless you specifically filter out events through other means.

When you set the MAX_DAYS_AGO, MAX_DAYS_HENCE, MAX_DIFF_SECS_AGO, or MAX_DIFF_SECS_HENCE settings in a stanza, and event timestamps fall outside of the parameters of the settings, then uses the following algorithm to determine the timestamp:

• The software uses the timestamp of the previous event to assign the timestamp of the current event.
• If the timestamp of the previous event can't be determined, then the software uses the current index time to assign a timestamp to the event.

does not drop events if they fall outside of the parameters of these settings.

Timestamp settings

The following timestamp configuration settings are available:

Setting	Description	Default
DATETIME_CONFIG = <filename relative to $SPLUNK_HOME>	Specify a file to use to configure the Splunk timestamp processor. Normally, you don't need to create your own timestamp processor file or modify the default datetime.xml file. The other props.conf settings are usually sufficient. However, if your data has a custom timestamp format, you might need your own version of the datetime.xml file. Set DATETIME_CONFIG = NONE to prevent the timestamp processor from running. When timestamp processing is off, does not look at the text of the event for the timestamp and instead uses the event time of receipt, the time the event arrives through its input. For file-based inputs, the event timestamp is taken from from the modification time of the input file. Set DATETIME_CONFIG = CURRENT to assign the current system time to each event as indexes it. Both CURRENT and NONE as values for DATETIME_CONFIG explicitly disable timestamp identification, so the default event boundary detection (BREAK_ONLY_BEFORE_DATE = true) might not work as desired. When using these settings, use SHOULD_LINEMERGE, BREAK_ONLY_* , MUST_BREAK_*, or both settings to control event merging.	$SPLUNK_HOME/etc/datetime.xml
TIME_PREFIX = <regular expression>	When set, uses the specified regular expression to look for a match before trying to extract a timestamp. The timestamp algorithm looks only for a timestamp in the event text that follows the end of the first regular expression match. Use a regular expression that points exactly before your event timestamp. For example, if the timestamp follows the phrase abc123 in your events, set TIME_PREFIX to abc123. If the TIME_PREFIX isn't found in the event text, the timestamp isn't extracted.	empty string
MAX_TIMESTAMP_LOOKAHEAD = <integer>	Specify how far, or how many characters into, an event that the Splunk platform looks for a timestamp. This constraint is applied starting from the location positioned by TIME_PREFIX. For example, if TIME_PREFIX positions a location 11 characters into the event and MAX_TIMESTAMP_LOOKAHEAD is set to 10, timestamp extraction will be constrained to characters 11 to 20. If set to 0 or -1, the length constraint for timestamp recognition is disabled. This can have negative performance implications, which scale with the length of input lines or with event size when LINE_BREAKER is redefined for event splitting.	128 characters
TIME_FORMAT = <strptime-style format>	Specifies a strptime() format string to extract the timestamp. strptime() is a Unix standard for designating time formats. For more information, see the section Enhanced strptime() support. TIME_FORMAT starts reading after the TIME_PREFIX or directly at the start of the event, if there is no TIME_PREFIX setting. If you use a TIME_PREFIX, it must match up to and including the character before the timestamp begins. If you don't set TIME_PREFIX but you do set TIME_FORMAT, the timestamp must appear at the very start of each event. If not, strptime will fail and Splunk displays a "Failed to parse timestamp" warning message. Splunk will also write the message onto splunkd.log, but NOT tied to each specific event. For best results, use <strptime-style format> to describe the day of the year and the time of day. If <strptime-style format> contains an hour component, but no minute component, TIME_FORMAT ignores the hour component. It treats the format as an anomaly and considers the precision to be date-only.	empty string
TZ = <timezone_identifier>	The time zone of an event is determined as follows: If the event has a time zone in its raw text such as UTC or -08:00, use that. Otherwise, if TZ is set to a valid time zone string, use that. Specify a time zone setting using a value from the TZ database of time zones. See [1] on Wikipedia. If an event that arrives at an indexer originates from a forwarder and both indexer and forwarder are version 6.0 or higher, then use the time zone that the forwarder provides. Otherwise, use the time zone of the system that runs splunkd. For more details and examples, see Specify time zones of timestamps.	empty string
TZ_ALIAS = <key=value>[,<key=value>]...	Provides admin-level control over how time zone strings extracted from events are interpreted. For example, EST can mean Eastern (US) Standard Time or Eastern (Australian) Standard Time. There are many other three letter time zone acronyms with multiple expansions. There is no requirement to use TZ_ALIAS if the default mappings for these values work as expected. For example, EST maps to the Eastern US by default. Has no effect on the TZ value. It affects only time zone strings from event text, either from any configured TIME_FORMAT or from pattern-based guess fallback. The setting is a list of key=value pairs, separated by commas. The key is matched against the text of the time zone specifier of the event, and the value is the time zone specifier to use when mapping the timestamp to UTC/GMT. The value is another TZ specifier that expresses the desired offset. For example: TZ_ALIAS = EST=GMT+10:00 See the props.conf specification file in the Admin Manual for more examples.	not set
MAX_DAYS_AGO = <integer>	Specifies the maximum number of days in the past, from the current date, that an extracted date is valid. For example, if MAX_DAYS_AGO = 10, the Splunk platform ignores dates older than 10 days from the current date and instead either uses the timestamp of the previous event or the current index time of the event if it can't determine a timestamp in the previous event. The maximum number of days you can set in the past is 10951.	2000 days If you have data that is more than 2000 days old, increase this setting.
MAX_DAYS_HENCE = <integer>	Specifies the maximum number of days in the future from the current date that an extracted date is valid. For example, if MAX_DAYS_HENCE = 3, the Splunk platform ignores dates that are more than 3 days in the future and instead either uses the timestamp of the previous event or uses the current index time of the event if it can't determine a timestamp from the previous event. False positives are less likely with a tighter window. Change this setting with caution. If your servers have the wrong date set or are in a time zone that is one day ahead, set this value to at least 3. This allows timestamp extractions that are up to a day in the future. The maximum number of days you can set is 10950.	2 days
MAX_DIFF_SECS_AGO = <integer>	If the event timestamp is more than <integer> seconds before the previous timestamp, the Splunk platform accepts it only if it has the same time format as the majority of timestamps from the source. If your timestamps are wildly out of order, consider increasing this value. The maximum number of seconds you can set is 2147483646.	3600 seconds (1 hour)
MAX_DIFF_SECS_HENCE = <integer>	If the event timestamp is more than <integer> seconds after the previous timestamp, the Splunk platform accepts it only if it has the same time format as the majority of timestamps from the source. If your timestamps are wildly out of order, or if you have logs that are written less than once a week, consider increasing this value. The maximum number of seconds you can set is 2147483646.	604800 seconds (1 week)

Enhanced strptime() support

Use the TIME_FORMAT setting in the props.conf file to configure timestamp parsing. This setting takes a strptime() format string, which it uses to extract the timestamp.

The Splunk platform implements an enhanced version of Unix strptime() that supports additional formats, allowing for microsecond, millisecond, any time width format, and some additional time formats for compatibility. For the rest of the supported strptime() variables, see Date and time format variables in the Search Reference manual.

The following additional formats are included through the enhanced version:

Format	Description
%N	For GNU date-time nanoseconds. Specify any sub-second parsing by providing the width: %3N = milliseconds, %6N = microseconds, %9N = nanoseconds. Sub-second parsing depends on the setting ADD_EXTRA_TIME_FIELDS. See props.conf in the Admin Manual.
%Q,%q	For milliseconds, microseconds for Apache Tomcat. %Q and %q can format any time resolution if the width is specified.
%I	For hours on a 12-hour clock format. If %I appears after %S or %s like %H:%M:%S.%l, it takes on the log4cpp meaning of milliseconds.
%+	For standard Unix date format timestamps.
%v	For BSD and OSX standard date format.
%z, %:z, %::z	The time zone offset designator in International Organization for Standardization (ISO) 8601 format. For example, -0800 for PST, +0000 for GMT, or nothing if the time zone can't be determined. Use %:z if the timestamp offset contains hours and minutes, like -08:00. Use %::z if the timestamp offset contains hours, minutes, and seconds, like -08:00:00.
%o	For AIX timestamp support. %o is used as an alias for %Y.
%p	The locale equivalent of AM or PM. There might be no equivalent.
%s	Epoch (10 digits).

A strptime() expression that ends with a literal dot and subsecond specifier such as %Q, %q, %N treats the terminal dot and conversion specifier as optional. If the .subseconds portion is absent from the text, the timestamp is still extracted.

strptime() format expression examples

Here are some sample date formats, with the strptime() expressions that handle them:

Date format	Expression
1998-12-31	%Y-%m-%d
98-12-31	%y-%m-%d
1998 years, 312 days	%Y years, %j days
Jan 24, 2003	%b %d, %Y
January 24, 2003	%B %d, %Y
1397477611.862	%s.%3N

The Splunk platform doesn't recognize non-English month names in timestamps. If you have an app that writes non-English month names to log files, reconfigure the app to use numerical months, if possible.

Examples

Your data might contain an easily recognizable timestamp, like the following example:

...FOR: 04/24/07 PAGE 01...

To extract that timestamp, add this stanza in the props.conf file:

[host::foo]
TIME_PREFIX = FOR: 
TIME_FORMAT = %m/%d/%y

You might also have a timestamp that includes time zone information:

…Valid_Until=Thu Dec 31 17:59:59 GMT-06:00 2020

To extract the timestamp, add this stanza to the props.conf file:

[host::bar]
TIME_PREFIX = Valid_Until=
TIME_FORMAT = %a %b %d %H:%M:%S %Z%:z %Y

Your data might contain other information that is parsed as timestamps. Take the following example:

...1989/12/31 16:00:00 Wed May 23 15:40:21 2007...

The Splunk platform extracts the date as Dec 31, 1989, which is not useful. In this case, configure the props.conf file to extract the correct timestamp from events from host::foo:

[host::foo]
TIME_PREFIX = \d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} \w+\s
TIME_FORMAT = %b %d %H:%M:%S %Y

This configuration assumes that all timestamps from host::foo are in the same format. Configure your props.conf stanza to be as granular as possible to avoid potential timestamping errors.

For more information on extracting the correct timestamp from events containing multiple timestamps, see Configure timestamp assignment for events with multiple timestamps.

Configure timestamps for specific needs

You can use the settings described in this topic to configure the timestamp extraction processor for some specialized purposes. For example, you can configure timestamps for the following needs:

• Apply time zone offsets. See Specify time zones for timestamps.
• Pull the correct timestamp from events with more than one timestamp. See Configure timestamp assignment for events with multiple timestamps.
• Improve indexing performance. See Tune timestamp recognition for better indexing performance.

Configure how timestamps appear in search results

You can use your browser locale setting to configure how Splunk Web displays timestamps in search results. For information on setting the browser locale, see Configure user language and locale in the Admin Manual.

Reconfigure how timestamps appear in raw data

Even though the Splunk platform uses the browser locale to configure how timestamps appear in search results, the raw data still remains in its original format. You might want to change this so that the data format is standardized in both raw data and search results. Do this with the props.conf and transforms.conf files.

For example, assume the timestamp data in the raw event looks like this:

06/07/2011 10:26:11 PM

However, you want it to correspond with how it appears in search results:

07/06/2011 10:26:11 PM

You can use the props.conf and transforms.conf files to transform the timestamp in the raw event.

In the transforms.conf file, add this stanza:

[resortdate]
REGEX = ^(\d{2})\/(\d{2})\/(\d{4})\s([^/]+)
FORMAT = $2/$1/$3 $4
DEST_KEY = _raw 

In props.conf, add this stanza, where <spec> qualifies your data:

[<spec>]
TRANSFORMS-sortdate = resortdate