Splunk® Enterprise

Getting Data In

Download manual as PDF

Download topic as PDF

Tune timestamp recognition for better indexing performance

To speed up indexing, you can use props.conf to adjust how far ahead into events the Splunk timestamp processor looks, or even turn off the timestamp processor altogether.

For general information on editing props.conf for timestamps, see Configure timestamp recognition. If you have Splunk Enterprise and need to modify timestamp extraction, perform the configuration on your indexer machines or, if forwarding data, use heavy forwarders and perform the configuration on the machines where the heavy forwarders run. If you have Splunk Cloud and need to modify timestamp extraction, use heavy forwarder and perform the configuration on the machines where the heavy forwarders run.

Adjust timestamp lookahead

Timestamp lookahead determines how far (how many characters) into an event the timestamp processor looks for a timestamp. Adjust how far the timestamp processor looks by setting the MAX_TIMESTAMP_LOOKAHEAD attribute.

The default number of characters that the timestamp processor looks into an event is 128. You can set MAX_TIMESTAMP_LOOKAHEAD to a lower value to speed up indexing. You should particularly do this if the timestamps always occur in the first part of the event.

Example:

Look for timetamps in the first 20 characters of events coming from source foo.

[source::foo]
MAX_TIMESTAMP_LOOKAHEAD = 20
...

Disable timestamp processor

You can turn off the timestamp processor entirely to improve indexing performance. Turn off timestamp processing for events matching a specified host, source, or sourcetype by setting the DATETIME_CONFIG attribute to NONE. When DATETIME_CONFIG=NONE, Splunk software does not look at the text of the event for the timestamp. Instead, it uses the event "time of receipt"; in other words, the time the event is received from its input. For file-based inputs (such as monitor) this means that the timestamp comes from the modification time of the input file.

You can also increase indexing performance by setting DATETIME_CONFIG to CURRENT, which assigns the current system time to each event at the time of indexing.

Example:

This example turns off timestamp extraction for events that come from the source foo.

[source::foo]
DATETIME_CONFIG = NONE
...

Note: Both CURRENT and NONE disable timestamp identification, so the default event boundary detection (BREAK_ONLY_BEFORE_DATE = true) might not work as you expect. When you use these settings, specify SHOULD_LINEMERGE or the BREAK_ONLY_* and MUST_BREAK_* settings to control event merging.

PREVIOUS
Specify time zones for timestamps
  NEXT
About indexed field extraction

This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.1, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.7, 6.1.8, 6.1.9, 6.1.10, 6.1.11, 6.1.12, 6.1.13, 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.2.14, 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.4.9, 6.4.10, 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.5.6, 6.5.7, 6.5.8, 6.5.9, 6.6.0, 6.6.1, 6.6.2, 6.6.3, 6.6.4, 6.6.5, 6.6.6, 6.6.7, 6.6.8, 6.6.9, 6.6.10, 6.6.11, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.2.0, 7.2.1


Comments

Typo (misspelling) in the example description: "timetamps" (sic).

More importantly, please consider adding the following important qualification—reworded, if you like—to "Adjust timestamp lookahead":

> This constraint is applied starting from the location positioned by TIME_PREFIX.

Graham Hannington
November 2, 2018

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters