How timestamp assignment works

Timestamp processing is a key step in event processing. Splunk software uses timestamps in the following ways:

• Correlating events by time
• Creating the timeline histogram in Splunk Web
• Setting time ranges for searches

Splunk software adds timestamps to events at index time. It assigns timestamp values automatically by using information that it finds in the raw event data. If there is no explicit timestamp in an event, Splunk software attempts to assign a timestamp value through other means. For some data, you might need to help Splunk software learn to recognize the timestamps. You do this by configuring timestamp extraction. To configure timestamp extraction, see the Configuring timestamp extraction section later in this topic.

Splunk software stores timestamp values in the _time field using Coordinated Universal Time (UTC) format.

If you have Splunk Cloud Platform and need to modify timestamp extraction, use a heavy forwarder to ingest the data, configure timestamp extraction and send it to the Splunk Cloud Platform instance. If you aren't using a heavy forwarder but have access to the Edge Processor solution, you can use Edge Processors to extract timestamps from event data and store them in the _time field if necessary. See Extract timestamps from event data using an Edge Processor in the Use Edge Processors manual.

For more information on event processing, see the Configure event processing chapter.

How Splunk software assigns timestamps

Splunk software uses the following precedence rules to assign timestamps to events:

1. The software looks for a time or date in the event itself using an explicit TIME_FORMAT, if provided. You configure the TIME_FORMAT attribute in the props.conf file.
2. If no TIME_FORMAT is configured for the data, Splunk software attempts to automatically identify a time or date in the event itself. It uses the source type of the event, which includes TIME_FORMAT information, to try to find the timestamp.
3. If an event has a time and date, but not a year, Splunk software determines the year and builds the timestamp from that date. See How Splunk software determines timestamps with no year.
4. If no events in the source have a date, Splunk software tries to find a date in the source name or file name. The events must have a time, even if they don't have a date.
5. For file sources, if no date can be identified in the file name, Splunk software uses the file modification time.
6. The software attempts to automatically identify a time or date in the event itself using the advanced timestamp recognition configured in datetime.xml.
7. As a last resort, Splunk software sets the timestamp to the current system time when indexing each event.

Splunk software can extract only dates from a source, not times. If you need to extract a time from a source, use a transform configuration. See Create custom fields at index time.

How Splunk software determines timestamps with no year

If Splunk software discovers a timestamp within an event that does not have a year element, it uses the following logic to determine the year:

1. It identifies the current date by using either the date of the event it last parsed or the current clock time.
2. It then uses the year from that date as a base and runs the year through several tests:
   1. If the date in the new event is December 31 and the current date is January 1, it decrements the base year.
   2. If the date in the new event is January 1 and the current date is December 31, it increments the base year.
   3. If the date in the new event is February 29, it determines if the current year is a leap year.
   4. If the current year is a leap year, it uses that year as the base year. If it is not, it uses the previous leap year.
3. If none of the previous tests results in a successful base year determination, the software uses the following procedure to determine the year:
   1. It determines the day of the year of the new event by calculating the number of days from January 1.
   2. If the date information of the previous event is available, and the day of the year of that event is greater than the day of the year of the new event plus 4 days, then it increments the base year.
   3. If the date information of the previous event is not available, and the day of the year of the new event is greater than the current day of the year plus 2 days, then it decrements the base year.
4. The software then assigns the base year to the timestamp for the event. The timestamp must still pass the time range check for the timestamp to be valid.

Example 1

If Splunk software encounters 26 Jun in a new event on May 26, 2017, and it is not able to determine the year in the previous events, it follows these steps to determine the event timestamp:

1. Since it was not able to determine the year in the previous event, it sets a base year of 2017 as that is the year of the current date.
2. The software checks if the date is December 31 or January 1. The December 31 and January 1 tests fail, so the base year remains 2017.
3. The software checks if the date occurs in a leap year. The leap year test fails, as the date is not February 29. The base year remains 2017.
4. Splunk software calculates the day of the year for June 26 as Day 177.
5. Since the software can't determine the year in the previous event, it adds 2 to this number to arrive at 179.
6. It then compares 179 to the day of the year of the current date, May 26, 2017, which is Day 147.
7. Since 179 is greater than 147, the software decrements the year from 2017 to 2016.
8. The software then builds the new timestamp: 26 Jun 2016.
9. If the new timestamp falls within the time range that has been set, the software adds the timestamp to the event.

Example 2

If Splunk software encounters 10 Apr in a new event on May 26, 2017, and it determined the year 2017 in previous events, it follows these steps to determine the event timestamp:

1. Since the software determined the year in the previous event, it sets 2017 as the base year.
2. The software checks if the date is December 31 or January 1. The December 31 and January 1 tests fail, so the base year remains 2017.
3. The software checks if the date occurs in a leap year. The leap year test fails, as the date is not February 29. The base year remains 2017.
4. Splunk software calculates the day of the year for April 10 as Day 100.
5. Since the year information in the previous event is available, it adds 4 to this number to arrive at 104.
6. It then compares 104 to the day of the year of the current date, May 26, 2017, which is Day 147.
7. Since 104 is less than 147, the software increments the year from 2017 to 2018.
8. The software then builds the new timestamp: 10 Apr 2018.
9. By default, this new timestamp is not legal, since it falls outside the default MAX_DAYS_HENCE setting, which limits valid timestamps to 2 days into the future. The software uses the current date of 26 May 2017 as the timestamp and applies that timestamp to the event.

Configuring timestamp extraction

Most events do not require special timestamp handling. Splunk software automatically recognizes and extracts their timestamps. For some sources and distributed deployments, you might need to configure how timestamps are extracted, so that they format properly.

There are two ways to configure timestamp extraction:

• Use the Set Source Type page in Splunk Web to interactively adjust timestamps on sample data. Once you are happy with the results, save the changes to a new source type and then apply that source type to your data inputs. See Assign the correct source types to your data.

• Edit the props.conf configuration file directly. See Configure timestamp recognition.

You can also configure timestamp extraction for the following purposes:

• Apply time zone offsets. See Specify time zones for timestamps.
• Extract the correct timestamp from events with more than one timestamp. See Configure timestamp assignment for events with multiple timestamps.
• Improve indexing performance. See Tune timestamp recognition for better indexing performance.

Considerations when adding data from new inputs

If you index data from a new input and then discover that you need to adjust the timestamp extraction process, you must reindex that data after you make the configuration changes.

Consider previewing your data to prevent the need to reindex your data. Alternatively, you can test new data inputs on a separate index on your production Splunk Cloud Platform environment. If you use Splunk Enterprise, you can create a separate test Splunk Enterprise deployment before adding data to your production instance. Creating a separate test deployment allows you to delete and reindex until you get the results you expect.