Splunk® Enterprise

Release Notes

Splunk Enterprise version 7.2 is no longer supported as of April 30, 2021. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see How to upgrade Splunk Enterprise.
This documentation does not apply to the most recent version of Splunk® Enterprise. For documentation on the most recent version, go to the latest release.

Timestamp recognition of dates with two-digit years fails beginning January 1, 2020

Review this Known Issue topic for the latest updates on this issue.
The last significant update was at 22:10 -0800 Wed 1 Jan 2020:

- Updated validation instructions to include year-2021 timestamps

Problem

Beginning on January 1, 2020, un-patched Splunk platform instances will be unable to recognize timestamps from events where the date contains a two-digit year. This means data that meets this criteria will be indexed with incorrect timestamps.

Beginning on September 13, 2020 at 12:26:39 PM Coordinated Universal Time (UTC), un-patched Splunk platform instances will be unable to recognize timestamps from events with dates that are based on Unix time, due to incorrect parsing of timestamp data.

Impact

This issue affects all un-patched Splunk platform instance types, on any operating system:

  • Splunk Cloud
  • Splunk Light
  • Splunk Enterprise
    • Indexers, clustered or not
    • Heavy forwarders
    • Search heads, clustered or not
    • Search head deployers
    • Deployment servers
    • Cluster manager (or master)
    • License manager (or master)
  • Splunk universal forwarders, under the following known conditions:
    • When they have been configured to process structured data, such as CSV, XML, and JSON files, using the INDEXED_EXTRACTIONS setting in props.conf
    • When they have been configured to process data locally, using the force_local_processing setting in props.conf

The issue appears when you have configured the input source to automatically determine timestamps, and can result in one or more of the following problems:

  • Incorrect timestamping of incoming data
  • Incorrect rollover of data buckets due to the incorrect timestamping
  • Incorrect retention of data with incorrect timestamps
  • Incorrect search results due to data ingested with incorrect timestamps

There is no method to correct the timestamps after the Splunk platform has ingested the data when the problem starts. If you ingest data with an un-patched Splunk platform instance beginning on January 1, 2020, you must patch the instance and re-ingest that data for its timestamps to be correct.

Cause

The Splunk platform input processor uses a file called datetime.xml to help the processor correctly determine timestamps based on incoming data. The file uses regular expressions to extract many different types of dates and timestamps from incoming data.

On un-patched Splunk platform instances, the file supports the extraction of two-digit years of "19", that is, up to December 31, 2019. Beginning on January 1, 2020, these un-patched instances will mistakenly treat incoming data as having an invalid timestamp year, and could either add timestamps using the current year, or misinterpret the date incorrectly and add a timestamp with the misinterpreted date.

Solutions

Splunk Cloud customers will receive the fix on their Splunk Cloud instances automatically. If you are a Splunk Cloud customer, your support representative will advise you when the upgrade will take place on your Splunk Cloud instance. As this is a critical update, there is no option to defer it.

While Splunk Cloud instances will receive automatic updates, you must perform one of these solutions on any self-deployed instances, such as heavy and universal forwarders that send data to your Splunk Cloud instance. You can perform one of these solutions at any time before or after your Splunk Cloud instance receives the fix. Contact your support representative if you have any questions.

There are four solutions to this problem for on-premises customers. You only need to perform one of the solutions to fix the problem:

  • Download and deploy an app to temporarily replace the defective datetime.xml with the fixed file
  • Download an updated version of datetime.xml and apply it to each of your Splunk platform instances
  • Upgrade Splunk platform instances to a version with an updated version of datetime.xml
  • Make modifications to the existing datetime.xml file on your Splunk platform instances

Download and deploy an app that temporarily replaces the defective datetime.xml file with the fixed file

Splunk has released a Splunk app that temporarily replaces the defective datetime.xml file with the fixed file. This option is the preferred path for customers who have large deployments including many universal forwarders and indexer and search head clusters.

You do not need to stop the Splunk platform before you deploy the apps, but you must restart each instance that receives the apps.

To ensure that you have no problems with universal forwarders and timestamp recognition, deploy this fix on all universal forwarders, even those that do not specifically suffer from this problem.

This solution is temporary only. Do not run it long-term. Carefully read the instructions that come with the app download and make sure to deploy the correct app to the correct Splunk platform instance type. Do not deploy both apps to a single instance. The preferred method to address this problem is to upgrade to a version that has the fixed datetime.xml file. After you complete upgrades, remove the apps immediately.

  1. Download the datetime.xml fix app archive (MD5 hash: 2e6b7520fa1379d72ac80ca21a54d45a) from splunk.com.
  2. Unarchive the .tgz file to a location that is accessible from all of your Splunk platform instances.
  3. Open the README file and follow the instructions to deploy one of the apps to each Splunk platform instance.

Download an updated version of datetime.xml and apply it to each of your Splunk platform instances

Splunk is providing an updated version of the datetime.xml file for download. This option is the preferred path for customers who cannot upgrade right away to a version of the Splunk platform with the fixed file, or who run an unsupported version that is lower than 6.6.x.

After you download the file, you must apply it directly over the existing datetime.xml file using the following procedure. You must apply the updated file to all affected on-premises Splunk platform instances prior to January 1, 2020, or you will experience timestamp recognition problems from that point forward.

  1. Download the datetime.zip timestamp recognition ZIP file (MD5 hash: 00dfc319e89001fa16d6725dbf042234) from splunk.com.
  2. Unarchive the ZIP file to a location that is accessible from all of your Splunk platform instances.
  3. On each Splunk platform instance, do the following:
    1. Using your operating system file management utilities, copy the updated datetime.xml from the location where you downloaded it to the $SPLUNK_HOME/etc directory on the Splunk platform instance. Ensure that the updated file overwrites the existing file.
    2. Confirm that the new datetime.xml has been written to the $SPLUNK_HOME/etc directory.
    3. Restart the Splunk platform. Your Splunk platform instance is now patched.

Upgrade Splunk platform instances to a version with an updated version of datetime.xml

Splunk has released updated versions of the Splunk platform that contain an updated datetime.xml.

To fix the problem, download and install a version of the Splunk platform that contains the fixed datetime.xml. The following table lists the minor versions that include the fixed file for each major version that Splunk currently supports, and provides links on how to upgrade Splunk Enterprise and Splunk Light. Apply the fixed version directly over the existing Splunk platform instance to patch it.

Major version Minor version with patched file Released? Splunk platform upgrade instructions
6.6 6.6.12.1 Yes Splunk Enterprise
7.0 7.0.13.1
(versions 7.0.12 and 7.0.13 are Splunk Cloud-only releases) 		Yes Splunk Enterprise
7.1 7.1.10 Yes Splunk Enterprise
7.2 7.2.9.1 Yes Splunk Enterprise
7.3 7.3.3 Yes Splunk Enterprise
8.0 8.0.1 Yes Splunk Enterprise

Make modifications to the existing datetime.xml file on your Splunk platform instances

If you feel comfortable with making changes to the datetime.xml file directly, you can do so without upgrading Splunk Enterprise or Splunk Light. If you are a Splunk Cloud customer that uses forwarders to send data to your Splunk Cloud instance, you can use this procedure to update those instances.

Use caution when making changes to the $SPLUNK_HOME/etc/datetime.xml file. Outside of fixing this specific issue, never make changes to this file. Typos and incompatible characters in the file can cause direct, lasting negative effects on timestamp recognition and data ingestion. If you are not comfortable with making these changes, consider one of the previous options for fixing this problem, or contact Splunk Support for assistance.

You must complete these changes on all affected Splunk platform instances by January 1, 2020, or you will experience timestamp recognition problems from that point forward.

  1. Using either a shell prompt or your operating system file management utilities, go to the $SPLUNK_HOME/etc directory in your Splunk Enterprise installation.
  2. Open datetime.xml for editing with a text editor.
  3. Search for and replace the following strings, according to this table:
    Search for this string Replace with this string 
    <text><![CDATA[(20\d\d|19\d\d|[901]\d(?!\d))]]></text>
    		 
    <text><![CDATA[(20\d\d|19\d\d|[9012]\d(?!\d))]]></text>
    <text><![CDATA[(?:^|source::).*?(?<!\d|\d\.|-)(?:20)?([901]\d)(0\d|1[012])([012]\d|3[01])(?!\d|-| {2,})]]></text>
    		 
    <text><![CDATA[(?:^|source::).*?(?<!\d|\d\.|-)(?:20)?([9012]\d)(0\d|1[012])([012]\d|3[01])(?!\d|-| {2,})]]></text>
    <text><![CDATA[(?:^|source::).*?(?<!\d|\d\.)(0\d|1[012])([012]\d|3[01])(?:20)?([901]\d)(?!\d| {2,})]]></text>
    		 
    <text><![CDATA[(?:^|source::).*?(?<!\d|\d\.)(0\d|1[012])([012]\d|3[01])(?:20)?([9012]\d)(?!\d| {2,})]]></text>
    <text><![CDATA[((?<=^|[\s#,"=\(\[\|\{])(?:1[012345]|9)\d{8}|^@[\da-fA-F]{16,24})(?:\.?(\d{1,6}))?(?![\d\(])]]></text>
    		 
    <text><![CDATA[((?<=^|[\s#,"=\(\[\|\{])(?:1[0123456]|9)\d{8}|^@[\da-fA-F]{16,24})(?:\.?(\d{1,6}))?(?![\d\(])]]></text>
  4. Save the file and close it. If the editor asks if you want to overwrite the file, answer yes.
  5. Confirm that the datetime.xml file has been updated.
  6. Restart the Splunk platform. Your Splunk platform instance is now patched.

Solutions for customers with large installations

If you have a large on-premises deployment of many Splunk platform instances, contact Professional Services for assistance. If you feel comfortable with deploying Splunk apps, you can also use the app deployment solution, as described in the "Solutions" section.

Validate timestamp extraction after an update

After you have applied one of the solutions described in this topic, you can use the following procedure to validate that timestamp extraction works as expected.

As part of this process, you must temporarily configure the input processor to accept timestamps that occur in the future. This procedure includes instructions for configuring the MAX_DAYS_HENCE setting in the default stanza of the props.conf configuration file. After you have successfully validated timestamp extraction, you can remove the MAX_DAYS_HENCE setting from props.conf to restore default timestamp extraction behavior.

  1. Paste the following text into a text editor: 
    date,message
19-12-31 23:58:44,Test Message  - datetime.xml testing - override - puppet managed forced restart
20-01-02 23:58:54,Test Message  - datetime.xml testing - override - puppet managed forced restart
21-01-01 23:59:04, Test Message - datetime.xml testing - override - puppet managed forced restart
  2. Save the text as a text file, for example, test_file.csv, to a place that is accessible from all of your Splunk platform instances.
  3. On the Splunk platform instance that you want to validate, adjust the MAX_DAYS_HENCE setting for the [default] stanza in the $SPLUNK_HOME/etc/system/local/props.conf configuration file. 
    [default]
MAX_DAYS_HENCE = 400
  4. Restart the Splunk platform.
  5. Using the Splunk CLI, add the text file you saved earlier as a oneshot monitor to the Splunk platform instance that you want to validate. 
    $SPLUNK_HOME/bin/splunk add oneshot -source test_file.csv -sourcetype csv -index main
  6. Perform a search on the text in Step 1. In the search results, the "Time" column should display a timestamp for the January 2, 2020 event with the correct two-digit year of 2020, and a timestamp for the January 1, 2021 event with the correct two-digit year of 2021.

File integrity errors after an update to the file

If you perform either of the file update or the direct file modification solutions, upon restart, Splunk Web will display a message like the following: 

File Integrity checks found 1 files that did not match the system-provided manifest. Review the list of problems reported by the InstalledFileHashChecker in splunkd.log File Integrity Check View ; potentially restore files from installation media, change practices to avoid changing files, or work with support to identify the problem.

You can safely disregard this message in these two scenarios only. After you upgrade to a version that has the fixed datetime.xml, this message should no longer appear.

More information

Last modified on 03 August, 2022
Field alias behavior change   Fixed issues

This documentation applies to the following versions of Splunk® Enterprise: 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.0.13, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.1.10, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8, 7.3.9, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.0.5, 8.0.6, 8.0.7, 8.0.8, 8.0.9, 8.0.10

Acrobat logo Download manual
Acrobat logo Download this page

Was this topic useful?







You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters