Splunk® Enterprise

Release Notes

Download manual as PDF

Download topic as PDF

Timestamp recognition of dates with two-digit years fails beginning January 1, 2020

Review this Known Issue topic for the latest updates on this issue.
The last significant update was at 15:00 -0800 Thu 5 Dec 2019

Problem

Beginning on January 1, 2020, un-patched Splunk platform instances will be unable to recognize timestamps from events where the date contains a two-digit year. This means data that meets this criteria will be indexed with incorrect timestamps.

Beginning on September 13, 2020 at 12:26:39 PM Coordinated Universal Time (UTC), un-patched Splunk platform instances will be unable to recognize timestamps from events with dates that are based on Unix time, due to incorrect parsing of timestamp data.

Impact

This issue affects all un-patched Splunk platform instance types, on any operating system:

  • Splunk Cloud
  • Splunk Light
  • Splunk Enterprise
    • Indexers, clustered or not
    • Heavy forwarders
    • Search heads, clustered or not
    • Search head deployers
    • Deployment servers
    • Cluster masters
    • License masters
  • Splunk universal forwarders, under the following known conditions:
    • When they have been configured to process structured data, such as CSV, XML, and JSON files, using the INDEXED_EXTRACTIONS setting in props.conf
    • When they have been configured to process data locally, using the force_local_processing setting in props.conf
    • When they have been configured with a monitor input without a source type, and that input subsequently encounters a file for which there is no pre-trained source type

The issue appears when you have configured the input source to automatically determine timestamps, and can result in one or more of the following problems:

  • Incorrect timestamping of incoming data
  • Incorrect rollover of data buckets due to the incorrect timestamping
  • Incorrect retention of data overall
  • Incorrect search results due to data ingested with incorrect timestamps

There is no method to correct the timestamps after the Splunk platform has ingested the data when the problem starts. If you ingest data with an un-patched Splunk platform instance beginning on January 1, 2020, you must patch the instance and re-ingest that data for its timestamps to be correct.

Cause

The Splunk platform input processor uses a file called datetime.xml to help the processor correctly determine timestamps based on incoming data. The file uses regular expressions to extract many different types of dates and timestamps from incoming data.

On un-patched Splunk platform instances, the file supports the extraction of two-digit years of "19", that is, up to December 31, 2019. Beginning on January 1, 2020, these un-patched instances will mistakenly treat incoming data as having an invalid timestamp year, and could either add timestamps using the current year, or misinterpret the date incorrectly and add a timestamp with the misinterpreted date.

Solutions

Splunk Cloud customers will receive the fix on their Splunk Cloud instances automatically. If you are a Splunk Cloud customer, your support representative will advise you when the upgrade will take place on your Splunk Cloud instance. As this is a critical update, there is no option to defer it.

While Splunk Cloud instances will receive automatic updates, you must perform one of these solutions on any self-deployed instances, such as heavy and universal forwarders that send data to your Splunk Cloud instance. You can perform one of these solutions at any time before or after your Splunk Cloud instance receives the fix. Contact your support representative if you have any questions.

There are four solutions to this problem for on-premises customers. You only need to perform one of the solutions to fix the problem:

  • Download and deploy an app to temporarily replace the defective datetime.xml with the fixed file
  • Download an updated version of datetime.xml and apply it to each of your Splunk platform instances
  • Upgrade Splunk platform instances to a version with an updated version of datetime.xml
  • Make modifications to the existing datetime.xml file on your Splunk platform instances

Download and deploy an app that temporarily replaces the defective datetime.xml file with the fixed file

Splunk has released a Splunk app that temporarily replaces the defective datetime.xml file with the fixed file. This option is the preferred path for customers who have large deployments including many universal forwarders and indexer and search head clusters.

You do not need to stop the Splunk platform before you deploy the apps, but you must restart each instance that receives the apps.

To ensure that you have no problems with universal forwarders and timestamp recognition, deploy this fix on all universal forwarders, even those that do not specifically suffer from this problem.

This solution is temporary only. Do not run it long-term. Carefully read the instructions that come with the app download and make sure to deploy the correct app to the correct Splunk platform instance type. Do not deploy both apps to a single instance. The preferred method to address this problem is to upgrade to a version that has the fixed datetime.xml file. After you complete upgrades, remove the apps immediately.

  1. Download the datetime.xml fix app archive (MD5 hash: 2e6b7520fa1379d72ac80ca21a54d45a) from splunk.com.
  2. Unarchive the .tgz file to a location that is accessible from all of your Splunk platform instances.
  3. Open the README file and follow the instructions to deploy one of the apps to each Splunk platform instance.

Download an updated version of datetime.xml and apply it to each of your Splunk platform instances

Splunk is providing an updated version of the datetime.xml file for download. This option is the preferred path for customers who cannot upgrade right away to a version of the Splunk platform with the fixed file, or who run an unsupported version that is lower than 6.6.x.

After you download the file, you must apply it directly over the existing datetime.xml file using the following procedure. You must apply the updated file to all affected on-premises Splunk platform instances prior to January 1, 2020, or you will experience timestamp recognition problems from that point forward.

  1. Download the datetime.zip timestamp recognition ZIP file from splunk.com.
  2. Unarchive the ZIP file to a location that is accessible from all of your Splunk platform instances.
  3. On each Splunk platform instance, do the following:
    1. Using your operating system file management utilities, copy the updated datetime.xml from the location where you downloaded it to the $SPLUNK_HOME/etc directory on the Splunk platform instance. Ensure that the updated file overwrites the existing file.
    2. Confirm that the new datetime.xml has been written to the $SPLUNK_HOME/etc directory.
    3. Restart the Splunk platform. Your Splunk platform instance is now patched.

Upgrade Splunk platform instances to a version with an updated version of datetime.xml

Splunk is releasing updated versions of the Splunk platform that contain an updated datetime.xml.

To fix the problem, download and install a version of the Splunk platform that contains the fixed datetime.xml. The following table lists the minor versions that include the fixed file and provide links on how to upgrade Splunk Enterprise and Splunk Light. Apply the fixed version directly over the existing Splunk platform instance to patch it.

Major version Minor version with patched file Released? Splunk platform upgrade instructions
6.6 6.6.12.1 Yes Splunk Enterprise
7.0 7.0.13.1
(versions 7.0.12 and 7.0.13 are Splunk Cloud-only releases)
Yes Splunk Enterprise
7.1 7.1.10 No
7.2 7.2.9.1 Yes Splunk Enterprise
7.3 7.3.3 Yes Splunk Enterprise
8.0 8.0.1 No

Make modifications to the existing datetime.xml file on your Splunk platform instances

If you feel comfortable with making changes to the datetime.xml file directly, you can do so without upgrading Splunk Enterprise or Splunk Light. If you are a Splunk Cloud customer that uses forwarders to send data to your Splunk Cloud instance, you can use this procedure to update those instances.

Use caution when making changes to the $SPLUNK_HOME/etc/datetime.xml file. Outside of fixing this specific issue, never make changes to this file. Typos and incompatible characters in the file can cause direct, lasting negative effects on timestamp recognition and data ingestion. If you are not comfortable with making these changes, consider one of the previous options for fixing this problem, or contact Splunk Support for assistance.

You must complete these changes on all affected Splunk platform instances by January 1, 2020, or you will experience timestamp recognition problems from that point forward.

  1. Using either a shell prompt or your operating system file management utilities, go to the $SPLUNK_HOME/etc directory in your Splunk Enterprise installation.
  2. Open datetime.xml for editing with a text editor.
  3. Search for and replace the following strings, according to this table:
    Search for this string Replace with this string
    <text><![CDATA[(20\d\d|19\d\d|[901]\d(?!\d))]]></text>
    <text><![CDATA[(20\d\d|19\d\d|[9012]\d(?!\d))]]></text>
    <text><![CDATA[(?:^|source::).*?(?<!\d|\d\.|-)(?:20)?([901]\d)(0\d|1[012])([012]\d|3[01])(?!\d|-| {2,})]]></text>
    <text><![CDATA[(?:^|source::).*?(?<!\d|\d\.|-)(?:20)?([9012]\d)(0\d|1[012])([012]\d|3[01])(?!\d|-| {2,})]]></text>
    <text><![CDATA[(?:^|source::).*?(?<!\d|\d\.)(0\d|1[012])([012]\d|3[01])(?:20)?([901]\d)(?!\d| {2,})]]></text>
    <text><![CDATA[(?:^|source::).*?(?<!\d|\d\.)(0\d|1[012])([012]\d|3[01])(?:20)?([9012]\d)(?!\d| {2,})]]></text>
    <text><![CDATA[((?<=^|[\s#,"=\(\[\|\{])(?:1[012345]|9)\d{8}|^@[\da-fA-F]{16,24})(?:\.?(\d{1,6}))?(?![\d\(])]]></text>
    <text><![CDATA[((?<=^|[\s#,"=\(\[\|\{])(?:1[0123456]|9)\d{8}|^@[\da-fA-F]{16,24})(?:\.?(\d{1,6}))?(?![\d\(])]]></text>
  4. Save the file and close it. If the editor asks if you want to overwrite the file, answer yes.
  5. Confirm that the datetime.xml file has been updated.
  6. Restart the Splunk platform. Your Splunk platform instance is now patched.

Solutions for customers with large installations

If you have a large on-premises deployment of many Splunk platform instances, contact Professional Services for assistance. If you feel comfortable with deploying Splunk apps, you can also use the app deployment solution, as described in the "Solutions" section.

Validate timestamp extraction after an update

After you have applied one of the solutions described in this topic, you can use the following procedure to validate that timestamp extraction works as expected.

As part of this process, you must temporarily configure the input processor to accept timestamps that occur in the future. This procedure includes instructions for configuring the MAX_DAYS_HENCE setting in the default stanza of the props.conf configuration file. After you have successfully validated timestamp extraction, you can remove the MAX_DAYS_HENCE setting from props.conf to restore default timestamp extraction behavior.

  1. Paste the following text into a text editor:
    date,message
    19-12-31 23:58:44,Test Message  - datetime.xml testing - override - puppet managed forced restart
    20-01-02 23:58:54,Test Message  - datetime.xml testing - override - puppet managed forced restart
    
  2. Save the text as a text file, for example, test_file.csv, to a place that is accessible from all of your Splunk platform instances.
  3. On the Splunk platform instance that you want to validate, adjust the MAX_DAYS_HENCE setting for the [default] stanza in the $SPLUNK_HOME/etc/system/local/props.conf configuration file.
    [default]
    MAX_DAYS_HENCE = 40
    
  4. Restart the Splunk platform.
  5. Using the Splunk CLI, add the text file you saved earlier as a oneshot monitor to the Splunk platform instance that you want to validate.
    $SPLUNK_HOME/bin/splunk add oneshot -source test_file.csv -sourcetype csv -index main
    
  6. Perform a search on the text in Step 1. The text with the two digit "20" should have a timestamp with the correct year of 2020.

File integrity errors after an update to the file

If you perform either of the file update or the direct file modification solutions, upon restart, Splunk Web will display a message like the following:

File Integrity checks found 1 files that did not match the system-provided manifest. Review the list of problems reported by the InstalledFileHashChecker in splunkd.log File Integrity Check View ; potentially restore files from installation media, change practices to avoid changing files, or work with support to identify the problem.

You can safely disregard this message in these two scenarios only. After you upgrade to a version that has the fixed datetime.xml, this message should no longer appear.

More information

PREVIOUS
Transparent huge memory pages and Splunk performance
  NEXT
Deprecated features

This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.0.15, 6.1, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.7, 6.1.8, 6.1.9, 6.1.10, 6.1.11, 6.1.12, 6.1.13, 6.1.14, 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.2.14, 6.2.15, 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.3.14, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.4.9, 6.4.10, 6.4.11, 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.5.6, 6.5.7, 6.5.8, 6.5.9, 6.5.10, 6.6.0, 6.6.1, 6.6.2, 6.6.3, 6.6.4, 6.6.5, 6.6.6, 6.6.7, 6.6.8, 6.6.9, 6.6.10, 6.6.11, 6.6.12, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.0.13, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 8.0.0


Comments

Where can we review a log of changes made to this page and to the datetime.xml fix app? Asking because it looks like there are now at least three versions of the app which have been released here, and because the "last significant update" date and time at the top of the page keeps changing.

Thank you.

Blackburnjm
December 6, 2019

Hi Mxg142 (and everyone),

A few updates in the last hour: We have released patches for versions 6.6, 7.0, and 7.2 today. Currently, that leaves versions 7.1 and 8.0 un-patched. We are working to provide patches for those versions but I can't state their availability time. Please continue watching this space.

On the effort to determine which instances need patching, there is a query for that in the app download. That said, we want to make sure what we deliver works for all of our customers ideally. That takes time.

On whether or not we're inducing panic - I understand the concern, yet respectfully disagree. We would rather over-report a problem and have you determine that it's not as big a deal for you, than under-report it and end up with some of our customers losing data. Obviously, we want to get it right on the nose.

We thank you again for your continued patience and understanding. We'll continue to post updates as they arrive. We will get through this!

Malmoore, Splunker
December 5, 2019

Thank you to Pratt daniel, Bjcross, Ssiat479 for helping out and providing a search query to validate risk for two digit year/Jan 1 deadline. Personally I think Splunk should have provided a query to begin with in the Release Notes and save all of us the hassle and panic since I bet most of us are not ingesting two digit years anyway. The September 2020 deadline is definitely a critical must-do, but I think the knee jerk of everyone reading this initially was "we must get this in by January!!" when in reality the risk for that deadline is quite minimal for most users.

Mxg142
December 5, 2019

Hi everyone,

Wanted to thank you for your continued patience and understanding as well as answer some additional questions with regard to this problem. While I can't provide future statements on releases, I can say without hesitation that we're working to get releases out as quickly as possible.

Version 7.2.9.1 was released this morning.

On Windows compatibility - to be safe, substitute '/' with '\' in configuration files. Otherwise, the apps should work as delivered.

On the issue with universal forwarders and file monitor input. I'm working on getting specific clarification. In my understanding the bug triggers if you don't specify a source type, and the monitor encounters a file for which it has no pre-trained source type. We'll update the release note when we have that specific information.

Again, thank you all for your continued patience and understanding on this.

Malmoore, Splunker
December 5, 2019

Building off of the searches provided by Bjcross and others, we have found the following search to be even more helpful in identifying specific hosts reporting two digit years in their timestamps:

index=*
| eval timestamp_length = timeendpos-timestartpos+1,
estimated_timestamp=ifnull(substr(_raw, timestartpos+1, timeendpos-timestartpos),"unknown"),
combined=timestamp_length." | ".estimated_timestamp
| where !(match(estimated_timestamp,"2019")) AND !(match(estimated_timestamp,"\d{10}"))
| rex field=estimated_timestamp "^(?<estimated_timestamp>.*)\.\d+$"
| rex field=estimated_timestamp "(?<bad>(?<!20|:)19(?!:))"
| where isnotnull(bad)
| stats latest(combined)
values(index)
values(sourcetype)
by host

Ssiat479
December 5, 2019

Does the provided app all_date_patch_props work with windows as well? Does it need changing of "/" to "\" to respect windows?

Rijom
December 5, 2019

Building off Pratt Daniel's search, this will reduce the output so that the list is more manageable.

index=*
| dedup sourcetype
| fields sourcetype, timeendpos,timestartpos, _raw
| eval timestamp_length = timeendpos-timestartpos+1
| dedup sourcetype, timestamp_length
| eval estimated_timestamp=ifnull(substr(_raw, timestartpos+1, timeendpos-timestartpos),"unknown")
| rex field=estimated_timestamp "(?<bad>(?<!20)19)"
| search bad=*
| table sourcetype, timestamp_length, estimated_timestamp

Bjcross
December 3, 2019

Hello, I also don't quite understand wording for UF:
When they have been configured with a monitor input, and that input subsequently encounters an unknown file type

What does this mean?

We have config like this:
[monitor:///var/log/messages]
disabled = false

Are we affected?

Andrisulass
December 3, 2019

When can we expect release of Splunk 8.0.1? Please release this update ASAP. Updating is the easiest solution for this issue in my company.

DfgrtKJH
December 3, 2019

Hi,
I read the following part, but couldn't understand what "unknown file type" means.
==========
When they have been configured with a monitor input, and that input subsequently encounters an unknown file type
==========

Is "unknown file type" about sourcetypes that doesn't include in pretrained source types shown in the following document?

-List of pretrained source types
https://docs.splunk.com/Documentation/Splunk/8.0.0/Data/Listofpretrainedsourcetypes

YKawasaki
December 2, 2019

Will the app be posted to Splunkbase?

Satyenshahusda
December 2, 2019

Hi everyone,

Wanted to provide an update as to where we stand with this issue. We understand the inconvenience and apologize for that. Again, we can't provide forward looking statements owing to legal reasons, which means we can't tell you when a specific updated version will be released, no matter how many times you ask, nor how nicely.

If you run a version of on-premises Splunk software that is lower than 6.6, your options are:
- Upgrade to at least 7.3, and 8.0 if you can
- Download our app solution and use deployment server/cluster master to distribute it amongst affected instances.

If you use the TIME_FORMAT setting to configure timestamp recognition, this problem should not affect you. But, to ensure you have no problems with recognition, it is a good idea to update any instance that ingest data anyway.

This page will continue to be updated as warranted in the near term. Thank you for your continued patience and understanding.

Malmoore, Splunker
December 2, 2019

@Mxg142 Here is a query I used that should assist you with this bug, and identifying the sourcetypes that have 2 digit year raw timestamps.

index=*
| dedup sourcetype
| fields sourcetype, timeendpos,timestartpos, _raw
| eval timestamp_length = timeendpos-timestartpos+1
| dedup sourcetype, timestamp_length
| eval estimated_timestamp=ifnull(substr(_raw, timestartpos+1, timeendpos-timestartpos),"unknown")
| table sourcetype, timestamp_length, estimated_timestamp

Pratt daniel
December 2, 2019

Any word when the rest of the patched datetime.xml versions of Splunk will become available for download? As of this moment, I only see 7.3.3 as released. Mostly interested in 7.2.9.1.

Adamsmith47
December 2, 2019

To which Version I need to upgrade? I have Splunk 6.5.4.
For permanent solution of the problem only version above 6.6. are specified above .
/tejas

Tejasgoel
December 2, 2019

When can we expect release of Splunk 8.0.1?

Net1993
November 28, 2019

Download worked. Thanks

Cyrmue
November 28, 2019

Hi
The Workaround with Apps:
"Download and deploy an app that temporarily replaces the defective datetime.xml file with the fixed file"
is not downloadable from google Drive? (https://drive.google.com/file/d/17WR7mBqyhDLEOwQAipfp-MW8aYWVT5wZ/view?usp=sharing)
Where can we download it?

Cyrmue
November 27, 2019

If `TIME_FORMAT` is set on the parsing node (Indexer, HF, and UF in some cases.), I think this problem have no effect.

Because I think that the `datetime.xml` is a file that is used to automatically extract timestamps without using `TIME_FORMAT`, and `TIME_FORMAT` follows the Unix strptime format NOT `datetime.xml`.

https://docs.splunk.com/Documentation/Splunk/8.0.0/Data/Configuretimestamprecognition#Enhanced_strptime.28.29_support
https://pubs.opengroup.org/onlinepubs/9699919799/functions/strptime.html

If this is correct, I think this workaround should be written in this doc.

Yutaka1005
November 27, 2019

Hi everyone,

First off, thank you in advance for your continued patience. We appreciate that this is very inconvenient to say the least. I will try to answer as many questions as I can.

On future patched versions: When those versions come out, this page will be updated.

On the support for versions lower than 6.6: Currently, you can download and apply the patched file to your systems, or modify it directly. While there is no official support, this should resolve the problem until you can upgrade to a fixed version.

On whether or not you can ignore warnings in splunkd.log, Monitoring Console, or the system bar: Yes, for this situation only.

We have updated our guidance on when this problem affects universal forwarders.

The best advice I can give everyone right now is to keep monitoring this page. I can't provide forward looking statements, owing to legal rules, and I kinda like this job, so I'm gonna try and keep it. :)

Thank you for your understanding.

Malmoore, Splunker
November 27, 2019

our instances have "version 4.0" which is same as the above provided link.
Do i still need to update/replace the file.

Neelamsantosh
November 26, 2019

Another note, when replacing the datetime.xml, be sure to match the file permissions to the original file so that it can be read.

Rchristian splunk, Splunker
November 26, 2019

This is the sort of thing that -should- trivially be able to be handled by the deployment server. I suspect that isn't the case, which is another sign the DS, overall, needs some attention.

Brians76
November 26, 2019

What happens for versions 6.0.15? Will this work for earlier Splunk versions before 6.6? We have this deployed worldwide and need guidance on what to do.

Ggb667
November 26, 2019

After updating the datetime.xml file, we found error "InstalledFilesHashChecker" in splunkd.log and our Splunk web UI.
Can we suppress the messages? Or it should be accepted for this patching?

Takaakinakajima
November 26, 2019

Do we need to apply to UF as well? Can it be done thru deployment server?

Junie
November 26, 2019

Can someone provide a search query to see the percentage of our ingested data uses just the two digit timestamp?

Mxg142
November 26, 2019

What Akram said, do UF need this file and if so what is the procedure to deploy through deployment server.

Rayadx
November 26, 2019

V.8.0.1 is mentioned in the upgrade path - when will that version be released?

/Claus

ClausBom
November 26, 2019

It looks like the patch postpones the issue to Jan 1 2030, and November 14 2023. When will a more robust fix be released?

Mewett
November 25, 2019

Does this patch only apply for servers? how about UF? Do we need to apply to UF as well? Can it be done thru deployment server?

Akram
November 25, 2019

Hi Davidstuffle,

At this time our guidance is as written: Stop, update the file, restart. If there is a change to that guidance, we will update this page. We thank you for your patience in this developing situation and apologize for what we understand is significant inconvenience.

Malmoore, Splunker
November 25, 2019

Is it necessary to stop...update file...start, or can the file be updated in advance and then restart Splunk?

Davidstuffle
November 25, 2019

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters