2010 timestamps not recognized by Splunk

2010 timestamps not recognized by Splunk

Update your installation with these instructions.

About the issue

Splunk is not auto-recognizing some timestamps from the year 2010. The problem is specific to two-digit year representations; the timestamp for these events are not correctly indexed by Splunk and so the events are not returned correctly by search. This is a particular issue with Windows Event Log events, but affects all events with timestamps that use two digits to represent the year.

The fix

If events from 2010 are not returned by searches, replace the datetime.xml file in your Splunk installation with this one:

http://download.splunk.com/support/config/2010fixed.datetime.xml.gz

The datetime.xml file is located in $SPLUNK_HOME/etc. You must apply this file to all indexers, and to regular Splunk forwarders (but you do not have to apply it to light forwarders, since indexing is not occurring on them.)

To apply this file to your instance:

1. Download the file.
2. Decompress it: gzip -d 2010fixed.datetime.xml.gz
3. Copy it to your install: cp 2010fixed.datetime.xml path/to/splunk/etc/datetime.xml

This issue will be resolved in version 3.4.13. You can find below the procedure for recovering events between 01/Jan/2010 00:00:00 and the time you replaced datetime.xml.

Reindexing data affected by the 2010 timestamp issue

The following is a step-by-step procedure for recovering events between 01/Jan/2010 00:00:00 and the time you replaced datetime.xml.

Caveats

• This procedure will store a new copy of all the data it processes, so be sure you have plenty of disk space.
• This procedure will re-index all the data that was mis-indexed, and this will count against your license. If you experience license violation issues upon following this procedure, contact Splunk Support.

Procedure

1. Be sure you have installed a fixed datetime.xml.
2. Create an index in which to store the reindexed events. The provided app defaults to 'test'.
3. Download the winevt_timefix app and unzip it into your Splunk instance, under the $SPLUNK_HOME/etc/apps directory.
4. If your temporary index name is not 'test', edit the etc/apps/winevt_timefix/bin/evtreindex.py script included with the app to set the target_index to the index you created for this purpose.
5. Since 3.x runs searchscripts from $SPLUNK_HOME/bin/scripts directory, copy the script from etc/apps/winevt_timefix/bin/evtreindex.py to this location.
6. Run a search that captures a finite amount of data. Recommended: sourcetype=wineventlog* */*/10 endtime=01/25/2010:00:00:00. The search should be bounded, so you can delete the data later, and you should remember the conditions, for example by running this from the command line.
7. After verifying the above search gets what you want, use the Splunk command line splunk dispatch command to run your_search | evtreindex.
8. Wait for the search to complete. You should be able to validate that new data is arriving in your target index with the correct time, by searching, for example, in index=test
9. Verify that you got all the data you expected, and that it looks correct.
10. Deleting the same data as you searched is not precisely possible in 3.x. You can just leave the old data present, or you can process the totality of the eventlog data for a date range. The following is an example of how to delete a subset of data: splunk search '|oldsearch delete::sourcetype::WinEventLog:Security starttime::01/01/2009:00:00:00 endtime::01/25/2010:00:00:00
11. Stop Splunk.
12. Merge the buckets from the test index into the main index. A script is provided for this purpose, substitute in the name of the index you used: splunk cmd python etc/apps/winevt_timefix/bin/index_merge.py test main. This topic discuss how to do this by hand: http://www.splunk.com/wiki/Community:MoveIndexes.
13. Start Splunk.

• Was this topic useful?
• Post a comment