Train Splunk to recognize a timestamp
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Contents
Train Splunk to recognize a timestamp
By default, Splunk recognizes most timestamps. If Splunk doesn't recognize timestamps you want to extract, you can train Splunk to recognize new timestamps. Splunk's Command Line Interface (CLI) provides an interactive timestamp training tool (the train command) that produces regular expressions you add to datetime.xml and props.conf to configure timestamp extraction.
Note: Use this feature only if you can't configure Splunk to recognize a timestamp in props.conf.
Caution: Do not modify $SPLUNK_HOME/etc/datetime.xml. Instead, copy datetime.xml into your own custom bundle, or into $SPLUNK_HOME/etc/bundles/local/ (put your custom datetime.xml file in any path). Read about bundle files before creating a custom bundle.
Steps to train Splunk to recognize a timestamp
Train Splunk to recognize timestamp formats using a plain text file that contains your timestamp data. Splunk will learn any pattern of timestamp data you want it to based on the patterns in the text file you provide.
Note: These instructions assume you've set a Splunk environment variable. If you haven't, navigate to $SPLUNK_HOME/bin and run commands using the form: ./splunk [command].
Run the train command
The train command allows you to interactively train Splunk to improve how it handles timestamps, field extraction, and sourcetypes. Use the dates argument to train Splunk to learn timestamps. Include the path to the file containing the timestamp data you are training as an argument after dates.
Note: Type ./splunk help train in the CLI to learn about all of the arguments you can use with train.
Example:
This example shows how to use train.
Start timestamp training in the CLI by typing:
splunk train dates ohnoes.txt
Splunk displays:
------------------------------------------------------
What operation do you want to perform? (default=learn)
------------------------------------------------------
Enter choice: [Learn]/Test/Quit > L
Type "L" or "learn" to perform the training operation.
Splunk displays:
Enter full filename from which to learn dates > /opt/ohnoes.txt
Enter the path of the file on your Splunk server (this step doesn't allow tab-complete).
Splunk displays:
SAMPLE LINE 1:
Tue Jul 10 21:23:06 PDT 2007 Received Trade 330 with detail user: user3456 date: date: 10Jul200721:
23:06 action: sell 3583 MNAG @ 42
--------------------------------------------------------------------------------
If the above sample line does not have a timestamp, hit Enter.
If it does have a timestamp,
Enter timestamp values as: month, day, year, hour, minute, second, ampm, timezone.
> 7, 10, 2007, 9, 23, 06, pm, PDT
Enter values for month, day, year, hour, minute, second, ampm, and timezone (as shown above). This trains Splunk to recognize the values you enter as the designated portions of the timestamp.
Patterns Learned. Manually add these to '/opt/splunk/etc/datetime.xml'
and add pattern names to timePatterns and datePatterns.
Edit datetime.xml
After running train, Splunk outputs a string:
<define name="_utcepoch" extract="utcepoch">
<text><![CDATA[((?<=^|[\s#,"=\(\[\|\{])(?:1[01]|9)\d{8}|^@[\da-fA-F]{16,24})(?:\d{3})?(?![\d\(])]]></text>
</define>
Paste the string returned from train into your custom datetime.xml before the <timePatterns> and <datePatterns> stanzas. Also, add <use name="define name"/> for both <timePatterns> and <datePatterns> with the string defined as the <define name="string".
Example:
This example shows the above training output added to a sample datetime.xml file.
<define name="_utcepoch" extract="utcepoch">
<text><![CDATA[((?<=^|[\s#,"=\(\[\|\{])(?:1[01]|9)\d{8}|^@[\da-fA-F]{16,24})(?:\d{3})?(?![\d\(])]]></text>
</define>
<timePatterns>
<use name="_time"/>
<use name="_hmtime"/>
<use name="_hmtime"/>
<use name="_dottime"/>
<use name="_combdatetime"/>
<use name="_utcepoch"/>
</timePatterns>
<define name="_utcepoch" extract="utcepoch">
<text><![CDATA[((?<=^|[\s#,"=\(\[\|\{])(?:1[01]|9)\d{8}|^@[\da-fA-F]{16,24})(?:\d{3})?(?![\d\(])]]></text>
</define>
<datePatterns>
<use name="_usdate"/>
<use name="_isodate"/>
<use name="_eurodate"/>
<use name="_bareurlitdate"/>
<use name="_orddate"/>
<use name="_combdatetime"/>
<use name="_masheddate"/>
<use name="_masheddate2"/>
<use name="_utcepoch"/>
</datePatterns>
Edit props.conf
Set Splunk to use your custom datetime.xml file by adding a DATETIME_CONFIG key to timestamp configuration stanzas in props.conf, and set the value of DATETIME_CONFIG to the path to your custom datetime.xml.
Note: See all of the keys you can set in a stanza to configure timestamp recognition.
Example:
This example applies a custom datetime.xml to events from the host, "london".
[host::london] DATETIME_CONFIG = /etc/bundles/local/datetime.xml
You can set custom timestamp extraction patterns for any host, source, or sourcetype by editing props.conf.
This documentation applies to the following versions of Splunk: 3.2.2 , 3.2.3 , 3.2.4 , 3.2.5 , 3.2.6 View the Article History for its revisions.