Train Splunk to recognize timestamps

Train Splunk to recognize timestamps

By default, Splunk will be able to recognize most timestamps. However, if you find that Splunk is not extracting your timestamp correctly, you can train Splunk on a new timestamp format.

Please note: Training Splunk via the CLI is not the recommended method for specifying timestamps. Training should only be used if you cannot configure Splunk to recognize your timestamp format via props.conf.

About training

Splunk's commandline interface provides an interactive timestamp training process that produces regex-based expressions to be added to your own custom datetime.xml and props.conf.

Please note: Do not modify $SPLUNK_HOME/etc/datetime.xml. Instead, copy datetime.xml into your own custom bundle, or into $SPLUNK_HOME/etc/bundles/local/. Please read about bundle files before creating a custom bundle.

via the CLI

Training steps

These commands assume you have set a Splunk environment variable. If you have not, you must navigate to $SPLUNK_HOME/bin and run the ./splunk command.

From the CLI, type:

# splunk train dates

You will be presented with the following interaction:

 
------------------------------------------------------
What operation do you want to perform? (default=learn)
------------------------------------------------------
                           Enter choice: [Learn]/Test/Quit > L 

Choose "L" to train Splunk on timestamps.

             Enter full filename from which to learn dates > /opt/dates.txt 

Enter the entire path of the file on your Splunk server. This step will not tab-complete.

SAMPLE LINE 1:
        Tue Jul 10 21:23:06 PDT 2007 Received Trade 330 with detail user: user3456 date: date: 10Jul200721:
        23:06 action: sell 3583 MNAG @ 42
--------------------------------------------------------------------------------
If the above sample line does not have a timestamp, hit Enter.
If it does have a timestamp,
Enter timestamp values as: month, day, year, hour, minute, second, ampm, timezone.
   > 7, 10, 2007, 9, 23, 06, pm, PDT

Enter the values as noted above.

Patterns Learned.  Manually add these to '/opt/splunk/etc/datetime.xml'
    and add pattern names to timePatterns and datePatterns.

Edit datetime.xml

When the interaction is finished running, it will output a string like so:

<define name="_utcepoch" extract="utcepoch">                                                                                                                                                                                                                                                    
    <text><![CDATA[((?<=^|[\s#,"=\(\[\|\{])(?:1[01]|9)\d{8}|^@[\da-fA-F]{16,24})(?:\d{3})?(?![\d\(])]]></text>                                                                                                                                                                                  
</define> 

This string must be pasted into your custom datetime.xml, before the <timePatterns> stanza.

The define name must be added as a use name to both the <timePatterns> and <datePatterns> stanzas:

<timePatterns>                                                                                                                                                                                                                                                                                  
      <use name="_time"/>                                                                                                                                                                                                                                                                       
      <use name="_hmtime"/>                                                                                                                                                                                                                                                                     
      <use name="_hmtime"/>                                                                                                                                                                                                                                                                     
      <use name="_dottime"/>                                                                                                                                                                                                                                                                    
      <use name="_combdatetime"/>                                                                                                                                                                                                                                                               
      <use name="_utcepoch"/>                                                                                                                                                                                                                                                                   
</timePatterns>                                                                                                                                                                                                                                                                                 
<datePatterns>                                                                                                                                                                                                                                                                                  
      <use name="_usdate"/>                                                                                                                                                                                                                                                                     
      <use name="_isodate"/>                                                                                                                                                                                                                                                                    
      <use name="_eurodate"/>                                                                                                                                                                                                                                                                   
      <use name="_bareurlitdate"/>                                                                                                                                                                                                                                                              
      <use name="_orddate"/>                                                                                                                                                                                                                                                                    
      <use name="_combdatetime"/>                                                                                                                                                                                                                                                               
      <use name="_masheddate"/>                                                                                                                                                                                                                                                                 
      <use name="_masheddate2"/>                                                                                                                                                                                                                                                                
</datePatterns> 

Edit props.conf

Edit props.conf in $SPLUNK_HOME/etc/bundles/local and add a stanza for your source, host or sourcetype to specify the new datetime.xml file you have just created. You will need to add a DATETIME_CONFIG value to the stanza, specifying the path to your new datetime.xml.

<spec>
DATETIME_CONFIG=$YOUR_CUSTOM_PATH

<spec> can be:

1. <sourcetype>, the sourcetype of an event
2. host::<host>, where <host> is the host for an event
3. source::<source>, where <source> is the source for an event

$YOUR_CUSTOM_PATH is the location of your new datetime.xml file.

Please note: when specifying the path, set it relative to $SPLUNK_HOME.

For example:

[host::london]
DATETIME_CONFIG = /etc/bundles/local/datetime.xml

This example will apply your new datetime.xml to all events coming from the host with value london. You can set custom timestamp extraction patterns for any host, source or sourcetype by editing props.conf in this way.

• Was this topic useful?
• Post a comment