Splunk® Enterprise

Getting Data In

Download manual as PDF

Download topic as PDF

Configure advanced timestamp recognition with datetime.xml

Datetime.xml is a file that Splunk software uses to extract dates and timestamps from events as it indexes them. The file contains regular expressions that describe how Splunk software performs those extractions from the raw event data.

In nearly all cases, you do not need to make modifications to datetime.xml. When you configure timestamp recognition with props.conf, Splunk software uses datetime.xml to configure its timestamp processor and extract timestamps out of the events for your source, source type, or host. If the software is not able to process the timestamps in your event data, you can train Splunk software to extract the timestamps by making a custom version of datetime.xml.

Structure of datetime.xml

The datetime.xml file has the following structure:

  • Code blocks that define individual elements of a time stamp, such as year, month, day, hour, and minute. The following example code block defines the regular expression that Splunk uses to extract a literal month element (for example, Jan, Mar) out of event data:
<define name="_litmonth"  extract="litmonth">
<text><![CDATA[(?<![\d\w])(jan|\x{3127}\x{6708}|feb|\x{4E8C}\x{6708}|mar|\x{4E09}\x{6708}|apr|\x{56DB}\x{6708}|may|\x{4E94}\x{6708}|jun|\x{516D}\x{6708}|jul|\x{4E03}\x{6708}|aug|\x{516B}\x{6708}|sep|\x{4E5D}\x{6708}|oct|\x{5341}\x{6708}|nov|\x{5341}\x{3127}\x{6708}|dec|\x{5341}\x{4E8C}\x{6708})[a-z,\.;]*]]></text>
</define>
  • Code blocks that use other elements that have been defined within the file. The following example code block defines the _time element which extracts hours, minutes, seconds, subseconds, period of day, and time zone:
<define name="_time" extract="hour, minute, second, subsecond, ampm, zone">
<text><![CDATA[(?<!\d)]]></text>
        <use name="_hour"/>
	<text><![CDATA[:]]></text>
        <use name="_minute"/>
	<text><![CDATA[:]]></text>
        <use name="_second"/> 
	<text><![CDATA[(?:(?: \d{4})?[:,\.](\d+))? {0,2}]]></text>
        <use name="_ampm"/>
	<text><![CDATA[ {0,2}]]></text>
        <use name="_zone"/>
	<text><![CDATA[(?!:\d)]]></text>
</define>
  • Extraction pattern code blocks that define the order in which to attempt extracting times and dates from incoming event data. The timePatterns block defines the order in which Splunk software attempts to extract a timestamp, and the datePatterns block defines how to extract dates.

Each definition code block has one or more <text> definitions that contain a regular expression that Splunk software uses to extract the timestamp element.

Create or modify a custom datetime.xml

In nearly all cases, you do not need to modify datetime.xml, and you should not. Instead, you should configure props.conf for timestamp extractions. See Edit timestamp properties in props.conf for instructions.

If you cannot configure Splunk software to extract dates and times properly with props.conf, you might need to modify or substitute datetime.xml with a custom version. You can use the splunk train CLI command to take a sample of the timestamp data and generate code that you can use to create a custom datetime.xml that extracts your timestamp.

The splunk train CLI command is deprecated, but is still available to help you create patterns for datetime.xml based on your sample timestamp data.

After you create a pattern file with splunk train, you can make a copy of the default datetime.xml file and add your modifications to it, or you can create a new datetime.xml that only contains your custom timestamp definitions.

Never make modifications directly to $SPLUNK_HOME/etc/datetime.xml. This file gets overwritten any time you upgrade, and any errors in the file that occur as the result of your changes can cause serious, lasting problems with data ingestion for both your custom source type and all other source types on the instance. If you want to make changes to the default file, save a copy to $SPLUNK_HOME/etc/system/local and make the changes there.

  1. Create a sample timestamp pattern file
  2. Run the splunk train CLI command against the file
  3. Use the output to create a custom datetime.xml file
  4. Reference the custom datetime.xml file in your timestamp configuration

Create a sample timestamp pattern file

  1. From a prompt or PowerShell window, create a text file.
  2. Paste in the sample of your timestamp data into this file.
  3. Save the file and close it.
  4. Change to the $SPLUNK_HOME/bin directory.

Run the splunk train CLI command against the file

  1. Change to the $SPLUNK_HOME/bin directory:
    cd $SPLUNK_HOME/bin
    
  2. Run the splunk train CLI command:
    ./splunk train dates
    
  3. After the software asks the action you want to perform, type in L, l, or learn to perform the "learn" action.
  4. Enter the path to the file that contains the timestamp sample.
  5. Splunk software displays the first line of your sample and prompts you to enter values that represent the timestamp:
    ------------------------------------
    Interactively learning date formats.
    ------------------------------------
    INSTRUCTIONS: If a sample line does not have a timestamp, hit Enter.
    If it does have a timestamp, enter the timestamp separated by commas
    in this order: month, day, year, hour, minute, second, ampm, timezone.
    Use a comma as a placeholder for missing values.  For example, for a
    sample line like this "[Jan/1/08 11:56:45 GMT] login", the input 
    should be: "Jan, 1, 08, 11, 56, 45, , GMT" (note missing AM/PM).
    Spaces are optional.
    SAMPLE LINE 1:
            Tue Jul 10 21:23:06 PDT 2007 Received Trade 330 with detail user: user3456 date: date: 10Jul200721:
            23:06 action: sell 3583 MNAG @ 42
    --------------------------------------------------------------------------------
    Enter timestamp values as: month, day, year, hour, minute, second, ampm, timezone.
       >
    

    Enter values for month, day, year, hour, minute, second, period of day (am/pm), and time zone.
  6. If the values are sufficient, Splunk software displays the following to show it has remembered the pattern:
    Learned pattern.
    ----------------------------------------------------------------------------------
    If you are satisfied that the timestamps formats have been learned, hit control-c.
    ----------------------------------------------------------------------------------
    

    Hit Ctrl-C if Splunk software has correctly learned the timestamp formats.
  7. Splunk software displays text similar to the following:
    Patterns Learned. 
    It is highly recommended that you make changes to a copy of the default datetime.xml file.
    For example, copy "/Applications/splunk/etc/datetime.xml" to "/Applications/splunk/etc/system/local/datetime.xml", and work with that file.
    In that custom file, add the below timestamp definitions, and add the pattern names 
    to timePatterns and datePatterns list.
    For more details, see http://www.splunk.com/doc/latest/admin/TrainTimestampRecognition
    --------------------------------------------------------------------------------
    <define name="mycustom_date" extract="day,litmonth,year,">
    <text><![CDATA[:\d+\s\w+\s(\d+)\s(\w+)\s(\d+)]]></text>
    </define>
    <define name="mycustom_time" extract="hour,minute,second,ampm,">
    <text><![CDATA[(\d+):(\d+):(\d+)\s(\w+)]]></text>
    </define>
    ------------------------------------------------------
    What operation do you want to perform? (default=learn)
    ------------------------------------------------------
                               Enter choice: [Learn]/Test/Quit > 
    

    Review the pattern definitions in the output. If the definition for your timestamp sample looks the way that you want, quit the splunk train session by typing in Q, q, or quit. Otherwise, type in L, l, or learn again to attempt the training operation again.

Use the output to create a custom datetime.xml file

After you successfully train Splunk software to understand your custom timestamp, you must add the definition that splunk train generated to a custom version of datetime.xml.

You have two options to create this file:

  • Add your timestamp definitions to an existing datetime.xml. This is the preferred method.
  • Create a new datetime.xml that contains only your customized timestamp definitions. This option is better when the source type for your data is in a very strict format and Splunk software was incorrectly choosing a broader default format.

See "Examples of custom datetime.xml configuration" later in this topic.

  1. Make a copy of datetime.xml in the $SPLUNK_HOME directory.
    cd $SPLUNK_HOME/etc
    cp datetime.xml etc/system/local/
    

    Never make edits to $SPLUNK_HOME/etc/datetime.xml. Always make a copy of this file and add your custom timestamp patters to the copy.

  2. Open $SPLUNK_HOME/etc/system/local/datetime.xml for editing.
  3. Copy the block of code that splunk train generated and that begins with define name, on its own line, into the file. This code block can go anywhere between the <datetime> and <timePatterns> entries. See “Examples of custom datetime.xml configuration” later in this topic for examples.
  4. Within the <timePatterns> block, add a line that references the definition line you added earlier in the datetime.xml file.
  5. Within the <datePatterns> code block, add the same line you added in the previous step.
  6. Save the custom datetime.xml file and close it.

Reference the custom datetime.xml file in your timestamp configuration

After you build your custom datetime.xml file, you can reference it in props.conf to extract your custom timestamps. You can set a custom timestamp extraction pattern for any host, source, or source type.

  1. In $SPLUNK_HOME/etc/system/local, create props.conf if it does not already exist.
  2. Open props.conf in $SPLUNK_HOME/etc/system/local for editing.
  3. Add a stanza for the host, source, or source type that requires the custom timestamp extraction, if it does not already exist.
  4. Within this stanza, add a DATETIME_CONFIG setting that points to the custom datetime.xml, relative to the $SPLUNK_HOME directory. For example:
    [mysourcetype]
    DATETIME_CONFIG = /etc/system/local/datetime.xml
    MAX_TIMESTAMP_LOOKAHEAD = 128
    MAX_DAYS_AGO = 28
    
  5. Repeat the previous steps as necessary for other hosts, sources, or source types that require the custom extraction.
  6. Save props.conf and close it.
  7. Restart Splunk software.
  8. Confirm that timestamps are being extracted properly for the events that match the host, source, or source type that contains the custom timestamp extraction pattern.

Examples of custom datetime.xml configuration

The following blocks of code are examples of how to properly configure a custom datetime.xml file.

If splunk train generated code like the following:

<define name=“mycustom_date" extract="day,litmonth,year,">
<text><![CDATA[:\d+\s\w+\s(\d+)\s(\w+)\s(\d+)]]></text>
</define>
<define name=“mycustom_time" extract="hour,minute,second,ampm,">
<text><![CDATA[(\d+):(\d+):(\d+)\s(\w+)]]></text>
</define>

Example 1a: Modification of existing datatime.xml

Then you would add these definition blocks to an existing datetime.xml in $SPLUNK_HOME/etc/system/local that you copied previously:

<datetime>
 
<define name=“mycustom_date" extract="day,litmonth,year,">
<text><![CDATA[:\d+\s\w+\s(\d+)\s(\w+)\s(\d+)]]></text>
</define>

<define name=“mycustom_time" extract="hour,minute,second,ampm,">
<text><![CDATA[(\d+):(\d+):(\d+)\s(\w+)]]></text>
</define>

<... existing configurations removed for clarity ...>
 
<timePatterns>
      <use name="_time"/>
      <use name="_hmtime"/>
      <use name="_hmtime"/>
      <use name="_dottime"/>
      <use name="_combdatetime"/>
      <use name="_utcepoch"/>
      <use name="_combdatetime2"/>
      <use name=“mycustom_time”/>
</timePatterns>

<datePatterns>
<use name="_usdate1"/> 
      <use name="_usdate2"/> 
      <use name="_isodate"/>
      <use name="_eurodate1"/> 
      <use name="_eurodate2"/> 
      <use name="_bareurlitdate"/> 
      <use name="_orddate"/>
      <use name="_combdatetime"/>
      <use name="_masheddate"/>
      <use name="_masheddate2"/>
      <use name="_combdatetime2"/>
      <use name=“mycustom_date”/>
</datePatterns>

</datetime>

Example 1b: New datetime.xml with only your timestamp configuration

Or, you would create a new datetime.xml in $SPLUNK_HOME/etc/system/local, as follows:

<datetime>

<define name=“mycustom_date" extract="day,litmonth,year,">
<text><![CDATA[:\d+\s\w+\s(\d+)\s(\w+)\s(\d+)]]></text>
</define>

<define name=“mycustom_time" extract="hour,minute,second,ampm,">
<text><![CDATA[(\d+):(\d+):(\d+)\s(\w+)]]></text>
</define>

 <timePatterns>
       <use name="mycustom_date"/>
 </timePatterns>
 
 <datePatterns>
       <use name="mycustom_time"/> 
 </datePatterns>
 
 </datetime>

Example 2: Reference of new datetime.xml in props.conf for your custom source type

You would then reference the custom datetime.xml file in the configuration for your source type in props.conf, as follows:

$SPLUNK_HOME/etc/system/local/props.conf

[my_custom_sourcetype]
DATETIME_CONFIG=/etc/system/local/datetime.xml
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
PREVIOUS
Configure timestamp assignment for events with multiple timestamps
  NEXT
Specify time zones for timestamps

This documentation applies to the following versions of Splunk® Enterprise: 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.4.9, 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.5.6, 6.5.7, 6.6.0, 6.6.1, 6.6.2, 6.6.3, 6.6.4, 6.6.5, 7.0.0, 7.0.1, 7.0.2


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters