Configure advanced timestamp recognition with the datetime.xml file

The Splunk platform uses the datetime.xml timestamp recognition file to extract dates and timestamps from events as it indexes them. The file contains regular expressions that describe how the Splunk platform is to perform those extractions from the raw event data.

In nearly all cases, you do not need to make modifications to the datetime.xml file. In those cases where you do make modifications to the file, you must take care in ensuring the its structure remains intact and that there are no typos, as this can cause significant problems with timestamp recognition.

If you need to modify this file on a Splunk Cloud Platform instance, file a support ticket. It is not possible to modify the file on a Splunk Cloud Platform instance directly. Instead, consider whether or not you need to modify the file at all. Configure the file, if necessary, on a universal or heavy forwarder on the machine that contains the data that you want to send to Splunk Cloud Platform.

On Splunk Enterprise, consider using the props.conf configuration file to configure timestamp recognition

In most cases, you do not need to make changes to the datetime.xml timestamp recognition file on Splunk Enterprise instances. The props.conf configuration file is responsible for most timestamp configuration changes.

When you configure timestamp recognition with the props.conf file, Splunk Enterprise uses the datetime.xml file to configure its timestamp processor and extract timestamps out of the events for the source, source type, or host information in your data. If the software can't process the timestamps in your event data, you can configure Splunk Enterprise to extract the timestamps by making a custom version of the datetime.xml file.

Structure of the datetime.xml file

The datetime.xml file has the following parts:

• Code blocks that define individual elements of a time stamp
• Code blocks with other elements defined within the file
• Extraction pattern code blocks

Each definition code block has one or more <text> definitions that contain a regular expression that Splunk Enterprise uses to extract the timestamp element.

Code blocks that define individual elements of a time stamp

These individual elements can contain information such as year, month, day, hour, and minute. The following example code block defines the regular expression that Splunk Enterprise uses to extract a literal month element (for example, Jan, Mar) out of event data:

<define name="_litmonth"  extract="litmonth">
<text><![CDATA[(?<![\d\w])(jan|\x{3127}\x{6708}|feb|\x{4E8C}\x{6708}|mar|\x{4E09}\x{6708}|apr|\x{56DB}\x{6708}|may|\x{4E94}\x{6708}|jun|\x{516D}\x{6708}|jul|\x{4E03}\x{6708}|aug|\x{516B}\x{6708}|sep|\x{4E5D}\x{6708}|oct|\x{5341}\x{6708}|nov|\x{5341}\x{3127}\x{6708}|dec|\x{5341}\x{4E8C}\x{6708})[a-z,\.;]*]]></text>
</define>

Code blocks with other elements defined within the file

The following example code block defines the _time element, which extracts hours, minutes, seconds, subseconds, period of day, and time zone:

<define name="_time" extract="hour, minute, second, subsecond, ampm, zone">
<text><![CDATA[(?<!\d)]]></text>
        <use name="_hour"/>
        <text><![CDATA[:]]></text>
        <use name="_minute"/>
        <text><![CDATA[:]]></text>
        <use name="_second"/> 
        <text><![CDATA[(?:(?: \d{4})?[:,\.](\d+))? {0,2}]]></text>
        <use name="_ampm"/>
        <text><![CDATA[ {0,2}]]></text>
        <use name="_zone"/>
        <text><![CDATA[(?!:\d)]]></text>
</define>

Extraction pattern code blocks

Extraction pattern code blocks define the order in which to attempt extracting times and dates from incoming event data. In general, the timePatterns block defines the order in which the Splunk platform attempts to extract a timestamp under most conditions, and the datePatterns block defines how to extract dates.

While extraction code blocks in general define when the Splunk platform attempts timestamp extraction, they do not strictly dictate when timestamp extraction occurs. If, for example, there are multiple matches for a timestamp, the platform uses heuristics that favor matches that contain more information or that occur earlier in the event to determine when to perform an extraction.

Examples of custom datetime.xml configuration

The following blocks of code are examples of how to properly configure a custom datetime.xml file.

For example, suppose that the splunk train command generated the following code:

<define name="mycustom_date" extract="day,litmonth,year,">
<text><![CDATA[:\d+\s\w+\s(\d+)\s(\w+)\s(\d+)]]></text>
</define>
<define name="mycustom_time" extract="hour,minute,second,ampm,">
<text><![CDATA[(\d+):(\d+):(\d+)\s(\w+)]]></text>
</define>

See the Example 1a and Example 1b sections for ways you can proceed from this code. Then, see the Example 2 section for the next step.

Example 1a: Modification of existing datatime.xml

Proceeding the previous example, you can then add these definition blocks to an existing datetime.xml in $SPLUNK_HOME/etc/system/local that you copied previously:

<datetime>
 
<define name="mycustom_date" extract="day,litmonth,year,">
<text><![CDATA[:\d+\s\w+\s(\d+)\s(\w+)\s(\d+)]]></text>
</define>

<define name="mycustom_time" extract="hour,minute,second,ampm,">
<text><![CDATA[(\d+):(\d+):(\d+)\s(\w+)]]></text>
</define>

<... existing configurations removed for clarity ...>
 
<timePatterns>
      <use name="_time"/>
      <use name="_hmtime"/>
      <use name="_hmtime"/>
      <use name="_dottime"/>
      <use name="_combdatetime"/>
      <use name="_utcepoch"/>
      <use name="_combdatetime2"/>
      <use name="mycustom_time"/>
</timePatterns>

<datePatterns>
<use name="_usdate1"/> 
      <use name="_usdate2"/> 
      <use name="_isodate"/>
      <use name="_eurodate1"/> 
      <use name="_eurodate2"/> 
      <use name="_bareurlitdate"/> 
      <use name="_orddate"/>
      <use name="_combdatetime"/>
      <use name="_masheddate"/>
      <use name="_masheddate2"/>
      <use name="_combdatetime2"/>
      <use name="mycustom_date"/>
</datePatterns>

</datetime>

Example 1b: New datetime.xml with only your timestamp configuration

Instead of the example shown in the Example 1a section, you can also create a new datetime.xml file in $SPLUNK_HOME/etc/system/local, as follows:

<datetime>

<define name="mycustom_date" extract="day,litmonth,year,">
<text><![CDATA[:\d+\s\w+\s(\d+)\s(\w+)\s(\d+)]]></text>
</define>

<define name="mycustom_time" extract="hour,minute,second,ampm,">
<text><![CDATA[(\d+):(\d+):(\d+)\s(\w+)]]></text>
</define>

 <timePatterns>
       <use name="mycustom_time"/>
 </timePatterns>
 
 <datePatterns>
       <use name="mycustom_date"/> 
 </datePatterns>
 
 </datetime>

Example 2: Reference of new datetime.xml in props.conf for your custom source type

After completing the previous examples, you can then reference the custom datetime.xml file in the configuration for your source type in props.conf, as follows:

$SPLUNK_HOME/etc/system/local/props.conf

[my_custom_sourcetype]
DATETIME_CONFIG=/etc/system/local/datetime.xml
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true