Syslog and timestamps

Selecting from multiple timestamps

Many Splunk add-ons collect event data from third-party products using syslog. You need to configure the third-party product to send syslog data to the Splunk platform in a format that the Splunk platform can understand. Configuration is slightly different for each event-producing or event-processing product.

It is very common practice in large enterprises to use multiple syslog systems to route and forward data before indexing in the Splunk platform. Depending on how these configuration choices are made, there can be multiple timestamps and/or hostnames included in an event when the Splunk platform sees it. If this is the case, we have no way of knowing which of the multiple timestamps or hostnames the customer wanted to use. Rather than ship unused extractions for all possible outcomes, the add-ons that expect syslog inputs have a single configuration. Customers may need to alter the regular expressions in these add-ons to extract the proper hostname and time stamp.

The Splunk platform normally looks at the text of the event for the timestamp, and by default will select the leftmost recognizable timestamp. Add-on specific configuration may have modified this behavior. Refer to the documentation for the individual add-on you are configuring. If there is an issue with using the timestamps included in the syslog events, you can modify props and transforms to select a different timestamp format.

Alternatively, you can change how the Splunk platform extracts timestamps. There may be cases where you would prefer to use the event's time of receipt, that is, the time the event is received to the Splunk platform, instead. To use the time of receipt as the event timestamp, add the setting DATETIME_CONFIG = NONE to the local props.conf for the add-on. For example, $SPLUNK_HOME/etc/apps/Splunk_TA_symantec-dlp/local/props.conf. This setting disables event-based timestamp processing, which means the Splunk platform will use the time the event is received at an indexer or heavy forwarder as the event timestamp. This can be preferred for multi-source syslog configurations with events from different time zones, because forcing a single relative time series makes correlation easier.

Selecting from multiple timezones

If an event timestamp does not include a recognizable timezone, the Splunk platform uses the time zone of the host that indexes the event. For example, if an event with field timestamp="2016-01-26 21:58:38.107" is indexed by a Splunk platform instance in GMT-8, it will subtract 8 hours from the timestamp and set the _time of event to: 2016-01-26T13:58:38.000-08:00. See How Splunk software determines time zones for details. The Splunk platform will also adjust the displayed _time based on the user's locale. See How browser locale affects timestamp formatting for more information.

It is not always possible to alter the logged time formats. Watch for potential issues at these points:

• Daylight savings time shifts. DST Fall Back events produce overlap in data streams. For example, if events are logging in EST (UTC-5) and shift to EDT (UTC-4), then new and old events will be interleaved. DST Spring Forward events produce gaps in data streams. For example, if events are logging in EST (UTC-5) and shift to EDT (UTC-4), then new events will be artificially offset from old events.
• Daylight savings time alterations. Daylight Savings is a legal construct rather than a physical one, and the rules change unpredictably with some regularity. These rule changes require OS and application level patches that are not reliably applied, which can cause systems in the same timezone to report with offsets from each other. Additionally, these offsets stop when patches are applied, which can be surprising.

Because this behavior can be confusing for data analysts in multiple timezones handling data sources from multiple timezones, Splunk recommends indexing data in UTC instead of local time. For information about timestamp configuration options, see Configure timestamp recognition in Getting Data In.