Multiline events

Multiline events

Many event logs have a strict one-line-per-event format, but some do not. Usually, Splunk can figure out where event boundaries are automatically. However, if event boundary recognition is not working as desired, you can set custom rules by configuring props.conf.

Configuration

To configure multi-line events, examine the format of the events. Determine a pattern in the events that can be used to set as the start or end of an event. Then, edit $SPLUNK_HOME/etc/bundles/local/props.conf, and set the necessary attributes for your data handling.

[<spec>]
SHOULD_LINEMERGE = <bool> (True)
  * When set to true splunk can combine several input lines into a single
    event, based on the following configuration attributes.
AUTO_LINEMERGE = <bool> (True)
  * Directs splunk to use automatic learning methods to determine
    where to break a stream of lines into events.
BREAK_ONLY_BEFORE_DATE = <bool> (False)
  * When set, splunk will create a new event if and only if it encounters
    a new line with a date.
BREAK_ONLY_BEFORE = <regular expression> (Empty)
  * When set, splunk will create a new event if and only if it encounters
    a new line that matches the regular expression
MUST_BREAK_AFTER = <regular expression> (Empty)
  * When set, and the regular expression matches the current line,
    splunk is guaranteed to create a new event for the next input line.
    Splunk may still break before the current line if another rule matches.
MUST_NOT_BREAK_AFTER = <regular expression> (Empty)
  * When set and the current line matches the regular expression, splunk will
    not break on any subsequent lines until the MUST_BREAK_AFTER expression
    matches. 
MAX_EVENTS = <integer> (256)
  * Specifies the maximum number of input lines that will be added to any
    event. Splunk will break after the specified number of lines are read.

Examples

[my_custom_sourcetype]
BREAK_ONLY_BEFORE = ^\d+\s*$

This example instructs Splunk to divide events in a file or stream by presuming any line that consists of all digits is the start of a new event, for any source whose source type was configured or determined by Splunk to be sourcetype::my_custom_sourcetype .

Another example:

The following log event contains several lines that are part of the same request. The differentiator between requests is "Path". The customer would like all these lines shown as one event entry.

{{"2006-09-21, 02:57:11.58",  122, 11, "Path=/LoginUser Query=CrmId=ClientABC&ContentItemId=TotalAccess&SessionId=3A1785URH117BEA&Ticket=646A1DA4STF896EE&SessionTime=25368&ReturnUrl=http://www.clientabc.com, Method=GET, IP=209.51.249.195, Content=", ""}}
{{"2006-09-21, 02:57:11.60",  122, 15, "UserData:<User CrmId="clientabc" UserId="p12345678"><EntitlementList></EntitlementList></User>", ""}}
{{"2006-09-21, 02:57:11.60",  122, 15, "New Cookie: SessionId=3A1785URH117BEA&Ticket=646A1DA4STF896EE&CrmId=clientabc&UserId=p12345678&AccountId=&AgentHost=man&AgentId=man, MANUser: Version=1&Name=&Debit=&Credit=&AccessTime=&BillDay=&Status=&Language=&Country=&Email=&EmailNotify=&Pin=&PinPayment=&PinAmount=&PinPG=&PinPGRate=&PinMenu=&", ""}}

To index this multiple line event properly, use the Path differentiator in your configuration. Add the following to your $SPLUNK_HOME/etc/bundles/local/props.conf:

[source::source-to-break]
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE = Path=

This code tells Splunk to merge the lines of the event, and only break before the term Path=.

• Was this topic useful?
• Post a comment