Rule-based association of sourcetypes

Rule-based association of sourcetypes

You can create rules to automatically assign source types in Splunk. Use props.conf to set source typing rules.

Source type precedence

Splunk associates source types using the following precedence:

1. Explicit specification of sourcetype per source by creating a stanza in props.conf:

[$SOURCE] 
sourcetype=$SOURCETYPE

1. Rule-based association of sourcetypes:

• allows you to match sources to sourcetypes using regular expressions in source type classification rules specified in rule:: stanzas in props.conf.

1. Intelligent document matching:

• will match similar-looking files and create a source type.

1. Delayed rules:

• work like rule-based associations, except you create a [delayedrule:: ] stanza in props.conf. These are useful "catch-all's" for source types, in case any were missed before Splunk attempted to create new source types.

1. New learned sourcetype creation:

• Splunk will create new source types based on sources that don't already have source types associated with them.

Configuration

A rule is created by adding a rule:: or delayedrule:: stanza to props.conf. Under the rule stanza, declare the name of the source type. After the source type declaration, list the rules to assign the source type. Rules are created based on a series of MORE_THAN, and LESS_THAN statements that must be matched. The statements are regular expressions that must be matched by the specified percentage of lines that match the regular expression. Any number of statements can be specified, and all statements must match in order for the source to fit the source type rule.

Add the following to $SPLUNK_HOME/etc/bundles/local/props.conf:

[rule::$RULE_NAME] OR [delayedrule::$RULE_NAME]
sourcetype=$SOURCETYPE
MORE_THAN = $REGEX
LESS_THAN = $REGEX

Please note: A rule can have many MORE_THAN and LESS_THAN patterns, and all are required for the rule to match.

Rules are created based on the percentage of the number of lines that contain the specified string. To match, a rule can be either MORE_THAN or LESS_THAN that percentage.

Examples

The following examples come from $SPLUNK_HOME/etc/bundles/default.

postfix syslog files

# postfix_syslog sourcetype rule
[rule::postfix_syslog]
sourcetype = postfix_syslog
# If 80% of lines match this regex, then it must be this type
MORE_THAN_80=^\w{3} +\d+ \d\d:\d\d:\d\d .* postfix(/\w+)?\[\d+\]:

delayed rule for breakable text

# breaks text on ascii art and blanklines if more than 10% of lines have
# ascii art or blanklines, and less than 10% have timestamps
[delayedrule::breakable_text]
sourcetype = breakable_text
MORE_THAN_10 = (^(?:---|===|\*\*\*|___|=+=))|^\s*$
LESSS_THAN_10 = [: ][012]?[0-9]:[0-5][0-9]

• Was this topic useful?
• Post a comment