Advanced source type overrides

Advanced source type overrides

This topic shows you how you can configure Splunk to override sourcetypes on a per-event basis. It includes an example that shows you how to use transforms.conf in tandem with props.conf to override sourcetypes for events associated with a specific host, and goes on to show how you can do this for event data coming from a particular input or source.

For more information on performing basic source type overrides for event data that comes from specific inputs, or which has a particular source, see "Override automatic source type assignment" in this manual.

Configuration

To do this you'll set up two stanzas, one in transforms.conf, and another in props.conf. Edit these files in $SPLUNK_HOME/etc/system/local/, or your own custom application directory in $SPLUNK_HOME/etc/apps/.

transforms.conf

The transforms.conf stanza should follow this format:

[<unique_stanza_name>]
REGEX = <your_regex>
FORMAT = sourcetype::<your_custom_sourcetype_value>
DEST_KEY = MetaData:Sourcetype

• <unique_stanza_name> should reflect that it involves a sourcetype. You'll use this name later in the props.conf stanza.
• <your_regex> is a regular expression that identifies the events that you want to apply a custom sourcetype to (such as events carrying a particular hostname or other field value).
• <your_custom_sourcetype_value> is the sourcetype value that you want to apply to the regex-selected events.

Note: For a primer on regular expression syntax and usage, see Regular-Expressions.info. You can test regexes by using them in searches with the rex search command. Splunk also maintains a list of useful third-party tools for writing and testing regular expressions.

props.conf

Next you create a stanza in props.conf that references the transforms.conf stanza, as follows.

[<spec>]
TRANSFORMS-<value> = <unique_stanza_name>

• <spec> can be:
  • <sourcetype>, the sourcetype value of an event.
  • host::<host>, where <host> is the host value for an event.
  • source::<source>, where <source> is the source value for an event.
• <value> is any value that you want to give to your stanza to identify it. In this case you might just use "sourcetype" to identify it as a sourcetype.
• <unique_stanza_name> is the name of your stanza from transforms.conf.

Example: sourcetyping events originating from different hosts, indexed from a single input

Let's say that you have a shared UDP input, UDP514. Your Splunk instance indexes a wide range of data from a number of hosts through this input. You've found that you need to apply a particular sourcetype--which, for the purposes of this example we'll call "my_log"--to data originating from three specific hosts (host1, host2, and host3) that reaches Splunk through UDP514.

To start, you can use the regex that Splunk typically uses to extract the host field for syslog events. You can find it in system/default/transforms.conf:

[syslog-host]
REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?)\.\w+\s+)*\[?(\w[\w\.\-]{2,})\]?\s
FORMAT = host::$1
DEST_KEY = MetaData:Host

You can easily modify this regex to only match events from the hostnames you want (for the purposes of this example we're calling them host1, host2, and host3):

REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?)\.\w+\s+)*\[?(host1|host2|host3)[\w\.\-]*\]?\s

Now you can use that modified regex in a transform that applies the my_log sourcetype to events that come from those three hosts:

[set_sourcetype_my_log_for_some_hosts]
REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?)\.\w+\s+)*\[?(host1|host2|host3)[\w\.\-]*\]?\s
FORMAT = sourcetype::my_log
DEST_KEY = MetaData:Sourcetype

And then you can refer that transform to props.conf, which in this case is used to identify the specific input that carries the events that you want to sourcetype:

[source::udp:514]
TRANSFORMS-changesourcetype = set_sourcetype_my_log_for_some_hosts

Note: The above configuration applies the sourcetype to your specified event data by host at index-time. Make yourself aware of the implications of making changes to index-time processing.

• Was this topic useful?
• Post a comment