Override source types on a per-event basis

This topic shows you how to override source types on a per-event basis. You do this at parse-time, after Splunk software has made its initial assignment as described in Why source types matter.

To configure per-event overrides, you use transforms.conf in tandem with props.conf.

Since this type of override occurs at parse-time, it works only on an indexer or heavy forwarder, not on a universal forwarder. See Configuration parameters and the data pipeline in the Admin manual for more information on what configurations are available at different points in the input/parsing/indexing process.

For information about configuring basic (not per-event) source type overrides for event data that comes from specific inputs or that has a particular source, see Override automatic source type assignment in this manual.

Configuration

To configure per-event overrides, you need to create two stanzas, one in transforms.conf and another in props.conf. Edit these files in $SPLUNK_HOME/etc/system/local/ or in your own custom application directory in $SPLUNK_HOME/etc/apps/. For more information about configuration files in general, see About configuration files in the Admin manual.

transforms.conf

Create a stanza in transforms.conf that follows this syntax:

[<unique_stanza_name>]
REGEX = <your_regex>
FORMAT = sourcetype::<your_custom_sourcetype_value>
DEST_KEY = MetaData:Sourcetype

Note the following:

• <unique_stanza_name> should reflect that it involves a source type. You'll use this name later in the props.conf stanza.
• <your_regex> is a regular expression that identifies the events that you want to apply a custom source type to (such as events carrying a particular hostname or other field value).
• <your_custom_sourcetype_value> is the source type that you want to apply to the regex-selected events.

Note: For a primer on regular expression syntax and usage, see Regular-Expressions.info. You can test regular expressions by using them in searches with the rex search command and using third-party tools.

props.conf

Next, create a stanza in props.conf that references the transforms.conf stanza:

[<spec>]
TRANSFORMS-<class> = <unique_stanza_name>

Note the following:

• <spec> can be:
  • <sourcetype>, the source type of an event.
  • host::<host>, where <host> is the host value for an event.
  • source::<source>, where <source> is the source value for an event.
• <class> is any unique identifier that you want to give to your transform.
• <unique_stanza_name> is the name of the stanza you created in transforms.conf.

Example: Assign a source type to events from a single input but different hosts

Let's say that you have a shared UDP input, "UDP514". Your Splunk deployment indexes a wide range of data from a number of hosts through this input. You've found that you need to apply a particular source type called "my_log" to data originating from three specific hosts (host1, host2, and host3) reaching your Splunk deployment through UDP514.

To start, you can use the regular expression that Splunk software typically uses to extract the host field for syslog events. You can find it in system/default/transforms.conf:

[syslog-host]
REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?)\.\w+\s+)*\[?(\w[\w\.\-]{2,})\]?\s
FORMAT = host::$1
DEST_KEY = MetaData:Host

You can easily modify this regular expression to only match events from the hostnames you want (in this example, host1, host2, and host3):

REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?)\.\w+\s+)*\[?(host1|host2|host3)[\w\.\-]*\]?\s

Now you can use the modified regular expression in a transform that applies the my_log source type to events that come from those three hosts:

[set_sourcetype_my_log_for_some_hosts]
REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?)\.\w+\s+)*\[?(host1|host2|host3)[\w\.\-]*\]?\s
FORMAT = sourcetype::my_log
DEST_KEY = MetaData:Sourcetype

Then you can specify that transform in a props.conf stanza that identifies the specific input for the events:

[source::udp:514]
TRANSFORMS-changesourcetype = set_sourcetype_my_log_for_some_hosts