Route specific events to different queues

Route specific events to different queues

In a distributed Splunk setup, you can send specific data to different queues for further processing. This topic discusses how to filter your data and send it specifically to nullQueue, or Splunk's /dev/null directory.

To filter certain events out before your data is indexed, use the instructions below to send those events to nullQueue.

Read more about how to filter and route to an alternate index.

Important: When you choose to filter your data depends on your distributed setup. However, the filtering needs to occur on the Splunk instance that parses the data; this may be either the indexer or the forwarder instance.

Configuration

To filter out specific events:

1. Identify an attribute of the event that can be used to separate it from others.

2. Create an entry in props.conf for the source, source type or host and specify a TRANSFORMS class and a TRANSFORMS name. The class name refers to a regular expression stanza you will place in transforms.conf.

3. Create an entry in transforms.conf with a regular expression that matches the identified attributes (from Step 1) and sets the DEST_KEY to queue and the FORMAT key to a specific queue (indexQueue, parsingQueue, nullQueue, etc).

Use the $SPLUNK_HOME/etc/system/README/props.conf.example and ../transforms.conf.example as examples, or create your own props.conf and transforms.conf. Make any changes in $SPLUNK_HOME/etc/system/local/, or your own custom application directory in $SPLUNK_HOME/etc/apps/. For more information on configuration files in general, see how configuration files work.

props.conf

In $SPLUNK_HOME/etc/system/local/props.conf add the following stanza:

[<spec>]
TRANSFORMS-$name=$UNIQUE_STANZA_NAME

<spec> can be:

• <sourcetype>, the sourcetype of an event
• host::<host>, where <host> is the host for an event
• source::<source>, where <source> is the source for an event

$NAME is whatever unique identifier you want to give to your transform.

$UNIQUE_STANZA_NAME must match the stanza name of the transform you create in transforms.conf.

transforms.conf

In $SPLUNK_HOME/etc/system/local/transforms.conf add the following stanza:

[$UNIQUE_STANZA_NAME]
REGEX = $YOUR_CUSTOM_REGEX
DEST_KEY = queue
FORMAT = nullQueue

Name your stanza with $UNIQUE_STANZA_NAME to match the name you specified in props.conf. Add $YOUR_CUSTOM_REGEX based on the attribute you've identified; it should specify the key term that identifies the events you want to remove.

Leave DEST_KEY and FORMAT with the above values to send identified events to the nullQueue (delete them before indexing).

Send matching events to nullQueue

This example sends all sshd events from /var/log/messages to nullQueue.

In props.conf:

[source::/var/log/messages]
TRANSFORMS-null= setnull

In transforms.conf:

[setnull]
REGEX = \[sshd\]
DEST_KEY = queue
FORMAT = nullQueue

Send matching WMI events to nullQueue

For those using WMI to capture events from Windows machines, the syntax is specific in props.conf on the source. This example will allow you to filter out two different event codes (592 or 593) using an "or" statement in regex.

In props.conf:

[wmi]
TRANSFORMS-foo=wminull

In transforms.conf:

[wminull]
REGEX=(?m)^EventCode=(592|593)
DEST_KEY=queue
FORMAT=nullQueue

Send matching events to indexQueue, everything else to nullQueue

This example is the reverse of the previous. The user wants to keep only sshd events from /var/log/messages; everything else goes to nullQueue. In this case, you need to define two transforms.

In props.conf:

[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing

In transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = \[sshd\]
DEST_KEY = queue
FORMAT = indexQueue

• Was this topic useful?
• Post a comment