Route events to specific queues

Route events to specific queues

When you configure Splunk to forward and receive data, you can send specific data to different queues for further processing. This topic discusses how to filter your data and send it specifically to nullQueue, or Splunk's /dev/null directory.

To filter certain events out (and drop them) before your data is indexed, use the instructions below to send those events to nullQueue.

If you're looking to send specific events to a non-default index, read about how to filter and route events to a specific index.

Important: When you choose to filter your data depends on your distributed setup. However, the filtering needs to occur on the Splunk instance that parses the data; this may be either the indexer or the forwarder instance. With the 'SplunkLightForwarder' app enabled, these settings go on the indexer side. With the regular 'SplunkForwarder' app, they go on the forwarder side.

Configure routing events to a specific queue

To filter out specific events:

1. Identify an attribute of the event that can be used to separate it from others.

2. Create an entry in props.conf for the source, source type or host and specify a TRANSFORMS class and a TRANSFORMS name. The class name refers to a regular expression stanza you will place in transforms.conf.

3. Create an entry in transforms.conf with a regular expression that matches the identified attributes (from Step 1) and sets the DEST_KEY to queue and the FORMAT key to a specific queue (indexQueue, parsingQueue, nullQueue, etc).

Use the $SPLUNK_HOME/etc/system/README/props.conf.example and ../transforms.conf.example as examples, or create your own props.conf and transforms.conf. Make any changes in $SPLUNK_HOME/etc/system/local/, or your own custom application directory in $SPLUNK_HOME/etc/apps/. For more information on configuration files in general, read about how configuration files work.

Edit props.conf

In $SPLUNK_HOME/etc/system/local/props.conf add the following stanza:

[<spec>]
TRANSFORMS-$name=$UNIQUE_STANZA_NAME

<spec> can be:

• <sourcetype>, the sourcetype of an event
• host::<host>, where <host> is the host for an event
• source::<source>, where <source> is the source for an event

$NAME is whatever unique identifier you want to give to your transform.

$UNIQUE_STANZA_NAME must match the stanza name of the transform you create in transforms.conf.

Edit transforms.conf

In $SPLUNK_HOME/etc/system/local/transforms.conf add the following stanza:

[$UNIQUE_STANZA_NAME]
REGEX = $YOUR_CUSTOM_REGEX
DEST_KEY = queue
FORMAT = nullQueue

Name your stanza with $UNIQUE_STANZA_NAME to match the name you specified in props.conf. Add $YOUR_CUSTOM_REGEX based on the attribute you've identified; it should specify the key term that identifies the events you want to remove.

Leave DEST_KEY and FORMAT with the above values to send identified events to the nullQueue (eg delete them before indexing).

Send matching events to nullQueue

This example sends all sshd events from /var/log/messages to nullQueue.

In props.conf:

[source::/var/log/messages]
TRANSFORMS-null= setnull

In transforms.conf:

[setnull]
REGEX = \[sshd\]
DEST_KEY = queue
FORMAT = nullQueue

Send matching WMI events to nullQueue

For those using WMI to capture events from Windows machines, the syntax is specific in props.conf on the source. This example will allow you to filter out two different event codes (592 or 593) using an "or" statement in regex.

In props.conf:

[wmi]
TRANSFORMS-foo=wminull

In transforms.conf:

[wminull]
REGEX=(?m)^EventCode=(592|593)
DEST_KEY=queue
FORMAT=nullQueue

Send matching events to indexQueue, everything else to nullQueue

This example is the reverse of the previous. You want to keep only sshd events from /var/log/messages; everything else goes to nullQueue. In this case, you need to define two transforms.

In props.conf:

[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing

In transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = \[sshd\]
DEST_KEY = queue
FORMAT = indexQueue

The key here is that setting the queue to nullQueue does not instantly delete the events, but just marks them so that later (after all transforms) they will be deleted. Some events are then marked differently, and are indexed.

• Was this topic useful?
• Post a comment