Route and filter data
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
- Discard specific events and keep the rest
- Keep specific events and discard the rest
- Filter WMI events
- Forward all internal index data
Route and filter data
Forwarders can filter and route data to specific receivers based on criteria such as source, sourcetype, or patterns in the events themselves. For example, a forwarder can send all data from one group of hosts to one Splunk server and all other data to a second Splunk server. A forwarder can also look inside the events and filter or route accordingly. For example, you might want to inspect WMI event codes to filter or route Windows events. This topic describes a number of typical routing scenarios.
Besides routing to receivers, forwarders can also filter and route data to specific queues or discard the data altogether by routing to the null queue.
Only regular forwarders can route or filter data at the event level. Light forwarders do not have the ability to inspect individual events.
Here's a simple illustration of a forwarder routing data to three Splunk receivers:
This topic describes how to route event data to Splunk instances. See Forward data to third-party systems in this manual for information on routing to non-Splunk systems.
This is the basic pattern for defining most routing scenarios:
1. Determine what criteria to use for routing. How will you identify categories of events, and where will you route them?
2. Edit props.conf to add a TRANSFORMS-routing attribute to determine routing based on event metadata:
<spec> can be:
<sourcetype>, the sourcetype of an event
host::<host>, where <host> is the host for an event
source::<source>, where <source> is the source for an event
<transforms_stanza_name> specified here when creating an entry in
Examples later in this topic show how to use this syntax.
3. Edit transforms.conf to specify target groups and to set additional criteria for routing based on event patterns:
[<transforms_stanza_name>] REGEX=<routing_criteria> DEST_KEY=_TCP_ROUTING FORMAT=<target_group>,<target_group>,....
<transforms_stanza_name>must match the name you defined in
- Enter the regex rules in
<routing_criteria>that determine which events get routed. This line is required. Use
REGEX = .if you don't need additional filtering beyond the metadata specified in
DEST_KEYshould be set to
_TCP_ROUTINGto send events via TCP. It can also be set to
_HTTPOUT_ROUTINGfor other output processors.
<target_group>that matches the group name you defined in
outputs.conf. A comma separated list will clone events to multiple target groups.
Examples later in this topic show how to use this syntax.
4. Edit outputs.conf to define the target group(s) for the routed data:
<target_group>to match the name you specified in
- Set the IP address and port to match the receiving server.
The use cases described in this topic generally follow this pattern.
Filter and route event data to target groups
In this example, the forwarder filters three types of events, routing them to different target groups. The forwarder filters and routes according to these criteria:
- Events with a sourcetype of "syslog" to a load-balanced target group
- Events containing the word "error" to a second target group
- All other events to a default target group
Here's how you do it:
$SPLUNK_HOME/etc/system/local to set two TRANSFORMS-routing attributes — one for syslog data and a default for all other data:
[default] TRANSFORMS-routing=errorRouting [syslog] TRANSFORMS-routing=syslogRouting
transforms.conf to set the routing rules for each routing transform:
[errorRouting] REGEX=error DEST_KEY=_TCP_ROUTING FORMAT=errorGroup [syslogRouting] REGEX=. DEST_KEY=_TCP_ROUTING FORMAT=syslogGroup
Note: In this example, if a syslog event contains the word "error", it will route to
errorGroup. This is due to the settings previously specified in
props.conf. Those settings dictated that all syslog events be filtered through the
syslogRouting transform, while all non-syslog (default) events be filtered through the
errorRouting transform. Therefore, only non-syslog events get inspected for errors.
outputs.conf to define the target groups:
[tcpout] defaultGroup=everythingElseGroup [tcpout:syslogGroup] server=10.1.1.197:9996, 10.1.1.198:9997 [tcpout:errorGroup] server=10.1.1.200:9999 [tcpout:everythingElseGroup] server=10.1.1.250:6666
syslogGroup and errorGroup receive events according to the rules specified in
transforms.conf. All other events get routed to the default group, everythingElseGroup.
Replicate a subset of data to a third-party system
This example uses data filtering to route two data streams. It forwards:
- All the data, in cooked form, to a Splunk indexer (10.1.12.1:9997)
- A replicated subset of the data, in raw form, to a third-party server (10.1.12.2:1234)
The example sends both streams as TCP. To send the second stream as syslog data, first route the data through an indexer.
[syslog] TRANSFORMS-routing = routeAll, routeSubset
[routeAll] REGEX=(.) DEST_KEY=_TCP_ROUTING FORMAT=Everything [routeSubset] REGEX=(SYSTEM|CONFIG|THREAT) DEST_KEY=_TCP_ROUTING FORMAT=Subsidiary,Everything
[tcpout] defaultGroup=nothing [tcpout:Everything] disabled=false server=10.1.12.1:9997 [tcpout:Subsidiary] disabled=false sendCookedData=false server=10.1.12.2:1234
For more information, see Forward data to third party systems in this manual.
Filter event data and send to queues
You can eliminate unwanted data by routing it to
/dev/null equivalent. When you filter out data in this way, the filtered data is not forwarded or added to the Splunk index at all, and doesn't count toward your indexing volume.
Although similar to forwarder-based routing, queue routing can be performed by either a forwarder or a full Splunk instance. It does not use the
outputs.conf file, just
Discard specific events and keep the rest
This example discards all
sshd events in
/var/log/messages by sending them to
props.conf, set the TRANSFORMS-null attribute:
[source::/var/log/messages] TRANSFORMS-null= setnull
2. Create a corresponding stanza in
DEST_KEY to "queue" and
FORMAT to "nullQueue":
[setnull] REGEX = \[sshd\] DEST_KEY = queue FORMAT = nullQueue
That does it.
Keep specific events and discard the rest
Here's the opposite scenario. In this example, you use two transforms to keep only the
sshd events. One transform routes
sshd events to
indexQueue, while another routes all other events to
Note: In this example, the order of the transforms in
transforms.conf matters. The null queue transform must come first; if it comes later, it will invalidate the previous transform and route all events to the null queue.
[source::/var/log/messages] TRANSFORMS-set= setnull,setparsing
[setnull] REGEX = . DEST_KEY = queue FORMAT = nullQueue [setparsing] REGEX = \[sshd\] DEST_KEY = queue FORMAT = indexQueue
Filter WMI events
To filter on WMI events, you must use the [wmi] sourcetype stanza in
props.conf. The following example uses regex to filter out two Windows event codes, 592 and 593:
[wminull] REGEX=(?m)^EventCode=(592|593) DEST_KEY=queue FORMAT=nullQueue
Filter data by target index
Splunk provides a forwardedindex filter that allows you to specify whether data gets forwarded, based on the data's target index. For example, if you have one data input targeted to "index1" and another targeted to "index2", you can use the filter to forward only the data targeted to index1, while ignoring the index2 data. The forwardedindex filter uses whitelists and blacklists to specify the filtering. For information on setting up multiple indexes, see the topic "Set up multiple indexes".
forwardedindex.<n>.whitelist|blacklist attributes in
outputs.conf to specify which data should get forwarded on an index-by-index basis. You set the attributes to regexes that filter the target indexes. By default, the forwarder forwards data targeted for all external indexes, as well as the data for the
_audit internal index. It does not forward data to other internal indexes. The default
outputs.conf file specifies that behavior with these attributes:
[tcpout] forwardedindex.0.whitelist = .* forwardedindex.1.blacklist = _.* forwardedindex.2.whitelist = _audit
In most deployments, you will not need to override the default settings. See outputs.conf for more information on how to whitelist and blacklist indexes.
Forward all internal index data
If you want to forward all internal index data (and not just the data in
_audit), you can override the default forwardedindex filter attributes like this:
#Forward everything [tcpout] forwardedindex.0.whitelist = .* # disable these forwardedindex.1.blacklist = forwardedindex.2.whitelist =
Note: In previous releases you could achieve this result (internal index forwarding) by specifying the
_TCP_ROUTING = * attribute/value in
inputs.conf. This attribute/value pair no longer achieves that result. If you wish to reinstate the 4.0.x simpler behavior, set
forwardedindex.filter.disable = true in