
Route and filter data
Forwarders can filter and route data to specific receivers based on criteria such as source, source type, or patterns in the events themselves. For example, a forwarder can send all data from one group of hosts to one indexer and all data from a second group of hosts to a second indexer. Heavy forwarders can also look inside the events and filter or route accordingly. For example, you could use a heavy forwarder to inspect WMI event codes to filter or route Windows events. This topic describes a number of typical routing scenarios.
Besides routing to receivers, forwarders can also filter and route data to specific queues or discard the data altogether by routing to the null queue.
Important: Only heavy forwarders can route or filter all data at the event level. Universal forwarders and light forwarders do not have the ability to inspect individual events except in the case of file header extractions, but they can still forward data based on a data stream's host, source, or source type. They can also route based on the data's input stanza, as described below, in the subtopic, "Route inputs to specific indexers based on the data's input". Some input types have the ability to filter out some types of data while acquiring them (see below).
Here's a simple illustration of a forwarder routing data to three indexers:
This topic describes how to filter and route event data to Splunk Enterprise instances. See "Forward data to third-party systems" in this manual for information on routing to non-Splunk systems.
This topic also describes how to perform selective indexing and forwarding, in which you index some data locally on a heavy forwarder and forward the non-indexed data to one or more separate indexers. See "Perform selective indexing and forwarding" later in this topic for details.
Configure routing
This is the basic pattern for defining most routing scenarios (using a heavy forwarder):
1. Determine what criteria to use for routing. How will you identify categories of events, and where will you route them?
2. Edit props.conf to add a TRANSFORMS-routing attribute to determine routing based on event metadata:
[<spec>] TRANSFORMS-routing=<transforms_stanza_name>
Note the following:
<spec>
can be:<sourcetype>
, the source type of an eventhost::<host>
, where <host> is the host for an eventsource::<source>
, where <source> is the source for an event
- If you have multiple TRANSFORMS attributes, use a unique name for each. For example: "TRANSFORMS-routing1", "TRANSFORMS-routing2", and so on.
<transforms_stanza_name>
must be unique.
Use the <transforms_stanza_name>
specified here when creating an entry in transforms.conf
(below).
Examples later in this topic show how to use this syntax.
3. Edit transforms.conf to specify target groups and to set additional criteria for routing based on event patterns:
[<transforms_stanza_name>] REGEX=<routing_criteria> DEST_KEY=_TCP_ROUTING FORMAT=<target_group>,<target_group>,....
Note:
<transforms_stanza_name>
must match the name you defined inprops.conf
.- In
<routing_criteria>
, enter the regex rules that determine which events get routed. This line is required. Use "REGEX = .
" if you don't need additional filtering beyond the metadata specified inprops.conf
. DEST_KEY
should be set to_TCP_ROUTING
to send events via TCP. It can also be set to_SYSLOG_ROUTING
or_HTTPOUT_ROUTING
for other output processors.- Set
FORMAT
to a<target_group>
that matches a target group name you defined inoutputs.conf
. A comma separated list will clone events to multiple target groups.
Examples later in this topic show how to use this syntax.
4. Edit outputs.conf to define the target group(s) for the routed data:
[tcpout:<target_group>] server=<ip>:<port>
Note:
- Set
<target_group>
to match the name you specified intransforms.conf
. - Set the IP address and port to match the receiving server.
The use cases described in this topic generally follow this pattern.
Filter and route event data to target groups
In this example, a heavy forwarder filters three types of events, routing them to different target groups. The forwarder filters and routes according to these criteria:
- Events with a source type of "syslog" to a load-balanced target group
- Events containing the word "error" to a second target group
- All other events to a default target group
Here's how you do it:
1. Edit props.conf
in $SPLUNK_HOME/etc/system/local
to set two TRANSFORMS-routing attributes — one for syslog data and a default for all other data:
[default] TRANSFORMS-routing=errorRouting [syslog] TRANSFORMS-routing=syslogRouting
2. Edit transforms.conf
to set the routing rules for each routing transform:
[errorRouting] REGEX=error DEST_KEY=_TCP_ROUTING FORMAT=errorGroup [syslogRouting] REGEX=. DEST_KEY=_TCP_ROUTING FORMAT=syslogGroup
Note: In this example, if a syslog event contains the word "error", it will route to syslogGroup
, not errorGroup
. This is due to the settings previously specified in props.conf
. Those settings dictated that all syslog events be filtered through the syslogRouting
transform, while all non-syslog (default) events be filtered through the errorRouting
transform. Therefore, only non-syslog events get inspected for errors.
3. Edit outputs.conf
to define the target groups:
[tcpout] defaultGroup=everythingElseGroup [tcpout:syslogGroup] server=10.1.1.197:9996, 10.1.1.198:9997 [tcpout:errorGroup] server=10.1.1.200:9999 [tcpout:everythingElseGroup] server=10.1.1.250:6666
syslogGroup and errorGroup receive events according to the rules specified in transforms.conf
. All other events get routed to the default group, everythingElseGroup.
Replicate a subset of data to a third-party system
This example uses data filtering to route two data streams. It forwards:
- All the data, in cooked form, to a Splunk Enterprise indexer (10.1.12.1:9997)
- A replicated subset of the data, in raw form, to a third-party server (10.1.12.2:1234)
The example sends both streams as TCP. To send the second stream as syslog data, first route the data through an indexer.
1. Edit props.conf
:
[syslog] TRANSFORMS-routing = routeAll, routeSubset
2. Edit transforms.conf
:
[routeAll] REGEX=(.) DEST_KEY=_TCP_ROUTING FORMAT=Everything [routeSubset] REGEX=(SYSTEM|CONFIG|THREAT) DEST_KEY=_TCP_ROUTING FORMAT=Subsidiary,Everything
3. Edit outputs.conf
:
[tcpout] defaultGroup=nothing [tcpout:Everything] disabled=false server=10.1.12.1:9997 [tcpout:Subsidiary] disabled=false sendCookedData=false server=10.1.12.2:1234
For more information, see "Forward data to third party systems" in this manual.
Filter event data and send to queues
Although similar to forwarder-based routing, queue routing can be performed by an indexer, as well as a heavy forwarder. It does not use the outputs.conf
file, just props.conf
and transforms.conf
.
You can eliminate unwanted data by routing it to nullQueue
, the Splunk Enterprise equivalent of the /dev/null
device. When you filter out data in this way, the filtered data is not forwarded or added to the Splunk Enterprise index at all, and doesn't count toward your indexing volume.
Note: See "Caveats for routing and filtering structured data" later in this topic.
Discard specific events and keep the rest
This example discards all sshd
events in /var/log/messages
by sending them to nullQueue
:
1. In props.conf
, set the TRANSFORMS-null attribute:
[source::/var/log/messages] TRANSFORMS-null= setnull
2. Create a corresponding stanza in transforms.conf
. Set DEST_KEY
to "queue" and FORMAT
to "nullQueue":
[setnull] REGEX = \[sshd\] DEST_KEY = queue FORMAT = nullQueue
That does it.
Keep specific events and discard the rest
Here's the opposite scenario. In this example, you use two transforms to keep only the sshd
events. One transform routes sshd
events to indexQueue
, while another routes all other events to nullQueue.
Note: In this example, the order of the transforms in props.conf
matters. The null queue transform must come first; if it comes later, it will invalidate the previous transform and route all events to the null queue.
1. In props.conf
:
[source::/var/log/messages] TRANSFORMS-set= setnull,setparsing
2. In transforms.conf
:
[setnull] REGEX = . DEST_KEY = queue FORMAT = nullQueue [setparsing] REGEX = \[sshd\] DEST_KEY = queue FORMAT = indexQueue
Filter WMI and Event Log events
You can filter WinEventLog directly at the forwarder level.
Otherwise to filter on WMI events, use [WMI:WinEventLog:Security] source type stanza in props.conf
. The following example uses regex to filter out two Windows event codes, 592 and 593:
In props.conf
:
[WinEventLog:Security] TRANSFORMS-wmi=wminull
Note: In pre-4.2.x versions of Splunk Enterprise, you must use [wmi] or [WMI::WinEventLog:Security] as the source type in order to send events to nullQueue.
In transforms.conf
:
[wminull] REGEX=(?m)^EventCode=(592|593) DEST_KEY=queue FORMAT=nullQueue
Filter data by target index
Forwarders have a forwardedindex filter that allows you to specify whether data gets forwarded, based on the data's target index. For example, if you have one data input targeted to "index1" and another targeted to "index2", you can use the filter to forward only the data targeted to index1, while ignoring the index2 data. The forwardedindex filter uses whitelists and blacklists to specify the filtering. For information on setting up multiple indexes, see the topic "Set up multiple indexes".
Note: The forwardedindex filter is only applicable under the global [tcpout]
stanza. This filter does not work if it is created under any other stanza in outputs.conf
.
Use the forwardedindex.<n>.whitelist|blacklist
attributes in outputs.conf
to specify which data should get forwarded on an index-by-index basis. You set the attributes to regexes that filter the target indexes.
Default behavior
By default, the forwarder forwards data targeted for all external indexes, including the default index and any user-created indexes. Regarding data for internal indexes, the default behavior varies according to who is doing the forwarding:
- The universal forwarder forwards the data for the
_audit
internal index only. It does not forward data for other internal indexes. Its defaultoutputs.conf
file located in$SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/default
specifies that behavior with these attributes:
[tcpout] forwardedindex.0.whitelist = .* forwardedindex.1.blacklist = _.* forwardedindex.2.whitelist = _audit
- Heavy forwarder and full Splunk instances with forwarding enabled (for example, a search head with forwarding enabled) forward the data for the
_audit
and_internal
internal indexes. Their defaultoutputs.conf
file, located in$SPLUNK_HOME/etc/system/default
, specifies that behavior with these attributes:
[tcpout] forwardedindex.0.whitelist = .* forwardedindex.1.blacklist = _.* forwardedindex.2.whitelist = (_audit|_internal)
Note: The default behavior for heavy forwarders and full Splunk Enterprise instances changed in Splunk Enterprise version 5.0.2. In earlier versions, the _internal
index was not forwarded by default. Those forwarder types had the same behavior as the universal forwarder: only data for the _audit
internal index was forwarded.
In most deployments, you will not need to override the default settings. See outputs.conf for more information on how to whitelist and blacklist indexes. For more information on default and custom outputs.conf
files and their locations, see "Types of outputs.conf files".
Forward all external and internal index data
If you want to forward all internal index data, as well as all external data, you can override the default forwardedindex filter attributes like this:
#Forward everything [tcpout] forwardedindex.0.whitelist = .* # disable these forwardedindex.1.blacklist = forwardedindex.2.whitelist =
Forward data for a single index only
If you want to forward only the data targeted for a single index (for example, as specified in inputs.conf
), and drop any data that's not target for that index, here's how to do it:
[tcpout] #Disable the current filters from the defaults outputs.conf forwardedindex.0.whitelist = forwardedindex.1.blacklist = forwardedindex.2.whitelist = #Forward data for the "myindex" index forwardedindex.0.whitelist = myindex
This first disables all filters from the default outputs.conf
file. It then sets the filter for your own index. Be sure to start the filter numbering with 0: forwardedindex.0
.
Note: If you've set other filters in another copy of outputs.conf
on your system, you must disable those as well.
You can use the CLI btools
command to ensure that there aren't any other filters located in other outputs.conf
files on your system:
splunk cmd btool outputs list tcpout
This command returns the content of the tcpout
stanza, after all versions of the configuration file have been combined.
Use the forwardedindex attributes with local indexing
On a heavy forwarder, you can index locally. To do that, you must set indexAndForward
attribute to "true". Otherwise, the forwarder just forwards your data and does not save it on the forwarder. On the other hand, the forwardedindex
attributes only filter forwarded data; they do not filter any data that gets saved to the local index.
In a nutshell, local indexing and forwarder filtering are entirely separate operations, which do not coordinate with each other. This can have unexpected implications when you're performing blacklist filtering:
- If you set
indexAndForward
to "true" and then filter out some data throughforwardedindex
blacklist attributes, the blacklisted data will not get forwarded, but it will still get locally indexed. - If you set
indexAndForward
to "false" (no local indexing) and then filter out some data, the filtered data will get dropped entirely, since it doesn't get forwarded and it doesn't get saved (indexed) on the forwarder.
Route inputs to specific indexers based on the data's input
There is one type of routing that doesn't require a heavy forwarder. In this scenario, you use inputs.conf
and outputs.conf
to route data to specific indexers, based on the data's input.
Here's an example that shows how this works.
1. In outputs.conf
, you create stanzas for each receiving indexer:
[tcpout:systemGroup] server=server1:9997 [tcpout:applicationGroup] server=server2:9997
2. In inputs.conf
, you use _TCP_ROUTING
to specify the stanza in outputs.conf
that each input should use for routing:
[monitor://.../file1.log] _TCP_ROUTING = systemGroup [monitor://.../file2.log] _TCP_ROUTING = applicationGroup
The forwarder will route data from file1.log
to server1
and data from file2.log
to server2
.
Perform selective indexing and forwarding
With a heavy forwarder only, you can index and store data locally, as well as forward the data onwards to a receiving indexer. There are two ways to do this:
- Index all the data before forwarding it. To do this, just enable the
indexAndForward
attribute inoutputs.conf
.
- Index a subset of the data before forwarding it or other data. This is called selective indexing. With selective indexing, you can index just some of the data locally and then forward it on to a receiving indexer. Alternatively, you can choose to forward only the data that you don't index locally.
Important: Do not enable the indexAndForward
attribute in the [tcpout]
stanza if you're also enabling selective indexing.
Configure selective indexing
To use selective indexing, you need to modify both your inputs.conf
and outputs.conf
files:
1. In outputs.conf
:
a. Add the [indexAndForward]
stanza:
[indexAndForward] index=true selectiveIndexing=true
The presence of this stanza, including the index
and selectiveIndexing
attributes, turns on selective indexing for the forwarder. It enables local indexing for any input (specified in inputs.conf
) that has the _INDEX_AND_FORWARD_ROUTING
attribute. Use the entire [indexAndForward]
stanza exactly as shown here.
Note: This is a global stanza, which only needs to appear once in outputs.conf
.
b. Include the usual target group stanzas for each set of receiving indexers:
[tcpout:<target_group>] server = <ip address>:<port>, <ip address>:<port>, ... ...
The named <target_group>
is used in inputs.conf
to route the inputs, as described below.
2. In inputs.conf
:
a. Add the _INDEX_AND_FORWARD_ROUTING
attribute to the stanzas of each input that you want to index locally:
[input_stanza] _INDEX_AND_FORWARD_ROUTING=<any_string> ...
The presence of the _INDEX_AND_FORWARD_ROUTING
attribute tells the heavy forwarder to index that input locally. You can set the attribute to any string value you want. The forwarder just looks for the attribute itself; the string value has no effect at all on behavior.
b. Add the _TCP_ROUTING
attribute to the stanzas of each input that you want to forward:
[input_stanza] _TCP_ROUTING=<target_group> ...
The <target_group>
is the name used in outputs.conf
to specify the target group of receiving indexers.
The next several sections show how to use selective indexing in a variety of scenarios.
Index one input locally and then forward the remaining inputs
In this example, the forwarder indexes data from one input locally but does not forward it. It also forwards data from two other inputs but does not index those inputs locally.
1. In outputs.conf
, create these stanzas:
[tcpout] defaultGroup=noforward disabled=false [indexAndForward] index=true selectiveIndexing=true [tcpout:indexerB_9997] server = indexerB:9997 [tcpout:indexerC_9997] server = indexerC:9997
Since the defaultGroup
is set to the non-existent group "noforward" (meaning that there is no defaultGroup
), the forwarder will only forward data that's been routed to explicit target groups in inputs.conf
. All other data will get dropped.
2. In inputs.conf
, create these stanzas:
[monitor:///mydata/source1.log] _INDEX_AND_FORWARD_ROUTING=local [monitor:///mydata/source2.log] _TCP_ROUTING=indexerB_9997 [monitor:///mydata/source3.log] _TCP_ROUTING=indexerC_9997
The result is that the forwarder:
- indexes the
source1.log
data locally but does not forward it (because there's no explicit routing in its input stanza and there's no default group inoutputs.conf
). - forwards the
source2.log
data to indexerB but does not index it locally. - forwards the
source3.log
data to indexerC but does not index it locally.
Index one input locally and then forward all inputs
This example is nearly identical to the previous one. The difference is that here, you index just one input locally, but then you forward all inputs, including the one you've indexed locally.
1. In outputs.conf
, create these stanzas:
[tcpout] defaultGroup=noforward disabled=false [indexAndForward] index=true selectiveIndexing=true [tcpout:indexerB_9997] server = indexerB:9997 [tcpout:indexerC_9997] server = indexerC:9997
This outputs.conf
is identical to the previous example.
2. In inputs.conf
, create these stanzas:
[monitor:///mydata/source1.log] _INDEX_AND_FORWARD_ROUTING=local _TCP_ROUTING=indexerB_9997 [monitor:///mydata/source2.log] _TCP_ROUTING=indexerB_9997 [monitor:///mydata/source3.log] _TCP_ROUTING=indexerC_9997
The only difference from the previous example is that here, you've specified the _TCP_ROUTING
attribute for the input that you're indexing locally. The forwarder will route both source1.log
and source2.log
to the indexerB_9997
target group, but it will only locally index the data from source1.log
.
Another way to index one input locally and then forward all inputs
You can achieve the same result as in the previous example by setting the defaultGroup
to a real target group.
1. In outputs.conf
, create these stanzas:
[tcpout] defaultGroup=indexerB_9997 disabled=false [indexAndForward] index=true selectiveIndexing=true [tcpout:indexerB_9997] server = indexerB:9997 [tcpout:indexerC_9997] server = indexerC:9997
This outputs.conf
sets the defaultGroup
to indexerB_9997
.
2. In inputs.conf
, create these stanzas:
[monitor:///mydata/source1.log] _INDEX_AND_FORWARD_ROUTING=local [monitor:///mydata/source2.log] _TCP_ROUTING=indexerB_9997 [monitor:///mydata/source3.log] _TCP_ROUTING=indexerC_9997
Even though you haven't set up an explicit routing for source1.log
, the forwarder will still forward it to the indexerB_9997
target group, since outputs.conf
specifies that group as the defaultGroup
.
Selective indexing and internal logs
Once you enable selective indexing in outputs.conf
, the forwarder will only index locally those inputs with the _INDEX_AND_FORWARD_ROUTING
attribute. This applies to the internal logs in the /var/log/splunk
directory (specified in the default etc/system/default/inputs.conf
). By default, they will not be indexed. If you want to index them, you must add their input stanza to your local inputs.conf
file (which takes precedence over the default file) and include the _INDEX_AND_FORWARD_ROUTING
attribute:
[monitor://$SPLUNK_HOME/var/log/splunk] index = _internal _INDEX_AND_FORWARD_ROUTING=local
Caveats for routing and filtering structured data
Splunk Enterprise does not parse structured data that has been forwarded to an indexer
When you forward structured data to an indexer, Splunk Enterprise does not parse this data once it arrives at the indexer, even if you have configured props.conf
on that indexer with INDEXED_EXTRACTIONS
. Forwarded data skips the following queues on the indexer, which precludes any parsing of that data on the indexer:
parsing
aggregation
typing
The forwarded data must arrive at the indexer already parsed. To achieve this, you must also set up props.conf
on the forwarder that sends the data. This includes configuration of INDEXED_EXTRACTIONS
and any other parsing, filtering, anonymizing, and routing rules. Universal forwarders are capable of performing these tasks solely for structured data. See "Forward data extracted from header files".
PREVIOUS Protect against loss of in-flight data |
NEXT Make a universal forwarder part of a system image |
This documentation applies to the following versions of Splunk® Enterprise: 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.3.14
Feedback submitted, thanks!