Forwarding Data

Route and filter data

Forwarders can filter and route data to specific receivers based on criteria such as source, source type, or patterns in the events themselves. For example, a forwarder can send all data from one group of hosts to one indexer and all data from a second group of hosts to a second indexer. Heavy forwarders can also look inside the events and filter or route accordingly. For example, you could use a heavy forwarder to inspect WMI event codes to filter or route Windows events. This topic describes a number of typical routing scenarios.

Besides routing to receivers, forwarders can also filter and route data to specific queues or discard the data altogether by routing to the null queue.

Important: Only heavy forwarders can route or filter all data at the event level. Universal forwarders and light forwarders do not have the ability to inspect individual events except in the case of file header extractions, but they can still forward data based on a data stream's host, source, or source type. They can also route based on the data's input stanza, as described below, in the subtopic, "Route inputs to specific indexers based on the data's input". Some input types have the ability to filter out some types of data while acquiring them (see below).

Here's a simple illustration of a forwarder routing data to three indexers:

This topic describes how to filter and route event data to Splunk Enterprise instances. See "Forward data to third-party systems" in this manual for information on routing to non-Splunk systems.

This topic also describes how to perform selective indexing and forwarding, in which you index some data locally on a heavy forwarder and forward the non-indexed data to one or more separate indexers. See "Perform selective indexing and forwarding" later in this topic for details.

Configure routing

This is the basic pattern for defining most routing scenarios (using a heavy forwarder):

1. Determine what criteria to use for routing. How will you identify categories of events, and where will you route them?

2. Edit props.conf to add a TRANSFORMS-routing attribute to determine routing based on event metadata:

[<spec>]
TRANSFORMS-routing=<transforms_stanza_name>

Note the following:

• <spec> can be:
  • <sourcetype>, the source type of an event
  • host::<host>, where <host> is the host for an event
  • source::<source>, where <source> is the source for an event
• If you have multiple TRANSFORMS attributes, use a unique name for each. For example: "TRANSFORMS-routing1", "TRANSFORMS-routing2", and so on.
• <transforms_stanza_name> must be unique.

Use the <transforms_stanza_name> specified here when creating an entry in transforms.conf (below).

Examples later in this topic show how to use this syntax.

3. Edit transforms.conf to specify target groups and to set additional criteria for routing based on event patterns:

[<transforms_stanza_name>]
REGEX=<routing_criteria>
DEST_KEY=_TCP_ROUTING
FORMAT=<target_group>,<target_group>,....

Note:

• <transforms_stanza_name> must match the name you defined in props.conf.
• In <routing_criteria>, enter the regex rules that determine which events get routed. This line is required. Use "REGEX = ." if you don't need additional filtering beyond the metadata specified in props.conf.
• DEST_KEY should be set to _TCP_ROUTING to send events via TCP. It can also be set to _SYSLOG_ROUTING or _HTTPOUT_ROUTING for other output processors.
• Set FORMAT to a <target_group> that matches a target group name you defined in outputs.conf. A comma separated list will clone events to multiple target groups.

Examples later in this topic show how to use this syntax.

4. Edit outputs.conf to define the target group(s) for the routed data:

[tcpout:<target_group>]
server=<ip>:<port>

Note:

• Set <target_group> to match the name you specified in transforms.conf.
• Set the IP address and port to match the receiving server.

The use cases described in this topic generally follow this pattern.

Filter and route event data to target groups

In this example, a heavy forwarder filters three types of events, routing them to different target groups. The forwarder filters and routes according to these criteria:

• Events with a source type of "syslog" to a load-balanced target group
• Events containing the word "error" to a second target group
• All other events to a default target group

Here's how you do it:

1. Edit props.conf in $SPLUNK_HOME/etc/system/local to set two TRANSFORMS-routing attributes — one for syslog data and a default for all other data:

[default]
TRANSFORMS-routing=errorRouting

[syslog]
TRANSFORMS-routing=syslogRouting

2. Edit transforms.conf to set the routing rules for each routing transform:

[errorRouting]
REGEX=error
DEST_KEY=_TCP_ROUTING
FORMAT=errorGroup

[syslogRouting]
REGEX=.
DEST_KEY=_TCP_ROUTING
FORMAT=syslogGroup

Note: In this example, if a syslog event contains the word "error", it will route to syslogGroup, not errorGroup. This is due to the settings previously specified in props.conf. Those settings dictated that all syslog events be filtered through the syslogRouting transform, while all non-syslog (default) events be filtered through the errorRouting transform. Therefore, only non-syslog events get inspected for errors.

3. Edit outputs.conf to define the target groups:

[tcpout]
defaultGroup=everythingElseGroup

[tcpout:syslogGroup]
server=10.1.1.197:9996, 10.1.1.198:9997

[tcpout:errorGroup]
server=10.1.1.200:9999

[tcpout:everythingElseGroup]
server=10.1.1.250:6666

syslogGroup and errorGroup receive events according to the rules specified in transforms.conf. All other events get routed to the default group, everythingElseGroup.

Replicate a subset of data to a third-party system

This example uses data filtering to route two data streams. It forwards:

• All the data, in cooked form, to a Splunk Enterprise indexer (10.1.12.1:9997)
• A replicated subset of the data, in raw form, to a third-party server (10.1.12.2:1234)

The example sends both streams as TCP. To send the second stream as syslog data, first route the data through an indexer.

1. Edit props.conf:

[syslog]
TRANSFORMS-routing = routeAll, routeSubset

2. Edit transforms.conf:

[routeAll]
REGEX=(.)
DEST_KEY=_TCP_ROUTING
FORMAT=Everything

[routeSubset]
REGEX=(SYSTEM|CONFIG|THREAT)
DEST_KEY=_TCP_ROUTING
FORMAT=Subsidiary,Everything

3. Edit outputs.conf:

[tcpout]
defaultGroup=nothing

[tcpout:Everything]
disabled=false
server=10.1.12.1:9997

[tcpout:Subsidiary]
disabled=false
sendCookedData=false
server=10.1.12.2:1234

For more information, see "Forward data to third party systems" in this manual.

Filter event data and send to queues

Although similar to forwarder-based routing, queue routing can be performed by an indexer, as well as a heavy forwarder. It does not use the outputs.conf file, just props.conf and transforms.conf.

You can eliminate unwanted data by routing it to nullQueue, the Splunk Enterprise equivalent of the /dev/null device. When you filter out data in this way, the filtered data is not forwarded or added to the Splunk Enterprise index at all, and doesn't count toward your indexing volume.

Note: See "Caveats for routing and filtering structured data" later in this topic.

Discard specific events and keep the rest

This example discards all sshd events in /var/log/messages by sending them to nullQueue:

1. In props.conf, set the TRANSFORMS-null attribute:

[source::/var/log/messages]
TRANSFORMS-null= setnull

2. Create a corresponding stanza in transforms.conf. Set DEST_KEY to "queue" and FORMAT to "nullQueue":

[setnull]
REGEX = \[sshd\]
DEST_KEY = queue
FORMAT = nullQueue

That does it.

Keep specific events and discard the rest

Here's the opposite scenario. In this example, you use two transforms to keep only the sshd events. One transform routes sshd events to indexQueue, while another routes all other events to nullQueue.

Note: In this example, the order of the transforms in props.conf matters. The null queue transform must come first; if it comes later, it will invalidate the previous transform and route all events to the null queue.

1. In props.conf:

[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing

2. In transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = \[sshd\]
DEST_KEY = queue
FORMAT = indexQueue

Filter WMI and Event Log events

You can filter WinEventLog directly at the forwarder level.

Otherwise to filter on WMI events, use [WMI:WinEventLog:Security] source type stanza in props.conf. The following example uses regex to filter out two Windows event codes, 592 and 593:

In props.conf:

[WinEventLog:Security]
TRANSFORMS-wmi=wminull

Note: In pre-4.2.x versions of Splunk Enterprise, you must use [wmi] or [WMI::WinEventLog:Security] as the source type in order to send events to nullQueue.

In transforms.conf:

[wminull]
REGEX=(?m)^EventCode=(592|593)
DEST_KEY=queue
FORMAT=nullQueue

Filter data by target index

Forwarders have a forwardedindex filter that allows you to specify whether data gets forwarded, based on the data's target index. For example, if you have one data input targeted to "index1" and another targeted to "index2", you can use the filter to forward only the data targeted to index1, while ignoring the index2 data. The forwardedindex filter uses whitelists and blacklists to specify the filtering. For information on setting up multiple indexes, see the topic "Set up multiple indexes".

Note: The forwardedindex filter is only applicable under the global [tcpout] stanza. This filter does not work if it is created under any other stanza in outputs.conf.

Use the forwardedindex.<n>.whitelist|blacklist attributes in outputs.conf to specify which data should get forwarded on an index-by-index basis. You set the attributes to regexes that filter the target indexes.

Default behavior

By default, the forwarder forwards data targeted for all external indexes, including the default index and any user-created indexes. Regarding data for internal indexes, the default behavior varies according to who is doing the forwarding:

• The universal forwarder forwards the data for the _audit internal index only. It does not forward data for other internal indexes. Its default outputs.conf file located in $SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/default specifies that behavior with these attributes:

   [tcpout]
   forwardedindex.0.whitelist = .*
   forwardedindex.1.blacklist = _.*
   forwardedindex.2.whitelist = _audit

• Heavy forwarder and full Splunk instances with forwarding enabled (for example, a search head with forwarding enabled) forward the data for the _audit and _internal internal indexes. Their default outputs.conf file, located in $SPLUNK_HOME/etc/system/default, specifies that behavior with these attributes:

   [tcpout]
   forwardedindex.0.whitelist = .*
   forwardedindex.1.blacklist = _.*
   forwardedindex.2.whitelist = (_audit|_internal)

Note: The default behavior for heavy forwarders and full Splunk Enterprise instances changed in Splunk Enterprise version 5.0.2. In earlier versions, the _internal index was not forwarded by default. Those forwarder types had the same behavior as the universal forwarder: only data for the _audit internal index was forwarded.

In most deployments, you will not need to override the default settings. See outputs.conf for more information on how to whitelist and blacklist indexes. For more information on default and custom outputs.conf files and their locations, see "Types of outputs.conf files".

Forward all external and internal index data

If you want to forward all internal index data, as well as all external data, you can override the default forwardedindex filter attributes like this:

#Forward everything
[tcpout]
forwardedindex.0.whitelist = .*
# disable these
forwardedindex.1.blacklist =
forwardedindex.2.whitelist =

Forward data for a single index only

If you want to forward only the data targeted for a single index (for example, as specified in inputs.conf), and drop any data that's not target for that index, here's how to do it:

[tcpout]
#Disable the current filters from the defaults outputs.conf
forwardedindex.0.whitelist = 
forwardedindex.1.blacklist =
forwardedindex.2.whitelist =

#Forward data for the "myindex" index
forwardedindex.0.whitelist = myindex

This first disables all filters from the default outputs.conf file. It then sets the filter for your own index. Be sure to start the filter numbering with 0: forwardedindex.0.

Note: If you've set other filters in another copy of outputs.conf on your system, you must disable those as well.

You can use the CLI btools command to ensure that there aren't any other filters located in other outputs.conf files on your system:

splunk cmd btool outputs list tcpout

This command returns the content of the tcpout stanza, after all versions of the configuration file have been combined.

Use the forwardedindex attributes with local indexing

On a heavy forwarder, you can index locally. To do that, you must set indexAndForward attribute to "true". Otherwise, the forwarder just forwards your data and does not save it on the forwarder. On the other hand, the forwardedindex attributes only filter forwarded data; they do not filter any data that gets saved to the local index.

In a nutshell, local indexing and forwarder filtering are entirely separate operations, which do not coordinate with each other. This can have unexpected implications when you're performing blacklist filtering:

• If you set indexAndForward to "true" and then filter out some data through forwardedindex blacklist attributes, the blacklisted data will not get forwarded, but it will still get locally indexed.
• If you set indexAndForward to "false" (no local indexing) and then filter out some data, the filtered data will get dropped entirely, since it doesn't get forwarded and it doesn't get saved (indexed) on the forwarder.

Route inputs to specific indexers based on the data's input

There is one type of routing that doesn't require a heavy forwarder. In this scenario, you use inputs.conf and outputs.conf to route data to specific indexers, based on the data's input.

Here's an example that shows how this works.

1. In outputs.conf, you create stanzas for each receiving indexer:

[tcpout:systemGroup]
server=server1:9997

[tcpout:applicationGroup]
server=server2:9997

2. In inputs.conf, you use _TCP_ROUTING to specify the stanza in outputs.conf that each input should use for routing:

[monitor://.../file1.log]
_TCP_ROUTING = systemGroup

[monitor://.../file2.log]
_TCP_ROUTING = applicationGroup

The forwarder will route data from file1.log to server1 and data from file2.log to server2.

Perform selective indexing and forwarding

With a heavy forwarder only, you can index and store data locally, as well as forward the data onwards to a receiving indexer. There are two ways to do this:

• Index all the data before forwarding it. To do this, just enable the indexAndForward attribute in outputs.conf.

• Index a subset of the data before forwarding it or other data. This is called selective indexing. With selective indexing, you can index just some of the data locally and then forward it on to a receiving indexer. Alternatively, you can choose to forward only the data that you don't index locally.

Important: Do not enable the indexAndForward attribute in the [tcpout] stanza if you're also enabling selective indexing.

Configure selective indexing

To use selective indexing, you need to modify both your inputs.conf and outputs.conf files:

1. In outputs.conf:

a. Add the [indexAndForward] stanza:

[indexAndForward]
index=true
selectiveIndexing=true 

The presence of this stanza, including the index and selectiveIndexing attributes, turns on selective indexing for the forwarder. It enables local indexing for any input (specified in inputs.conf) that has the _INDEX_AND_FORWARD_ROUTING attribute. Use the entire [indexAndForward] stanza exactly as shown here.

Note: This is a global stanza, which only needs to appear once in outputs.conf.

b. Include the usual target group stanzas for each set of receiving indexers:

[tcpout:<target_group>]
server = <ip address>:<port>, <ip address>:<port>, ...
... 

The named <target_group> is used in inputs.conf to route the inputs, as described below.

2. In inputs.conf:

a. Add the _INDEX_AND_FORWARD_ROUTING attribute to the stanzas of each input that you want to index locally:

[input_stanza]
_INDEX_AND_FORWARD_ROUTING=<any_string>
...

The presence of the _INDEX_AND_FORWARD_ROUTING attribute tells the heavy forwarder to index that input locally. You can set the attribute to any string value you want. The forwarder just looks for the attribute itself; the string value has no effect at all on behavior.

b. Add the _TCP_ROUTING attribute to the stanzas of each input that you want to forward:

[input_stanza]
_TCP_ROUTING=<target_group>
...

The <target_group> is the name used in outputs.conf to specify the target group of receiving indexers.

The next several sections show how to use selective indexing in a variety of scenarios.

Index one input locally and then forward the remaining inputs

In this example, the forwarder indexes data from one input locally but does not forward it. It also forwards data from two other inputs but does not index those inputs locally.

1. In outputs.conf, create these stanzas:

[tcpout]
defaultGroup=noforward
disabled=false

[indexAndForward]
index=true
selectiveIndexing=true

[tcpout:indexerB_9997]
server = indexerB:9997

[tcpout:indexerC_9997]
server = indexerC:9997

Since the defaultGroup is set to the non-existent group "noforward" (meaning that there is no defaultGroup), the forwarder will only forward data that's been routed to explicit target groups in inputs.conf. All other data will get dropped.

2. In inputs.conf, create these stanzas:

[monitor:///mydata/source1.log]
_INDEX_AND_FORWARD_ROUTING=local

[monitor:///mydata/source2.log]
_TCP_ROUTING=indexerB_9997

[monitor:///mydata/source3.log]
_TCP_ROUTING=indexerC_9997

The result is that the forwarder:

• indexes the source1.log data locally but does not forward it (because there's no explicit routing in its input stanza and there's no default group in outputs.conf).
• forwards the source2.log data to indexerB but does not index it locally.
• forwards the source3.log data to indexerC but does not index it locally.

Index one input locally and then forward all inputs

This example is nearly identical to the previous one. The difference is that here, you index just one input locally, but then you forward all inputs, including the one you've indexed locally.

1. In outputs.conf, create these stanzas:

[tcpout]
defaultGroup=noforward
disabled=false

[indexAndForward]
index=true
selectiveIndexing=true

[tcpout:indexerB_9997]
server = indexerB:9997

[tcpout:indexerC_9997]
server = indexerC:9997

This outputs.conf is identical to the previous example.

2. In inputs.conf, create these stanzas:

[monitor:///mydata/source1.log]
_INDEX_AND_FORWARD_ROUTING=local
_TCP_ROUTING=indexerB_9997

[monitor:///mydata/source2.log]
_TCP_ROUTING=indexerB_9997

[monitor:///mydata/source3.log]
_TCP_ROUTING=indexerC_9997

The only difference from the previous example is that here, you've specified the _TCP_ROUTING attribute for the input that you're indexing locally. The forwarder will route both source1.log and source2.log to the indexerB_9997 target group, but it will only locally index the data from source1.log.

Another way to index one input locally and then forward all inputs

You can achieve the same result as in the previous example by setting the defaultGroup to a real target group.

1. In outputs.conf, create these stanzas:

[tcpout]
defaultGroup=indexerB_9997
disabled=false

[indexAndForward]
index=true
selectiveIndexing=true

[tcpout:indexerB_9997]
server = indexerB:9997

[tcpout:indexerC_9997]
server = indexerC:9997

This outputs.conf sets the defaultGroup to indexerB_9997.

2. In inputs.conf, create these stanzas:

[monitor:///mydata/source1.log]
_INDEX_AND_FORWARD_ROUTING=local

[monitor:///mydata/source2.log]
_TCP_ROUTING=indexerB_9997

[monitor:///mydata/source3.log]
_TCP_ROUTING=indexerC_9997

Even though you haven't set up an explicit routing for source1.log, the forwarder will still forward it to the indexerB_9997 target group, since outputs.conf specifies that group as the defaultGroup.

Selective indexing and internal logs

Once you enable selective indexing in outputs.conf, the forwarder will only index locally those inputs with the _INDEX_AND_FORWARD_ROUTING attribute. This applies to the internal logs in the /var/log/splunk directory (specified in the default etc/system/default/inputs.conf). By default, they will not be indexed. If you want to index them, you must add their input stanza to your local inputs.conf file (which takes precedence over the default file) and include the _INDEX_AND_FORWARD_ROUTING attribute:

[monitor://$SPLUNK_HOME/var/log/splunk]
index = _internal
_INDEX_AND_FORWARD_ROUTING=local

Caveats for routing and filtering structured data

Splunk Enterprise does not parse structured data that has been forwarded to an indexer

When you forward structured data to an indexer, Splunk Enterprise does not parse this data once it arrives at the indexer, even if you have configured props.conf on that indexer with INDEXED_EXTRACTIONS. Forwarded data skips the following queues on the indexer, which precludes any parsing of that data on the indexer:

• parsing
• aggregation
• typing

The forwarded data must arrive at the indexer already parsed. To achieve this, you must also set up props.conf on the forwarder that sends the data. This includes configuration of INDEXED_EXTRACTIONS and any other parsing, filtering, anonymizing, and routing rules. Universal forwarders are capable of performing these tasks solely for structured data. See "Forward data extracted from header files".