Splunk® Enterprise

Forwarding Data

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Route and filter data

This topic describes how to filter and route event data to specific Splunk instances. See "Forward data to third-party systems" in this manual for information on routing to non-Splunk systems.

This topic also describes how to perform selective indexing and forwarding, in which you index some data locally on a heavy forwarder and forward the non-indexed data to one or more separate indexers. See "Perform selective indexing and forwarding" later in this topic for details.

Routing and filtering capabilities of forwarders

Heavy forwarders can filter and route data to specific receivers based on criteria such as source, source type, or patterns in the events themselves. For example, it can send all data from one group of hosts to one indexer and all data from a second group of hosts to a second indexer. Heavy forwarders can also look inside the events and filter or route accordingly. For example, you could use a heavy forwarder to inspect WMI event codes to filter or route Windows events. This topic describes a number of typical routing scenarios.

Besides routing to receivers, forwarders can also filter and route data to specific queues or discard the data altogether by routing to the null queue.

Only heavy forwarders can route or filter all data at the event level. Universal forwarders and light forwarders do not have the ability to inspect individual events except in the case of extracting fields from files with structured data, but they can still forward data based on a host, source, or source type. They can also route based on the data's input stanza, as described below, in the subtopic, "Route inputs to specific indexers based on the data's input". Some input types have the ability to filter out some types of data while acquiring them (see below).

A simple illustration of a forwarder routing data to three indexers follows:

30 admin13 forwardreceive-datarouting 60.png

Configure routing

These instructions can only be performed on a heavy forwarder.

1. Determine the criteria to use for routing by answering the following questions:

  • How will you identify categories of events?
  • Where will you route the events?

2. On the instance that does the routing, open a shell or command prompt.

3. Edit props.conf to add a TRANSFORMS-routing attribute to determine routing based on event metadata:

[<spec>]
TRANSFORMS-routing=<transforms_stanza_name>

In this props.con stanza:

  • <spec> can be:
    • <sourcetype>, the source type of an event
    • host::<host>, where <host> is the host for an event
    • source::<source>, where <source> is the source for an event
  • If you have multiple TRANSFORMS attributes, use a unique name for each. For example: "TRANSFORMS-routing1", "TRANSFORMS-routing2", and so on.
  • <transforms_stanza_name> must be unique.

Use the <transforms_stanza_name> specified here when creating an entry in transforms.conf (below).

Examples later in this topic show how to use this syntax.

3. Edit transforms.conf to specify target groups and to set additional criteria for routing based on event patterns:

[<transforms_stanza_name>]
REGEX=<routing_criteria>
DEST_KEY=_TCP_ROUTING
FORMAT=<target_group>,<target_group>,....

In this transforms.con stanza:

  • <transforms_stanza_name> must match the name you defined in props.conf.
  • In <routing_criteria>, enter the regular expression rules that determine which events get routed. This line is required. Use "REGEX = ." if you don't need additional filtering beyond the metadata specified in props.conf.
  • DEST_KEY should be set to _TCP_ROUTING to send events via TCP. It can also be set to _SYSLOG_ROUTING or _HTTPOUT_ROUTING for other output processors.
  • Set FORMAT to a <target_group> that matches a target group name you defined in outputs.conf. If you specify more than one target group, use commas to separate them. A comma separated list clones events to multiple target groups.

Examples later in this topic show how to use this syntax.

4. Edit outputs.conf to define the target group(s) for the routed data:

[tcpout:<target_group>]
server=<ip>:<port>

In this outputs.conf stanza:

  • Set <target_group> to match the name you specified in transforms.conf.
  • Set the IP address and port to match the receiving server.

The use cases described in this topic generally follow this pattern.

Filter and route event data to target groups

In this example, a heavy forwarder filters three types of events and routes them to different target groups. The forwarder filters and routes according to these criteria:

  • Events with a source type of "syslog" to a load-balanced target group
  • Events containing the word "error" to a second target group
  • All other events to a default target group

1. On the instance that is to do the routing, open a command or shell prompt.

2. Edit props.conf in $SPLUNK_HOME/etc/system/local to set two TRANSFORMS-routing attributes — one for syslog data and a default for all other data:

[default]
TRANSFORMS-routing=errorRouting

[syslog]
TRANSFORMS-routing=syslogRouting

3. Edit transforms.conf to set the routing rules for each routing transform:

[errorRouting]
REGEX=error
DEST_KEY=_TCP_ROUTING
FORMAT=errorGroup

[syslogRouting]
REGEX=.
DEST_KEY=_TCP_ROUTING
FORMAT=syslogGroup

In this example, if a syslog event contains the word "error", it routes to syslogGroup, not errorGroup. This is due to the settings you previously specified in props.conf. Those settings dictated that all syslog events should be filtered through the syslogRouting transform, while all non-syslog (default) events should be filtered through the errorRouting transform. Therefore, only non-syslog events get inspected for errors.

4. Edit outputs.conf to define the target groups:

[tcpout]
defaultGroup=everythingElseGroup

[tcpout:syslogGroup]
server=10.1.1.197:9996, 10.1.1.198:9997

[tcpout:errorGroup]
server=10.1.1.200:9999

[tcpout:everythingElseGroup]
server=10.1.1.250:6666

syslogGroup and errorGroup receive events according to the rules specified in transforms.conf. All other events get routed to the default group, everythingElseGroup.

Replicate a subset of data to a third-party system

This example uses data filtering to route two data streams. It forwards:

  • All the data, in cooked form, to a Splunk indexer (10.1.12.1:9997)
  • A replicated subset of the data, in raw form, to a third-party host (10.1.12.2:1234)

The example sends both streams as TCP. To send the second stream as syslog data, first route the data through an indexer.

1. Edit props.conf:

[syslog]
TRANSFORMS-routing = routeAll, routeSubset

2. Edit transforms.conf:

[routeAll]
REGEX=(.)
DEST_KEY=_TCP_ROUTING
FORMAT=Everything

[routeSubset]
REGEX=(SYSTEM|CONFIG|THREAT)
DEST_KEY=_TCP_ROUTING
FORMAT=Subsidiary,Everything

3. Edit outputs.conf:

[tcpout]
defaultGroup=nothing

[tcpout:Everything]
disabled=false
server=10.1.12.1:9997

[tcpout:Subsidiary]
disabled=false
sendCookedData=false
server=10.1.12.2:1234

Filter event data and send to queues

Although similar to forwarder-based routing, queue routing can be performed by an indexer or a heavy forwarder. It does not use the outputs.conf file, only props.conf and transforms.conf.

You can eliminate unwanted data by routing it to nullQueue, the Splunk equivalent of the /dev/null device. Data routed to nullQueue is discarded and doesn't count toward your indexing volume.

See "Caveats for routing and filtering structured data" later in this topic.

Discard specific events and keep the rest

This example discards all sshd events in /var/log/messages by sending them to nullQueue:

1. In props.conf, set the TRANSFORMS-null attribute:

[source::/var/log/messages]
TRANSFORMS-null= setnull

2. Create a corresponding stanza in transforms.conf. Set DEST_KEY to "queue" and FORMAT to "nullQueue":

[setnull]
REGEX = \[sshd\]
DEST_KEY = queue
FORMAT = nullQueue

3. Restart the Splunk instance.

Keep specific events and discard the rest

This scenario is opposite to the previous scenario. In this example, you use two transforms to keep only the sshd events. One transform routes sshd events to indexQueue, while another routes all other events to nullQueue.

Note: In this example, the order of the transforms in props.conf matters. The null queue transform must come first; if it comes later, it will invalidate the previous transform and route all events to the null queue.

1. In props.conf:

[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing

2. In transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = \[sshd\]
DEST_KEY = queue
FORMAT = indexQueue

3. Restart the Splunk instance.

Filter WMI and Event Log events

You can filter WinEventLog directly at the forwarder level.

Otherwise to filter on WMI events, use [WMI:WinEventLog:Security] source type stanza in props.conf. The following example uses regular expressions to filter out two Windows event codes, 592 and 593:

1. Edit props.conf:

[WinEventLog:Security]
TRANSFORMS-wmi=wminull

2. Edit transforms.conf:

[wminull]
REGEX=(?m)^EventCode=(592|593)
DEST_KEY=queue
FORMAT=nullQueue

3. Restart the Splunk instance.

Filter data by target index

Forwarders have a forwardedindex filter that lets you specify whether data gets forwarded, based on the data's target index. For example, if you have one data input targeted to "index1" and another targeted to "index2", you can use the filter to forward only the data targeted to index1, while ignoring the index2 data. The forwardedindex filter uses whitelists and blacklists to specify the filtering. For information on setting up multiple indexes, see the topic Set up multiple indexes in the Managing Indexers and Clusters of Indexers manual.

Note: The forwardedindex filter is only applicable under the global [tcpout] stanza. This filter does not work if you create it under any other stanza in outputs.conf.

Use the forwardedindex.<n>.whitelist|blacklist attributes in outputs.conf to specify which data should get forwarded on an index-by-index basis. You set the attributes to regexes that filter the target indexes.

Default behavior

By default, the forwarder forwards data targeted for all external indexes, including the default index and any user-created indexes. Regarding data for internal indexes, the default behavior varies according to who is doing the forwarding:

  • The universal forwarder forwards the data for the _audit internal index only. It does not forward data for other internal indexes. Its default outputs.conf file located in $SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/default specifies that behavior with these attributes:
   [tcpout]
   forwardedindex.0.whitelist = .*
   forwardedindex.1.blacklist = _.*
   forwardedindex.2.whitelist = _audit
  • Heavy forwarder and full Splunk instances with forwarding enabled (for example, a search head with forwarding enabled) forward the data for the _audit and _internal internal indexes. Their default outputs.conf file, located in $SPLUNK_HOME/etc/system/default, specifies that behavior with these attributes:
   [tcpout]
   forwardedindex.0.whitelist = .*
   forwardedindex.1.blacklist = _.*
   forwardedindex.2.whitelist = (_audit|_internal)

In most deployments, you do not need to override the default settings. See outputs.conf for more information on how to whitelist and blacklist indexes. For more information on default and custom outputs.conf files and their locations, see "Types of outputs.conf files".

Forward all external and internal index data

If you want to forward all internal index data, as well as all external data, you can override the default forwardedindex filter attributes:

#Forward everything
[tcpout]
forwardedindex.0.whitelist = .*
# disable these
forwardedindex.1.blacklist =
forwardedindex.2.whitelist =

Forward data for a single index only

If you want to forward only the data targeted for a single index (for example, as specified in inputs.conf), and drop any data that is not a target for that index, configure outputs.conf in this way:

[tcpout]
#Disable the current filters from the defaults outputs.conf
forwardedindex.0.whitelist = 
forwardedindex.1.blacklist =
forwardedindex.2.whitelist =

#Forward data for the "myindex" index
forwardedindex.0.whitelist = myindex

This first disables all filters from the default outputs.conf file. It then sets the filter for your own index. Be sure to start the filter numbering with 0: forwardedindex.0.

Note: If you've set other filters in another copy of outputs.conf on your system, you must disable those as well.

You can use the CLI btools command to ensure that there aren't any other filters located in other outputs.conf files on your system:

splunk cmd btool outputs list tcpout

This command returns the content of the tcpout stanza, after all versions of the configuration file have been combined.

Use the forwardedindex attributes with local indexing

On a heavy forwarder, you can index locally. To do that, you must set the indexAndForward attribute to "true". Otherwise, the forwarder just forwards your data and does not save it on the forwarder. On the other hand, the forwardedindex attributes only filter forwarded data; they do not filter any data that gets saved to the local index.

In a nutshell, local indexing and forwarder filtering are entirely separate operations, which do not coordinate with each other. This can have unexpected implications when you performing blacklist filtering:

  • If you set indexAndForward to "true" and then filter out some data through forwardedindex blacklist attributes, the forwarder does not forward the blacklisted data, but it does still index the data.
  • If you set indexAndForward to "false" (no local indexing) and then filter out some data, the forwarder drops the filtered data entirely (because it neither forwards nor indexes the filtered data.)

Route inputs to specific indexers based on the data's input

In this scenario, you use inputs.conf and outputs.conf to route data to specific indexers, based on the data's input. Universal and light forwarders can perform this kind of routing.

Here's an example that shows how this works.

1. In outputs.conf, you create stanzas for each receiving indexer:

[tcpout:systemGroup]
server=server1:9997

[tcpout:applicationGroup]
server=server2:9997

2. In inputs.conf, you use _TCP_ROUTING to specify the stanza in outputs.conf that each input should use for routing:

[monitor://.../file1.log]
_TCP_ROUTING = systemGroup

[monitor://.../file2.log]
_TCP_ROUTING = applicationGroup

The forwarder routes data from file1.log to server1 and data from file2.log to server2.

Perform selective indexing and forwarding

You can index and store data locally, as well as forward the data onwards to a receiving indexer. There are two ways to do this:

  • Index all the data before forwarding it. To do this, enable the indexAndForward attribute in outputs.conf.
  • Index a subset of the data before forwarding it or other data. This is called selective indexing. With selective indexing, you can index some of the data locally and then forward it on to a receiving indexer. Alternatively, you can choose to forward only the data that you don't index locally.

Do not enable the indexAndForward attribute in the [tcpout] stanza if you also plan to enable selective indexing.

Configure selective indexing

To use selective indexing, modify both your inputs.conf and outputs.conf files:

1. In outputs.conf:

a. Add the [indexAndForward] stanza:

[indexAndForward]
index=true
selectiveIndexing=true 

The presence of this stanza, including the index and selectiveIndexing attributes, turns on selective indexing for the forwarder. It enables local indexing for any input (specified in inputs.conf) that has the _INDEX_AND_FORWARD_ROUTING attribute. Use the entire [indexAndForward] stanza exactly as shown here.

Note: This is a global stanza, which only needs to appear once in outputs.conf.

b. Include the usual target group stanzas for each set of receiving indexers:

[tcpout:<target_group>]
server = <ip address>:<port>, <ip address>:<port>, ...
... 

The named <target_group> is used in inputs.conf to route the inputs, as described below.

2. In inputs.conf:

a. Add the _INDEX_AND_FORWARD_ROUTING attribute to the stanzas of each input that you want to index locally:

[input_stanza]
_INDEX_AND_FORWARD_ROUTING=<any_string>
...

The presence of the _INDEX_AND_FORWARD_ROUTING attribute tells the heavy forwarder to index that input locally. You can set the attribute to any string value you want. The forwarder just looks for the attribute itself; the string value has no effect at all on behavior.

b. Add the _TCP_ROUTING attribute to the stanzas of each input that you want to forward:

[input_stanza]
_TCP_ROUTING=<target_group>
...

The <target_group> is the name used in outputs.conf to specify the target group of receiving indexers.

The next several sections show how to use selective indexing in a variety of scenarios.

Index one input locally and then forward the remaining inputs

In this example, the forwarder indexes data from one input locally but does not forward it. It also forwards data from two other inputs but does not index those inputs locally.

1. In outputs.conf, create these stanzas:

[tcpout]
defaultGroup=noforward
disabled=false

[indexAndForward]
index=true
selectiveIndexing=true

[tcpout:indexerB_9997]
server = indexerB:9997

[tcpout:indexerC_9997]
server = indexerC:9997

Since the defaultGroup is set to the non-existent group "noforward" (meaning that there is no defaultGroup), the forwarder only forwards data that has been routed to explicit target groups in inputs.conf. It drops all other data.

2. In inputs.conf, create these stanzas:

[monitor:///mydata/source1.log]
_INDEX_AND_FORWARD_ROUTING=local

[monitor:///mydata/source2.log]
_TCP_ROUTING=indexerB_9997

[monitor:///mydata/source3.log]
_TCP_ROUTING=indexerC_9997

The result is that the forwarder:

  • indexes the source1.log data locally but does not forward it (because there is no explicit routing in its input stanza and there is no default group in outputs.conf).
  • forwards the source2.log data to indexerB but does not index it locally.
  • forwards the source3.log data to indexerC but does not index it locally.

Index one input locally and then forward all inputs

This example is nearly identical to the previous one. The difference is that here, you index just one input locally, but then you forward all inputs, including the one you've indexed locally.

1. In outputs.conf, create these stanzas:

[tcpout]
defaultGroup=noforward
disabled=false

[indexAndForward]
index=true
selectiveIndexing=true

[tcpout:indexerB_9997]
server = indexerB:9997

[tcpout:indexerC_9997]
server = indexerC:9997

This outputs.conf is identical to the previous example.

2. In inputs.conf, create these stanzas:

[monitor:///mydata/source1.log]
_INDEX_AND_FORWARD_ROUTING=local
_TCP_ROUTING=indexerB_9997

[monitor:///mydata/source2.log]
_TCP_ROUTING=indexerB_9997

[monitor:///mydata/source3.log]
_TCP_ROUTING=indexerC_9997

The only difference from the previous example is that here, you have specified the _TCP_ROUTING attribute for the input that you are indexing locally. The forwarder routes both source1.log and source2.log to the indexerB_9997 target group, but it only locally indexes the data from source1.log.

Another way to index one input locally and then forward all inputs

You can achieve the same result as in the previous example by setting the defaultGroup to a real target group.

1. In outputs.conf, create these stanzas:

[tcpout]
defaultGroup=indexerB_9997
disabled=false

[indexAndForward]
index=true
selectiveIndexing=true

[tcpout:indexerB_9997]
server = indexerB:9997

[tcpout:indexerC_9997]
server = indexerC:9997

This outputs.conf sets the defaultGroup to indexerB_9997.

2. In inputs.conf, create these stanzas:

[monitor:///mydata/source1.log]
_INDEX_AND_FORWARD_ROUTING=local

[monitor:///mydata/source2.log]
_TCP_ROUTING=indexerB_9997

[monitor:///mydata/source3.log]
_TCP_ROUTING=indexerC_9997

Even though you haven't set up an explicit routing for source1.log, the forwarder still forwards it to the indexerB_9997 target group, since outputs.conf specifies that group as the defaultGroup.

Selective indexing and internal logs

Once you enable selective indexing in outputs.conf, the forwarder only indexes locally those inputs with the _INDEX_AND_FORWARD_ROUTING attribute. This applies to the internal logs in the /var/log/splunk directory (specified in the default etc/system/default/inputs.conf). By default, the forwarder does not index those logs. If you want to index them, you must add their input stanza to your local inputs.conf file (which takes precedence over the default file) and include the _INDEX_AND_FORWARD_ROUTING attribute:

[monitor://$SPLUNK_HOME/var/log/splunk]
index = _internal
_INDEX_AND_FORWARD_ROUTING=local

Caveats for routing and filtering structured data

Splunk indexers do not parse structured data

When you forward structured data to an indexer, the indexer does not parse it, even if you have configured props.conf on that indexer with INDEXED_EXTRACTIONS and its associated attributes. Forwarded data skips the following queues on the indexer, which precludes any parsing of that data on the indexer:

  • parsing
  • aggregation
  • typing

The forwarded data must arrive at the indexer already parsed. To achieve this, you must also set up props.conf on the forwarder that sends the data. This includes configuration of INDEXED_EXTRACTIONS and any other parsing, filtering, anonymizing, and routing rules. Universal forwarders are capable of performing these tasks solely for structured data. See "Forward data extracted from header files".

PREVIOUS
Protect against loss of in-flight data
  NEXT
Forward data to third-party systems

This documentation applies to the following versions of Splunk® Enterprise: 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.4.9, 6.4.10, 6.4.11


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters