Configure forwarders with outputs.conf
The outputs.conf file defines how forwarders send data to receivers.
While you can specify some output configurations through Splunk Web (heavy/light forwarders only) or the CLI, most advanced configuration settings require that you edit outputs.conf
. The topics describing various topologies, such as load balancing and data routing, provide detailed examples on configuring outputs.conf
to support those topologies.
Although outputs.conf
is a critical file for configuring forwarders, it only addresses where the forwarder should send data. To specify what data the forwarder should collect, you must separately configure the inputs. For details on configuring inputs, see Add data and configure inputs in Getting Data In.
Types of outputs.conf files
A single forwarder can have multiple outputs.conf
files (for instance, one located in an apps directory and another in /system/local
). No matter how many outputs.conf
files the forwarder has and where they reside, the forwarder combines all their settings, using the rules of location precedence, as described in Configuration file precedence in the Admin Manual.
Default versions
Splunk Enterprise ships with a single default outputs.conf
file, located in $SPLUNK_HOME/etc/system/default
.
The universal folder has two default outputs.conf files. See Configure forwarding with outputs.conf in the Universal Forwarder manual.
Do not touch default versions of any configuration files, for reasons explained in About configuration files.
Custom versions
When you configure forwarding, changes get saved in custom versions of outputs.conf
. There are several ways you can specify forwarding behavior:
- While installing the forwarder (on Windows universal forwarder only)
- By running CLI commands
- By using Splunk Web (on heavy and light forwarders only)
- By directly editing an
outputs.conf
file
The forwarder automatically creates or edits custom versions of outputs.conf
in response to the first three methods. The locations of those versions vary, depending on the type of forwarder and other factors.
When you enable a heavy/light forwarder through Splunk Web or the CLI, an outputs.conf
file gets created in the directory of the active app on the instance. For example, if you're working in the search app, Splunk Enterprise creates the file in $SPLUNK_HOME/etc/apps/search/local/
. You can then edit it there.
In addition to any outputs.conf
files that you create and edit indirectly (for example, through the CLI), you can also create or edit an outputs.conf
file directly. A Splunk best practice is to work with just a single copy of the file, which you place in $SPLUNK_HOME/etc/system/local/
. (If a copy of the file already exists in that directory, because of configuration changes made through the CLI, just edit that copy.) For purposes of distribution and management simplicity, you can combine settings from all non-default versions into a single custom outputs.conf
file.
After you make changes to outputs.conf
, you must restart the forwarder for the changes to take effect.
For detailed information on outputs.conf
, see the outputs.conf spec file.
Configuration levels
There are two types of output processors: tcpout
and syslog
.
You can configure them at three levels of stanzas:
- Global. (Optional) At the global level, you specify any attributes that you want to apply globally, as well as certain attributes only configurable at the system-wide level for the output processor.
- Target group. A target group defines settings for one or more receiving indexers. There can be multiple target groups per output processor. Most configuration settings can be specified at the target group level.
- Single server. (Optional) You can specify configuration values for single hosts (receivers) within a target group. .
Configurations at the more specific level take precedence. For example, if you specify compressed=true
for a target group, the forwarder will send the hosts in that target group compressed data, even if compressed
is set to "false" for the global level.
Note: This discussion focuses on the tcpout
processor, which uses the [tcpout]
header. For information on the syslog output processor, see Forward data to third-party systems.
Global stanza
Here you set any attributes that you want to apply globally. This stanza is optional. However, there are several attributes that you can set only at the global level, including defaultGroup
and indexAndForward
.
The global stanza for the tcpout
procesor is specified with the [tcpout]
header.
Here's an example of a global tcpout stanza:
[tcpout] defaultGroup=indexer1 indexAndForward=true
This global stanza includes two attribute/value pairs:
- defaultGroup=indexer1 This tells the forwarder to send all data to the "indexer1" target group. See Default target groups for more information.
- indexAndForward=true This tells the forwarder to index the data locally as well as forward it to receiving indexers in the target groups. If set to "false" (the default), the forwarder forwards data but does not index it. This attribute is only available for heavy forwarders; universal and light forwarders cannot index data.
Default target groups
To set default groups for automatic forwarding, include the defaultGroup
attribute at the global level, in your [tcpout]
stanza:
[tcpout] defaultGroup= <target_group1>, <target_group2>, ...
The defaultGroup specifies one or more target groups that you define later with tcpout:<target_group>
stanzas. The forwarder sends all events to the specified groups.
If you do not want to forward data automatically, don't set the defaultGroup
attribute.
For some examples of using the defaultGroup
attribute, see Route and filter data.
Target group stanza
The target group identifies a set of receivers. It also specifies how the forwarder sends data to those receivers. You can define multiple target groups.
Here's the basic pattern for the target group stanza:
[tcpout:<target_group>] server=<receiving_server1>, <receiving_server2>, ... <attribute1> = <val1> <attribute2> = <val2> ...
To specify a receiving host in a target group, use the format <hostname_or_ip_address>:<port>
, where <port>
is the receiving server's receiving port. For example, myhost.Splunk.com:9997
. You can specify multiple receivers and the forwarder will load balance among them.
See Define typical deployment topologies, later in this topic, for information on how to use the target group stanza to define several deployment topologies.
Single-server stanza
You can define a specific configuration for an individual receiving indexer. However, the receiver must also be a member of a target group.
When you define an attribute at the single-server level, it takes precedence over any definition at the target group or global level.
Here is the syntax for defining a single-server stanza:
[tcpout-server://<ipaddress_or_servername>:<port>] <attribute1> = <val1> <attribute2> = <val2> ...
Example
The following outputs.conf
example contains three stanzas for sending tcpout to Splunk Enterprise receivers:
- Global settings. In this example, there is one setting, to specify a
defaultGroup
. - Settings for a single target group that consists of two receivers. This example specifies a load-balanced target group consisting of two receivers.
- Settings for one receiver within the target group. In this stanza, you can specify any specific settings for the
mysplunk_indexer1
receiver.
[tcpout] defaultGroup=my_indexers [tcpout:my_indexers] server=mysplunk_indexer1:9997, mysplunk_indexer2:9996 [tcpout-server://mysplunk_indexer1:9997]
Define typical deployment topologies
This section describes how to configure a forwarder to support several typical deployment topologies.
Load balancing
To perform load balancing, specify one target group with multiple receivers. In this example, the target group consists of three receivers:
[tcpout:my_LB_indexers] server=10.10.10.1:9997,10.10.10.2:9996,10.10.10.3:9995
The forwarder balances load between the specified three receivers. If one receiver goes down, the forwarder automatically switches to the next available receiver.
Data cloning
To perform data cloning, specify multiple target groups, each in its own stanza. In data cloning, the forwarder sends copies of all its events to the receivers in two or more target groups. Data cloning usually results in similar, but not necessarily exact, copies of data on the receiving indexers. An example of how you set up data cloning follows:
[tcpout] defaultGroup=indexer1,indexer2 [tcpout:indexer1] server=10.1.1.197:9997 [tcpout:indexer2] server=10.1.1.200:9997
The forwarder will send duplicate data streams to the servers specified in both the indexer1
and indexer2
target groups.
Data cloning with load balancing
You can combine load balancing with data cloning. For example:
[tcpout] defaultGroup=cloned_group1,cloned_group2 [tcpout:cloned_group1] server=10.10.10.1:9997, 10.10.10.2:9997, 10.10.10.3:9997 [tcpout:cloned_group2] server=10.1.1.197:9997, 10.1.1.198:9997, 10.1.1.199:9997, 10.1.1.200:9997
The forwarder sends full data streams to both cloned_group1
and cloned_group2.
The data will be load-balanced within each group, rotating among receivers every 30 seconds (the default frequency).
For syslog and other output types, you must explicitly specify routing. See Route and filter data in this manual.
Commonly used attributes
The outputs.conf
file provides a large number of configuration options that offer considerable control and flexibility in forwarding. Of the attributes available, several are of particular interest:
Attribute | Default | Where configured | Value |
---|---|---|---|
defaultGroup | n/a | global stanza | A comma-separated list of one or more target groups. Forwarder sends all events to all specified target groups. Don't set this attribute if you don't want events automatically forwarded to a target group. |
indexAndForward | false | global stanza | If set to "true", the forwarder will index all data locally, in addition to forwarding the data to a receiving indexer.
Note: This attribute is only available for heavy forwarders. A universal forwarder cannot index locally. |
server | n/a | target group stanza | Required. Specifies the server(s) that will function as receivers for the forwarder. This must be set to a value using the format <hostname_or_ip_address>:<port> , where <port> is the receiving port.
|
disabled | false | any stanza level | Specifies whether the stanza is disabled. If set to "true", it is equivalent to the stanza not being there. |
sendCookedData | true | global or target group stanza | Specifies whether the forwarder processes the data before forwarding. |
compressed | false | global or target group stanza | Specifies whether the forwarder sends compressed data. |
ssl.... | n/a | any stanza level | Set of attributes for configuring SSL. See About securing data from forwarders in Securing Splunk Enterprise for information on how to use these attributes. |
useACK | false | global or target group stanza | Specifies whether the forwarder waits for indexer acknowledgment confirming that the data has been written to the file system. See Protect against loss of in-flight data. |
dnsResolutionInterval | 300 | global or target group stanza | Specifies the base time interval in seconds at which indexer DNS names will be resolved to IP addresses. See DNS resolution interval in this manual. |
The outputs.conf.spec
file, which you can find here, along with several examples, provides details for these and all other configuration options. In addition, most of these settings are discussed in topics dealing with specific forwarding scenarios.
DNS resolution interval
The dnsResolutionInterval
attribute specifies the base time interval (in seconds) at which receiver DNS names will be resolved to IP addresses. This value is used to compute the run-time interval as follows:
run-time interval = dnsResolutionInterval + (number of receivers in 'server' attribute - 1) * 30
The run-time interval extends by 30 seconds for each additional receiver specified in the server
attribute (for each additional receiver across which the forwarder is load balancing.) The dnsResolutionInterval
attribute defaults to 300 seconds.
For example, if you leave the attribute at the default setting of 300 seconds and the forwarder load-balancs across 20 indexers, DNS resolution occurs about every 14 minutes:
(300 + ((20 - 1) * 30)) = 870 seconds = 14 minutes
If you change dnsResolutionInterval
to 600 seconds, and keep the number of load-balanced indexers at 20, DNS resolution occurs every 19.5 minutes:
(600 + ((20 - 1) * 30)) = 1170 seconds = 19.5 minutes
Configure data collection on forwarders with inputs.conf | Upgrade heavy and light forwarders |
This documentation applies to the following versions of Splunk Cloud Platform™: 8.2.2112, 8.2.2201, 8.2.2202, 8.2.2203, 9.0.2205, 9.0.2208, 9.0.2209, 9.0.2303, 9.0.2305, 9.1.2308, 9.1.2312, 9.2.2403 (latest FedRAMP release), 9.2.2406
Feedback submitted, thanks!