Configure forwarding with outputs.conf

The outputs.conf file defines how forwarders send data to receivers. You can specify some output configurations at installation time (Windows universal forwarders only) or the CLI, but most advanced configuration settings require that you edit outputs.conf.

The topics that describe various forwarding topologies, such as load balancing and intermediate forwarding, provide detailed examples on configuring outputs.conf to support those topologies.

Although outputs.conf is a required file for configuring forwarders, it addresses only the outputs from the forwarder, where you want the forwarder to send the data it collects. To specify the data that you want to collect from the forwarder, you must separately configure the inputs, as you would for any Splunk instance. See Add data and configure inputs in Getting Data In.

Edit outputs.conf to configure forwarding

This procedure details the steps you must take to edit the default outputs.conf which is in $SPLUNK_HOME/etc/system/local. You might have to edit the file in other places, as sections in this topic explain. For an example of what an outputs.conf file looks like, see "Examples of outputs.conf" later in this topic.

1. On the host that forwards that data that you want to collect, open a shell or command prompt or PowerShell window.
2. Go to the configuration directory for the forwarder.
   

   Unix	Windows
   cd $SPLUNK_HOME/etc/system/local	cd %SPLUNK_HOME%\etc\system\local

3. Open outputs.conf for editing with a text editor.
   

   Unix	Windows
   vi outputs.conf	notepad outputs.conf

4. Edit outputs.conf. Add a minimum of at least one forwarding target group or a single receiving host.
5. Save the outputs.conf file and close it.
6. Restart the universal forwarder to complete your changes.
   

   Unix	Windows
   cd $SPLUNK_HOME/bin ./splunk restart	cd %SPLUNK_HOME%\bin .\splunk restart

Types of outputs.conf files

A single forwarder can have multiple outputs.conf files. For example, one can be located in an apps directory and another in $SPLUNK_HOME/etc/system/local. No matter how many outputs.conf files the forwarder has and where they reside, the forwarder combines all their settings, using the rules of configuration file precedence. The forwarder contains both default and custom outputs.conf files.

Default versions of outputs.conf

The universal forwarder ships with these default versions of outputs.conf:

• One in $SPLUNK_HOME/etc/system/default.
• Another in $SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/default.

The default version in the SplunkUniversalForwarder app has precedence over the version under /etc/system/default.

Do not edit default versions of any configuration files. See About configuration files.

Custom versions of outputs.conf

When you configure forwarding behavior, those changes get saved in custom versions of outputs.conf. There are several ways you can specify forwarding behavior:

• While installing the forwarder (on the Windows universal forwarder only.)
• By running CLI commands.
• By directly editing an outputs.conf file.

The forwarder automatically creates or edits custom versions of outputs.conf in response to the first three methods. The locations of those versions vary, depending on the type of forwarder and other factors.

• If you use the CLI to make changes to universal forwarder output behavior, the CLI creates or edits a copy of outputs.conf in $SPLUNK_HOME/etc/system/local.
• The Windows installation process writes configuration changes to an outputs.conf file located in the MSICreated app.

In addition to any outputs.conf files that you create and edit indirectly (for example, through the CLI), you can also create or edit an outputs.conf file directly with a text editor. You should work with a single copy of the file, which you place in $SPLUNK_HOME/etc/system/local/. If a copy of the file already exists in that directory, because of configuration changes made through the CLI, edit that copy. For purposes of distribution and management simplicity, you can combine settings from all non-default versions into a single custom outputs.conf file.

The universal forwarder must be restarted after you make changes to outputs.conf.

For information on outputs.conf, see the outputs.conf spec file.

Configuration levels for outputs.conf

There are two types of output processors for forwarding data: tcpout and syslog. The universal forwarder only has the tcpout processor, which uses the [tcpout] header in outputs.conf.

You can configure the tcpout processor at three levels of stanzas:

• Global. (Optional) At the global level, you specify any attributes that you want to apply globally, as well as certain attributes only configurable at the system-wide level for the output processor.
• Target group. A target group defines settings for one or more receiving indexers. There can be multiple target groups per output processor. Most configuration settings can be specified at the target group level.
• Single server. (Optional) You can specify configuration values for single servers (receivers) within a target group.

Configurations at the more specific levels take precedence over the global level. For example, if you specify compressed=true for a target group, the forwarder sends the hosts in that target group compressed data, even if you set the compressed attribute to "false" for the global level.

Outputs.conf global stanza

The global stanza in outputs.conf lets you set any attributes that you want to apply globally. While this stanza is optional, there are several attributes that you can set only at the global level, including defaultGroup.

The [tcpout] header specifies the global stanza for the tcpout processor. Following is an example of a global tcpout stanza.

[tcpout]
defaultGroup=indexer1
compressed=true

This global stanza includes two attribute/value pairs:

• defaultGroup=indexer1 This tells the forwarder to send all data to the "indexer1" target group. See "Default target groups".
• compressed=true This tells the forwarder to compress the data before it forwards the data to receiving indexers in the target groups. If you set compressed to "false", the forwarder sends raw data.

Set default target groups in outputs.conf

The defaultGroup attribute lets you set default groups for automatic forwarding at the global level, in your [tcpout] stanza.

The defaultGroup specifies one or more target groups that you define later in tcpout:<target_group> stanzas. The forwarder sends all events to the specified groups.

[tcpout]
defaultGroup= <target_group1>, <target_group2>, ...

If you do not want to forward data automatically, do not set the defaultGroup attribute.

Outputs.conf target group stanza

The target group identifies a set of receivers. It also specifies how the forwarder sends data to those receivers. You can define multiple target groups.

Here is the basic pattern for the target group stanza.

[tcpout:<target_group>]
server=<receiving_server1>, <receiving_server2>, ...
<attribute1> = <val1>
<attribute2> = <val2>
...

You can specify a receiving server in a target group by using the format <ipaddress_or_hostname>:<port>, where <port> is the receiving host receiving port. For example, myhost.splunk.com:9997. When you specify multiple receivers, the forwarder load balances among them.

A target group stanza name cannot have spaces or colons in it. Splunk software ignores target groups whose stanza names contain spaces or colons in them.

See Define typical deployment topologies later in this topic for information on how to use the target group stanza to define several deployment topologies.

Outputs.conf single-host stanza

You can define a specific configuration for an individual receiving indexer. However, the receiver must also be a member of a target group.

When you define an attribute at the single-host level, it takes precedence over any definition at the target group or global level.

Here is the syntax for defining a single-host stanza:

[tcpout-server://<ipaddress_or_hostname>:<port>]
<attribute1> = <val1>
<attribute2> = <val2>
...

Examples of outputs.conf

The following outputs.conf example contains three stanzas for sending data to Splunk receivers.

• Global settings. In this example, there is one setting, to specify a defaultGroup.
• Settings for a single target group consisting of two receivers. Here, we specify a load-balanced target group consisting of two receivers.
• Settings for one receiver within the target group. In this stanza, you can specify any settings specific to the mysplunk_indexer1 receiver.

[tcpout]
defaultGroup=my_indexers

[tcpout:my_indexers]
server=mysplunk_indexer1:9997, mysplunk_indexer2:9996

[tcpout-server://mysplunk_indexer1:9997]

Define typical forwarder deployment topologies

You can configure a forwarder to support several typical deployment topologies. See the other topics in the "Forward data" chapter for information on how to configure forwarders for other topologies.

Configure load balancing on a universal forwarder with outputs.conf

When you specify a target group with multiple receivers in outputs.conf on a forwarder, the forwarder performs load balancing between the receivers.

In the example that follows, the target group consists of three receivers. The forwarder balances load between the three receivers you specify. If one receiver goes down, the forwarder automatically switches to the next available receiver.

[tcpout:my_LB_indexers]
server=10.10.10.1:9997,10.10.10.2:9997,10.10.10.3:9997

Note: While 9997 is the standard network port for receiving data from forwarders, you can specify any network port above 1024 to receive data.

Configure data cloning on a universal forwarder with outputs.conf

When you specify multiple target groups with a separate stanza for each group in outputs.conf, the forwarder performs data cloning between the groups. In data cloning, the forwarder sends copies of all its events to the receivers in two or more target groups. Data cloning usually results in similar, but not necessarily exact, copies of data on the receiving indexers. An example of how to configure data cloning follows.

[tcpout]
defaultGroup=indexer1,indexer2

[tcpout:indexer1]
server=10.1.1.197:9997

[tcpout:indexer2]
server=10.1.1.200:9997

The forwarder sends duplicate data streams to the servers specified in both the indexer1 and indexer2 target groups.

Configure data cloning with load balancing on a universal forwarder

You can combine load balancing with data cloning. For example:

[tcpout]
defaultGroup=cloned_group1,cloned_group2

[tcpout:cloned_group1]
server=10.10.10.1:9997, 10.10.10.2:9997, 10.10.10.3:9997

[tcpout:cloned_group2]
server=10.1.1.197:9997, 10.1.1.198:9997, 10.1.1.199:9997, 10.1.1.200:9997

The forwarder sends full data streams to both the cloned_group1 and cloned_group2 groups. The forwarders load-balance the data within each group, rotating among receivers every 30 seconds (the default frequency).

Common attributes for outputs.conf

The outputs.conf file provides a large number of configuration options that offer considerable control and flexibility in forwarding. Of the attributes available, several are of particular interest:

Attribute	Default	Where configured	Value
defaultGroup	n/a	global stanza	A comma-separated list of one or more target groups. Forwarder sends all events to all specified target groups.
server	n/a	target group stanza	Required. Specifies the hosts that function as receivers for the forwarder. This must be set to a value using the format <ipaddress_or_servername>:<port>, where <port> is the receiving server's receiving port.
disabled	false	any stanza level	Specifies whether the stanza is disabled. If set to "true", it is equivalent to the stanza not being there.
sendCookedData	true	global or target group stanza	Specifies whether data is cooked before forwarding.
compressed	false	global or target group stanza	Specifies whether the forwarder sends compressed data.
ssl....	n/a	any stanza level	Set of attributes for configuring SSL. See "Configure Splunk indexing and forwarding to use TLS certificates" in the Securing Splunk Enterprise manual for information on how to use these attributes.
useACK	false	global or target group stanza	Specifies whether the forwarder waits for indexer acknowledgment confirming that the data has been written to the file system.
dnsResolutionInterval	300	global or target group stanza	Specifies base time interval in seconds at which indexer DNS names will be resolved to IP address.
autoLBVolume	0	global or target group stanza	Specifies, in bytes, how much data a forwarder that has been configured for load balancing sends to an indexer before it selects another indexer.

The outputs.conf.spec file, which you can find here, along with several examples, provides details for these and all other configuration options. In addition, most of these settings are discussed in topics that deal with specific forwarding scenarios.

DNS resolution interval

The dnsResolutionInterval attribute specifies the base time interval (in seconds) at which receiver DNS names will be resolved to IP addresses. The forwarder uses this value to compute the run-time interval as follows:

run-time interval = dnsResolutionInterval + (number of receivers in server attribute - 1) * 30

The run-time interval increases by 30 seconds for each additional receiver that you specify in the server attribute (each additional receiver across which the forwarder load-balances.) The dnsResolutionInterval attribute defaults to 300 seconds.

For example, if you leave the attribute at the default setting of 300 seconds and the forwarder is load-balancing across 20 indexers, DNS resolution will occur every 14 1/2 minutes:

(300 + ((20 - 1) * 30)) = 870 seconds = 14.5 minutes

If you change dnsResolutionInterval to 600 seconds, and keep the number of load-balanced indexers at 20, DNS resolution will occur every 19 1/2 minutes:

(600 + ((20 - 1) * 30)) = 1170 seconds = 19.5 minutes

Configure the universal forwarder to send data over HTTP

Configure the universal forwarder to send data over hyper text transfer protocol (HTTP) between Splunk platform instances when you are unable to open network traffic to use the Splunk to Splunk (S2S) service.

Note the following limitations:

• A Splunk universal forwarder instance can perform either httpout or tcpout, but not both at the same time. There is currently no support to send ACKs to the client transaction.
• HTTPOUT cannot be used to send data among universal forwarders. It is used for the communication between the universal forwarder and an indexer or the universal forwarder and a heavy forwarder.
• EVENT_BREAKER and EVENT_BREAKER_ENABLE in props.conf can be used to help find event breaking boundaries for unparsed data indexing.

To configure a universal forwarder to send data over HTTP, add an httpout stanza to the outputs.conf file on your universal forwarder.

If httpout is configured, chunkedlinebreaker will be disabled.

Example httpout stanza

[httpout]
httpEventCollectorToken = eb514d08-d2bd-4e50-a10b-f71ed9922ea0
uri = https://10.222.22.122:8088

Example httpout stanza, with batch control

[httpout]
httpEventCollectorToken = eb514d08-d2bd-4e50-a10b-f71ed9922ea0
uri = https://10.222.22.122:8088
batchSize = 32768 #32kb batch size instead of 64kb default
batchTimeout = 10 #10 second timeout instead of 30s default

Available parameters for the httpout stanza

Parameter name	Description
httpEventCollectorToken= <authToken>	Authentication Token is used by the HEC endpoint to configure and validate against the HTTP transaction.
uri = <uri>	A Uniform Resource Identifier (URI) is the IP:Port combination of the indexer or load balancer. An empty URI disables the processor.
batchSize = <size in Bytes>	(Optional) Indicates how much data can be batched in one HTTP request. Default batch size is 64KB.
batchTimeout = <time in secs>	(Optional) Indicates if the above batch size does not fill, then instead of waiting, sends a timeout. Default is 30 seconds.

If batchTimeout becomes true, the accumulated pipeline data is sent over the HTTP connection.

Send data over HTTP using a load balancer

Use this example to configure a load balancer configuration using NGINX.

Note: The Splunk Universal Forwarder supports Network Load Balancers (NLB) and Application Load Balancers (ALB) only when you use HTTP out.

events {
        worker_connections 768;
         multi_accept on;
}
http {
        upstream hec {
                keepalive 32
                server <HEC Server 1 IP>:8088;
                server <HEC Server 2 IP>:8088;
                server <HEC Server 3 IP>:8088;
        }
        server {
                listen 80;
                location / {
                        proxy_pass http://hec;
                        proxy_http_version 1.1;
                        proxy_set_header Connection "";
                }
        }
        server {
                listen 443 ssl;
                ssl_certificate server.pem;
                ssl_certificate_key server.pem;
                location / {
                        proxy_pass https://hec;
                        proxy_http_version 1.1;
                        proxy_set_header Connection "";
                }
        }
        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log; 
}

For more information on load balancer configuration, see the Configure load balancing for Splunk Enterprise topic in the Forwarding manual

LB_CHUNK_BREAKER configurations

Starting in version 8.1.1 of the Splunk software, LB_CHUNK_BREAKER has been deprecated in favor of EVENT BREAKER.

LB_CHUNK_BREAKER is a configuration option for breaking events on your Splunk universal forwarder for sending over HTTP.

Since HTTP is a synchronous protocol, it is possible that a chunk of events read by the universal forwarder can be sent with one or more events breaking before sending. The LB_CHUNK_BREAKER configuration aids the universal forwarder in properly defining event boundaries to avoid any events being improperly broken before sending.

If LB_CHUNK_BREAKER is not defined then the universal forwarder will use your deployment's EVENT BREAKER settings. if EVENT BREAKER and LB_CHUNK_BREAKER are not defined, then the default pattern ([\r\n]+) will be used.

Splunk TCP leverages an asynchronous protocol that prevents this type of event breaking from occurring. If you are able to use Splunk TCP settings it is the preferred method for sending and receiving data in Splunk Enterprise and Splunk Cloud Platform from Splunk forwarders.

Example props.conf files on universal forwarder

#The capture group of (.) is the new event, it must be preceded by the string "xyz". 
[customersource1]
LB_CHUNK_BREAKER = xyz(.)

#The capture group of (\n) defines new events and must be preceded by the string "xyz"
[customersource2]
LB_CHUNK_BREAKER = xyz(\n)

# The following regex pattern establishes event boundaries for Java traces.
[log4j]

LB_CHUNK_BREAKER = ([\r\n]+)\d{4}-\d\d-\d\d

#Carriage return and a new line feed is the default pattern for LB_CHUNK_BREAKER. It will be used if no stanza is added by the user. This pattern also matched most access logs.
[access_combined]
LB_CHUNK_BREAKER = ([\r\n]+)

Splunk TCP and HTTP output stanza precedence

When a single Splunk Universal Forwarder has both tcp output settings and http output settings, http output will take precedence. The tcp output settings will be ignored in favor of http output configurations.

Troubleshoot HTTP Out

• In case of partial insertion of data, the client will resend all the data in the HTTP transaction, potentially resulting in duplicate events appearing on your receiver or indexer. Currently, the server sends how many segments it inserts successfully. The client has to implement to keep track of those segment counts and replay only the errored segments.
• There can be multiple chunks/segments of data in the same HTTP transaction
• If some segments inserted without an error but later receive an error (for example, "queue full", or "server busy"), then the server replies with an HTTP error.