Splunk® Enterprise

Distributed Deployment Manual

Download manual as PDF

Splunk Enterprise version 5.0 reached its End of Life on December 1, 2017. Please see the migration information.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Configure forwarders with outputs.conf

The outputs.conf file defines how forwarders send data to receivers. You can specify some output configurations at installation time (Windows universal forwarders only) or through Splunk Web (heavy/light forwarders only) or the CLI, but most advanced configuration settings require that you directly edit outputs.conf. The topics describing various topologies, such as load balancing and data routing, provide detailed examples on configuring outputs.conf to support those topologies.

Important: Although outputs.conf is a critical file for configuring forwarders, it specifically addresses the outputs from the forwarder. To specify the inputs to a forwarder, you must separately configure the inputs, as you would for any Splunk instance. For details on configuring inputs, see "Add data and configure inputs" in the Getting Data In manual.

Types of outputs.conf files

A single forwarder can have multiple outputs.conf files (for instance, one located in an apps directory and another in /system/local). No matter how many outputs.conf files the forwarder has and where they reside, the forwarder combines all their settings, using the rules of location precedence, as described in "Configuration file precedence". Your installation will contain both default and custom outputs.conf files.

Default versions

Splunk ships with these default versions of outputs.conf:

  • On the universal forwarder: The universal forwarder has two default outputs.conf files, one in $SPLUNK_HOME/etc/system/default and the other in $SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/default. The default version in the SplunkUniversalForwarder app has precedence over the version under /etc/system.
  • On heavy and light forwarders: These have a single default outputs.conf file, located in $SPLUNK_HOME/etc/system/default.

Important: Do not touch default versions of any configuration files, for reasons explained in "About configuration files".

Custom versions

When you configure forwarding behavior, those changes get saved in custom versions of outputs.conf. There are several ways you can specify forwarding behavior:

  • While installing the forwarder (Windows universal forwarder only)
  • By running CLI commands
  • By using Splunk Web (heavy/light forwarders only)
  • By directly editing an outputs.conf file

Splunk automatically creates or edits custom versions of outputs.conf in response to the first three methods. The locations of those versions vary, depending on the type of forwarder and other factors:

  • The universal forwarder. If you use the CLI to make changes to universal forwarder output behavior, it creates or edits a copy of outputs.conf in $SPLUNK_HOME/etc/system/local. However, the Windows installation process writes configuration changes to an outputs.conf file located in the MSICreated app. For more information on configuring the universal forwarder, look here.
  • Heavy and light forwarders. When you enable a heavy/light forwarder through Splunk Web or the CLI, Splunk creates an outputs.conf file in the directory of the currently running app. For example, if you're working in the search app, Splunk places the file in $SPLUNK_HOME/etc/apps/search/local/. You can then edit it there.

In addition to any outputs.conf files that you create and edit indirectly (for example, through the CLI), you can also create or edit an outputs.conf file directly. It is recommended that you work with just a single copy of the file, which you place in $SPLUNK_HOME/etc/system/local/. (If a copy of the file already exists in that directory, because of configuration changes made through the CLI, just edit that copy.) For purposes of distribution and management simplicity, you can combine settings from all non-default versions into a single custom outputs.conf file.

After making changes to outputs.conf, you must restart the forwarder for the changes to take effect.

For detailed information on outputs.conf, look here for the spec and examples.

Configuration levels

There are two types of output processors: tcpout and syslog. You can configure them at three levels of stanzas:

  • Global. At the global level, you specify any attributes that you want to apply globally, as well as certain attributes only configurable at the system-wide level for the output processor. This stanza is optional.
  • Target group. A target group defines settings for one or more receiving indexers. There can be multiple target groups per output processor. Most configuration settings can be specified at the target group level.
  • Single server. You can specify configuration values for single servers (receivers) within a target group. This stanza type is optional.

Configurations at the more specific level take precedence. For example, if you specify compressed=true for a target group, the forwarder will send the servers in that target group compressed data, even if compressed is set to "false" for the global level.

Note: This discussion focuses on the tcpout processor, which uses the [tcpout] header. For the syslog output processor, see "Forward data to third-party systems" for details.

Global stanza

Here you set any attributes that you want to apply globally. This stanza is not required. However, there are several attributes that you can set only at the global level, including defaultGroup and indexAndForward.

The global stanza for the tcpout procesor is specified with the [tcpout] header.

Here's an example of a global tcpout stanza:

[tcpout]
defaultGroup=indexer1
indexAndForward=true

This global stanza includes two attribute/value pairs:

  • defaultGroup=indexer1 This tells the forwarder to send all data to the "indexer1" target group. See "Default target groups" for more information.
  • indexAndForward=true This tells the forwarder to index the data locally, as well as forward the data to receiving indexers in the target groups. If set to "false" (the default), the forwarder just forwards data but does not index it. This attribute is only available for heavy forwarders; universal and light forwarders cannot index data.

Default target groups

To set default groups for automatic forwarding, include the defaultGroup attribute at the global level, in your [tcpout] stanza:

[tcpout]
defaultGroup= <target_group1>, <target_group2>, ...

The defaultGroup specifies one or more target groups, defined later in tcpout:<target_group> stanzas. The forwarder will send all events to the specified groups.

If you do not want to forward data automatically, don't set the defaultGroup attribute. (Prior to 4.2, you were required to set the defaultGroup to some value. This is no longer necessary.)

For some examples of using the defaultGroup attribute, see "Route and filter data".

Target group stanza

The target group identifies a set of receivers. It also specifies how the forwarder sends data to those receivers. You can define multiple target groups.

Here's the basic pattern for the target group stanza:

[tcpout:<target_group>]
server=<receiving_server1>, <receiving_server2>, ...
<attribute1> = <val1>
<attribute2> = <val2>
...

To specify a receiving server in a target group, use the format <ipaddress_or_servername>:<port>, where <port> is the receiving server's receiving port. For example, myhost.Splunk.com:9997. You can specify multiple receivers and the forwarder will load balance among them.

Note: Starting with Splunk version 4.3, you can use an IPv6 address when specifying the receiving indexer. For more information, see "Configure Splunk for IPv6" in the Admin manual.

See "Define typical deployment topologies", later in this topic, for information on how to use the target group stanza to define several deployment topologies.

Single-server stanza

You can define a specific configuration for an individual receiving indexer. However, the receiver must also be a member of a target group.

When you define an attribute at the single-server level, it takes precedence over any definition at the target group or global level.

Here is the syntax for defining a single-server stanza:

[tcpout-server://<ipaddress_or_servername>:<port>]
<attribute1> = <val1>
<attribute2> = <val2>
...

Example

The following outputs.conf example contains three stanzas for sending tcpout to Splunk receivers:

  • Global settings. In this example, there is one setting, to specify a defaultGroup.
  • Settings for a single target group consisting of two receivers. Here, we are specifying a load-balanced target group consisting of two receivers.
  • Settings for one receiver within the target group. In this stanza, you can specify any settings specific to the mysplunk_indexer1 receiver.
[tcpout]
defaultGroup=my_indexers

[tcpout:my_indexers]
server=mysplunk_indexer1:9997, mysplunk_indexer2:9996

[tcpout-server://mysplunk_indexer1:9997]

Define typical deployment topologies

This section shows how you can configure a forwarder to support several typical deployment topologies. See the other topics in the "Forward data" section of this book for information on configuring forwarders for other topologies.

Load balancing

To perform load balancing, specify one target group with multiple receivers. In this example, the target group consists of three receivers:

[tcpout:my_LB_indexers]
server=10.10.10.1:9997,10.10.10.2:9996,10.10.10.3:9995

The forwarder will load balance between the three receivers listed. If one receiver goes down, the forwarder automatically switches to the next one available.

Data cloning

To perform data cloning, specify multiple target groups, each in its own stanza. In data cloning, the forwarder sends copies of all its events to the receivers in two or more target groups. Data cloning usually results in similar, but not necessarily exact, copies of data on the receiving indexers. Here's an example of how you set up data cloning:

[tcpout]
defaultGroup=indexer1,indexer2

[tcpout:indexer1]
server=10.1.1.197:9997

[tcpout:indexer2]
server=10.1.1.200:9997

The forwarder will send duplicate data streams to the servers specified in both the indexer1 and indexer2 target groups.

Data cloning with load balancing

You can combine load balancing with data cloning. For example:

[tcpout]
defaultGroup=cloned_group1,cloned_group2

[tcpout:cloned_group1]
server=10.10.10.1:9997, 10.10.10.2:9997, 10.10.10.3:9997

[tcpout:cloned_group2]
server=10.1.1.197:9997, 10.1.1.198:9997, 10.1.1.199:9997, 10.1.1.200:9997

The forwarder will send full data streams to both cloned_group1 and cloned_group2. The data will be load-balanced within each group, rotating among receivers every 30 seconds (the default frequency).

Note: For syslog and other output types, you must explicitly specify routing as described here: "Route and filter data".

Commonly used attributes

The outputs.conf file provides a large number of configuration options that offer considerable control and flexibility in forwarding. Of the attributes available, several are of particular interest:

Attribute Default Where configured Value
defaultGroup n/a global stanza A comma-separated list of one or more target groups. Forwarder sends all events to all specified target groups. Don't set this attribute if you don't want events automatically forwarded to a target group.
indexAndForward false global stanza If set to "true", the forwarder will index all data locally, in addition to forwarding the data to a receiving indexer.

Important: This attribute is only available for heavy forwarders. A universal forwarder cannot index locally.

server n/a target group stanza Required. Specifies the server(s) that will function as receivers for the forwarder. This must be set to a value using the format <ipaddress_or_servername>:<port>, where <port> is the receiving server's receiving port.

Note: Starting with Splunk version 4.3, you can use an IPv6 address when specifying the receiving indexer. For more information, see "Configure Splunk for IPv6" in the Admin manual.

disabled false any stanza level Specifies whether the stanza is disabled. If set to "true", it is equivalent to the stanza not being there.
sendCookedData true global or target group stanza Specifies whether data is cooked before forwarding.
compressed false global or target group stanza Specifies whether the forwarder sends compressed data.
ssl.... n/a any stanza level Set of attributes for configuring SSL. See "About securing data from forwarders" in the Securing Splunk manual for information on how to use these attributes.
useACK false global or target group stanza Specifies whether the forwarder waits for indexer acknowledgment confirming that the data has been written to the file system. See "Protect against loss of in-flight data".
dnsResolutionInterval 300 global or target group stanza Specifies base time interval in seconds at which indexer DNS names will be resolved to IP address. See "DNS resolution interval".

The outputs.conf.spec file, which you can find here, along with several examples, provides details for these and all other configuration options. In addition, most of these settings are discussed in topics dealing with specific forwarding scenarios.

Note: In 4.2, the persistent queue capability was much improved. It is now a feature of data inputs and is therefore configured in inputs.conf. It is not related in any way to the previous, deprecated persistent queue capability, which was configured through outputs.conf. See "Use persistent queues to prevent data loss" for details.

DNS resolution interval

The dnsResolutionInterval attribute specifies the base time interval (in seconds) at which receiver DNS names will be resolved to IP addresses. This value is used to compute the run-time interval as follows:

run-time interval = dnsResolutionInterval + (number of receivers in server attribute - 1) * 30

The run-time interval is extended by 30 seconds for each additional receiver specified in the server attribute; that is, for each additional receiver across which the forwarder is load balancing. The dnsResolutionInterval attribute defaults to 300 seconds.

For example, if you leave the attribute at the default setting of 300 seconds and the forwarder is load-balancing across 20 indexers, DNS resolution will occur every 14 minutes:

(300 + ((20 - 1) * 30)) = 870 seconds = 14 minutes

If you change dnsResolutionInterval to 600 seconds, and keep the number of load-balanced indexers at 20, DNS resolution will occur every 19.5 minutes:

(600 + ((20 - 1) * 30)) = 1170 seconds = 19.5 minutes
PREVIOUS
Enable a receiver
  NEXT
Protect against loss of in-flight data

This documentation applies to the following versions of Splunk® Enterprise: 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters