About forwarding and receiving

About forwarding and receiving

You can forward data from a Splunk instance to another Splunk server or even to a non-Splunk system. The Splunk instance that performs the forwarding is typically a smaller footprint version of Splunk, called a forwarder. The forwarder functions as a lightweight, all-purpose agent.

A Splunk server that receives data from a forwarder is called a receiver. The receiver is either a Splunk indexer or another forwarder, configured to receive data from one or more forwarders.

The forwarding and receiving capability makes possible all sorts of interesting Splunk topologies to handle functions like data consolidation, load balancing, data cloning, and data routing.

Forwarders vs. light forwarders

Splunk forwarders come in two flavors: regular and light. These differ according to their functionality and the corresponding size of their footprints.

A regular forwarder, also referred to as just a forwarder, has a smaller footprint than a Splunk server but retains most of the capability, except that it lacks the ability to do distributed searches. Much of its default functionality, such as Splunk Web, can be disabled, if necessary, to further reduce the size of its footprint. A forwarder parses data before forwarding it and can route data based on criteria such as source or type of event.

A light forwarder has a small footprint with limited functionality. Its size makes it ideal for forwarding data from workstations or non-Splunk production servers to a Splunk server for consolidation. It forwards only unparsed data and, therefore, cannot perform content-based routing. In addition, it does not include Splunk Web and its throughput is limited to 256kbs.

For detailed information on the capabilities of regular and light forwarders, see More about forwarders in this manual.

Both types of forwarders can perform automatic load balancing, with the regular forwarder also offering round-robin load balancing. Forwarders represent a much more robust solution for data forwarding than raw network feeds, with their capabilities for:

• Tagging of metadata (source, sourcetype, and host)
• Configurable buffering
• Data compression
• SSL security
• Use of any available network ports

Types of data

Forwarders can transmit three types of data:

• Raw
• Unparsed
• Parsed

A light forwarder can send raw or unparsed data. A regular forwarder can send raw or parsed data.

With raw data, the data stream is forwarded as raw TCP; it is not converted into Splunk's communications format. The forwarder just collects the data and forwards it on. This is particularly useful for sending data to a non-Splunk system.

With unparsed data, a light forwarder performs only minimal processing. It does not examine the data stream, but it does tag the entire stream with metadata to identify source, sourcetype, and host. It also divides the data stream into 32K blocks and performs some rudimentary timestamping on the stream, for use by the receiving indexer in case the events themselves have no discernable timestamps. The light forwarder does not identify, examine, or tag individual events.

With parsed data, a regular forwarder breaks the data into individual events, which it tags and then forwards to a Splunk server. It can also examine the events. Because the data has been parsed, the forwarder can perform conditional routing based on event data, such as field values.

The parsed and unparsed formats are both referred to as cooked data, to distinguish them from raw data. By default, forwarders send cooked data — in the light forwarder's case, unparsed data, and in the regular forwarder's case, parsed data. To send raw data instead, set the sendCookedData=false atribute/value pair in outputs.conf.

Deployment topologies

You can deploy Splunk forwarders in a wide variety of scenarios. These are some typical topologies.

Data consolidation

Data consolidation represents one of the most common topologies, with multiple forwarders sending data to a single Splunk server. The scenario typically involves light forwarders forwarding unparsed data from workstations or production non-Splunk servers to a central Splunk server for consolidation and indexing. With a lighter footprint, these forwarders have minimal impact on the performance of the systems they reside on. In other scenarios, regular forwarders send parsed data to a central Splunk server.

Here, three light forwarders are sending data to a single Splunk server:

Load balancing

Load balancing simplifies the process of distributing data across several Splunk servers to handle considerations such as high data volume, horizontal scaling for enhanced search performance, and fault tolerance. In load balancing, the forwarder routes data sequentially to different servers at specified intervals.

Splunk load balancing comes in two flavors:

• Automatic load balancing
• Round-robin load balancing

For most needs, automatic load balancing, in which the forwarder switches receivers at set time intervals, offers the better solution. For details on the relative advantages of automatic vs. round-robin load balancing, see Load balancing in this manual.

In this diagram, three light forwarders are each performing automatic load balancing between two receivers:

Routing and filtering

In routing, a forwarder routes events to specific Splunk or third-party servers, based on criteria such as source, sourcetype, or patterns in the events themselves. Routing at the event level requires a regular forwarder.

A forwarder can also filter and route events to specific queues, or discard them altogether by routing to the null queue.

Here, a regular forwarder routes data to three Splunk servers based on event patterns:

Cloning

With cloning, the forwarder sends duplicate copies of data to multiple Splunk servers. If cloning is combined with load balancing, the forwarder sends duplicate copies of data to multiple groups of servers. This second scenario is particularly useful for situations requiring data redundancy to promote data availability. If any of the servers in a load-balanced group goes down while receiving data, another server in the group automatically takes over the receiver function, ensuring that each group of servers still contains a clone of the data.

In this simple scenario, three forwarders are sending duplicate copies of data to two Splunk servers:

You can also clone data from one full Splunk instance to another. This scenario can be useful if you need to keep complete sets of indexed data at separate locations to ensure fast local access; for instance, the North American office in San Francisco and the European office in London.

In another type of cloning, a regular forwarder can retain indexed data locally while also forwarding parsed data to a Splunk server.

Forwarding to non-Splunk systems

You can send raw data to a third-party system such as a syslog aggregator. You can combine this with data routing, sending some data to a non-Splunk system and other data to one or more Splunk servers.

Here, three forwarders are routing data to two Splunk servers and a non-Splunk system:

Intermediate forwarding

To handle some advanced use cases, you might want to insert an intermediate forwarder between a group of forwarders and the indexer. In this type of scenario, the end-point forwarders send data to a consolidating forwarder, which then forwards the data on to an indexer, usually after indexing it locally. Because any Splunk instance, including a forwarder, can be configured as a receiver, this topology is easy to implement, if called for.

Important: Splunk recommends that the intermediate forwarder always be a heavy forwarder, not a light forwarder.

Typical use cases are situations where you need an intermediate index, either for "store-and-forward" requirements or to enable localized searching. In those cases, the intermediate forwarder would need to be a heavy forwarder, since that is the only type of forwarder with the ability to create indexes.

Key set-up steps

Once you've determined your Splunk deployment topology and what sort of data forwarding is necessary to implement it, the steps for setting up forwarding are simple:

1. Install the Splunk instances that will serve as forwarders and receivers. See the Installation Manual for details.

2. Use Splunk Web or the CLI to enable receiving on the instances designated as receivers. See Set up receiving in this manual.

3. Use Splunk Web or the CLI to enable forwarding on the instances designated as forwarders. See Set up forwarding in this manual.

4. Specify data inputs for the Splunk forwarders in the usual manner. See Add data and configure inputs in this manual.

5. Perform any advanced configuration on each forwarder by editing the outputs.conf file. See Configure forwarders with outputs.conf in this manual.

6. Test the results to confirm that forwarding, along with any configured behaviors like load balancing or routing, is occurring as expected.

In large environments with muliple forwarders, you might find it helpful to use the deployment server to manage your forwarders. See Deploy to other Splunk instances in this manual.

• Was this topic useful?
• Post a comment