Configure a forwarder to handle multiple pipeline sets
You can configure a forwarder to use multiple processing pipelines to increase forwarding throughput for machines that forward a large amount of data and have more than one core available.
Forwarding pipelines
The universal forwarder can be configured to handle multiple pipeline sets. A pipeline set is an instance of the event processing section of the data pipeline. A universal forwarder with multiple pipeline sets can process multiple events at once, thus increasing forwarder throughput and getting events to indexers faster. The feature that both indexers and forwarders use to handle multiple streams is called index parallelization.
Other than the fact that the forwarders can send more than one stream of data at a time, forwarders with parallelization enabled look just like other forwarders to indexers. As well, indexers process data streams from forwarders with parallelization enabled in the same way that they process forwarders with one data stream.
While it is possible to increase the number of pipelines by any number, you should not increase it to more than two unless you receive instruction to do so by Splunk Professional Services.
When you enable multiple pipeline sets on a forwarder, any throughput-related settings apply to each pipeline set on the forwarder, rather than to the forwarder itself. For example, if you have two pipeline sets enabled on a forwarder, the maximum network throughput default of 256KBps (2 Mbps) is for each pipeline, for a total of 512KBps (4Mbps) for the forwarder.
For more information about how forwarders use parallelization to process more data, see Forwarders and multiple pipeline sets in the Splunk Enterprise Managing Indexers and Clusters of Indexers manual.
Configure a universal forwarder to use multiple pipeline sets
1. At a command or shell prompt on the universal forwarder, change to the local configuration directory:
| Unix | Windows | |
|---|---|---|
cd $SPLUNK_HOME/etc/system/local |
cd %SPLUNK_HOME%\etc\system\local |
2. Create the file server.conf in this directory.
3. Open the server.conf file for editing.
4. Add the following stanza to enable parallelization:
[general] parallelIngestionPipelines = 2
5. Save the file and close it.
6. Restart the universal forwarder.
| Configure an intermediate forwarder | Configure forwarding to Splunk Enterprise indexer clusters |
This documentation applies to the following versions of Splunk® Universal Forwarder: 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.0.13, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.1.10, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8, 7.3.9, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.0.5, 8.0.6, 8.0.7, 8.0.8, 8.0.9, 8.0.10, 8.1.0, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.10, 8.1.11, 8.1.12, 8.1.13, 8.1.14, 8.2.0, 8.2.1, 8.2.2, 8.2.3
Download manual
Feedback submitted, thanks!