Ways to get data into an indexer cluster

Cluster peer nodes can get their data directly from any of the same sources as a non-clustered indexer. However, if data fidelity matters to you, you will use forwarders to initially consume the data before forwarding it to the peer nodes, rather than ingesting the data directly into the nodes.

Advantages of forwarders for data input

There are several key reasons for using forwarders to send data to your cluster:

• To ensure that all incoming data gets indexed. By activating the forwarder's optional indexer acknowledgment feature, you can ensure that all incoming data gets indexed and stored on the cluster. With indexer acknowledgement, when a source peer receives a block of data from a forwarder, it sends the forwarder an acknowledgment after it indexes the data and successfully replicates it to the target peers. If the forwarder does not receive an acknowledgment from the source peer, the forwarder resends the data. The forwarder continues to resend the data until it gets the acknowledgment. Indexer acknowledgment is the only way to ensure end-to-end data fidelity. See "How indexer acknowledgment works."

• To handle potential node failure. With load-balanced forwarders, if one peer in the group goes down, the forwarder continues to send its data to the remaining peers in the group. If, instead, you use direct inputs to the peers, the data source cannot continue to send data when its receiving peer goes down. See "How load balancing works."

• To simplify the process of connecting data sources and peer nodes. By enabling indexer discovery on your forwarders, the forwarders automatically load balance across all available peer nodes, including any that are later added to the cluster. See "Advantages of the indexer discovery method."

Configure inputs directly on the peers

If you decide not to use forwarders to handle your data inputs, you can set up inputs on each peer in the usual way; for example, by editing inputs.conf on the peers. For information on configuring inputs, read "Configure your inputs" in the Getting Data In Manual.