persistent queue

persistent queue


A feature that you configure to take data that is in an input queue and store it to files on disk. Using a persistent queue can prevent data loss if the forwarder or indexer has too much data to process at one time.

By default, forwarders and indexers have an in-memory input queue of 500KB. Without persistent queues, there is a potential for data loss if the input stream overflows the in-memory queue. With persistent queues, when the in-memory queue is full, the forwarder or indexer writes the input stream to files on disk. The forwarder or indexer then processes data from the queues (in-memory and disk) until it reaches the point when it can again start processing directly from the data stream.

You enable persistent queues in inputs.conf on an input-by-input basis. Persistent queues are disabled by default.

For more information

In Getting Data In: