Related Answers
Download topic as PDF
Use persistent queues to help prevent data loss
Persistent queuing lets you store data in an input queue to disk. This can help prevent data loss if the forwarder or indexer gets backed up.
By default, forwarders and indexers have an in-memory input queue of 500KB. If the input stream runs at a faster rate than the forwarder or indexer can process, to a point where the queue is maxed out, undesired consequences occur. In the case of UDP, data drops off the queue and gets lost. For other input types, the application generating the data gets backed up.
By implementing persistent queues, you can help prevent this from happening. With persistent queuing, once the in-memory queue is full, the forwarder or indexer writes the input stream to files on disk. It then processes data from the queues (in-memory and disk) until it reaches the point when it can again start processing directly from the data stream.
Note: While persistent queues help prevent data loss if processing gets backed up, you can still lose data if Splunk software crashes. For example, Splunk software holds some input data in the in-memory queue as well as in the persistent queue files. The in-memory data can get lost if a crash occurs. Similarly, data that is in the parsing or indexing pipeline but that has not yet been written to disk can get lost in the event of a crash.
When can you use persistent queues?
Persistent queuing is available for certain types of inputs, but not all. Generally speaking, it is available for inputs of an ephemeral nature, such as network inputs, but not for inputs that have their own form of persistence, such as file monitoring.
Persistent queues are available for these input types:
- TCP
- UDP
- FIFO
- Scripted inputs
- Windows Event Log inputs
Persistent queues are not available for these input types:
- Monitor
- Batch
- File system change monitor
- splunktcp (input from Splunk forwarders)
Configure a persistent queue
Use the inputs.conf file to configure a persistent queue.
Inputs do not share queues. You configure a persistent queue in the stanza for the specific input.
Syntax
To create the persistent queue, specify these two attributes within the particular input's stanza:
persistentQueueSize = <integer>(KB|MB|GB|TB) * Max size of the persistent queue file on disk. * Defaults to 0 (no persistent queue).
Example
Here's an example of specifying a persistent queue for a tcp input:
[tcp://9994] persistentQueueSize=100MB
Persistent queue location
The persistent queue has a hardcoded location, which varies according to the input type.
For network inputs, the persistent queue is located here:
$SPLUNK_HOME/var/run/splunk/[tcpin|udpin]/pq__<port>
Note: There are two underscores in the file name: pq__<port>, not pq_<port>.
For example:
- The persistent queue for TCP port 2012:
$SPLUNK_HOME/var/run/splunk/tcpin/pq__2012
- The persistent queue for UDP port 2012:
$SPLUNK_HOME/var/run/splunk/udpin/pq__2012
For FIFO inputs, the persistent queue resides under $SPLUNK_HOME/var/run/splunk/fifoin/<encoded path>.
For scripted inputs, it resides under $SPLUNK_HOME/var/run/splunk/exec/<encoded path>. The FIFO/scripted input stanza in inputs.conf derives the <encoded path>.
|
PREVIOUS Use a test index to test your inputs |
NEXT Troubleshoot the input process |
This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.0.15, 6.1, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.7, 6.1.8, 6.1.9, 6.1.10, 6.1.11, 6.1.12, 6.1.13, 6.1.14, 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.2.14, 6.2.15, 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.3.14, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.6, 6.4.8, 6.4.9, 6.4.10, 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2, 6.5.3, 6.4.11, 6.5.4, 6.5.5, 6.5.6, 6.5.7, 6.5.8, 6.5.9, 6.5.10, 6.6.0, 6.6.1, 6.6.2, 6.6.3, 6.6.4, 6.6.5, 6.6.6, 6.6.7, 6.6.8, 6.6.9, 6.6.10, 6.6.11, 6.6.12, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.3.0, 7.3.1, 7.3.2, 8.0.0, 6.4.5, 6.4.7
Comments
It would be nice if this page also explained what will happen if a persistent queue fills up. Specifically, I am building a modular input and I wonder whether or not the Splunk code on the reading end of my XML output stream will stop reading so that I might block on writing in this condition. Or...will Splunk code keep reading and just silently drop bytes. If so, which bytes? Newest, oldest, undefined?
Answer to Dflodstrom's question posted here:
https://answers.splunk.com/answers/440891/does-a-universal-forwarders-persistent-queue-exist.html
"For example, Splunk software holds some input data in the in-memory queue as well as in the persistent queue files." - Does this mean that if the forwarder/system is shutdown cleanly that the 500K of cached events will get written to persistent queue? Is the persistent queue maintained through a reboot?
This page states that persistentQueueSize is available for Windows Event Log inputs, however, this page, nor the inputs.conf page say how this is set up.