Splunk® Enterprise

Getting Data In

Download manual as PDF

Splunk version 4.x reached its End of Life on October 1, 2013. Please see the migration information.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Use persistent queues to help prevent data loss

Persistent queuing lets you store data in an input queue to disk. This can help prevent data loss if the forwarder or indexer gets backed up.

By default, forwarders and indexers have an in-memory input queue of 500KB. If the input stream is running at a faster rate than the forwarder or indexer can process, to a point where the queue is maxed out, undesired consequences will occur. In the case of UDP, data will drop off the queue and get lost. For other input types, the application generating the data will get backed up.

By implementing persistent queues, you can help prevent this from happening. With persistent queuing, once the in-memory queue is full, the forwarder or indexer writes the input stream to files on disk. It then processes data from the queues (in-memory and disk) until it reaches the point when it can again start processing directly from the data stream.

Important: Persistent queues help prevent data loss if Splunk gets backed up. They are not a panacea, however. You can still lose data if Splunk crashes. For example, some input data is held in the in-memory queue, as well as in the persistent queue files. The in-memory data can get lost if there's a crash. Similarly, data that's in the parsing or indexing pipeline but that has not yet been written to disk can get lost in the event of a crash.

Note: In 4.2, the persistent queue capability has been re-implemented, in a much improved fashion. It is now a feature of data inputs and is therefore configured in inputs.conf. It is not related in any way to the previous, deprecated persistent queue capability, which was configured through outputs.conf.

When can you use persistent queues?

Persistent queuing is available for certain types of inputs, but not all. Generally speaking, it is available for inputs of an ephemeral nature, such as network inputs, but not for inputs that have their own form of persistence, such as file monitoring.

Persistent queues are available for these input types:

  • TCP
  • UDP
  • FIFO
  • Scripted input, including Windows scripted inputs

Persistent queues are not available for these input types:

  • Monitor
  • Batch
  • File system change monitor
  • Windows event log data
  • splunktcp (input from Splunk forwarders)

Configure a persistent queue

Use the inputs.conf file to configure a persistent queue.

Inputs do not share queues. You configure a persistent queue in the stanza for the specific input.

Syntax

To create the persistent queue, specify these two attributes within the particular input's stanza:

queueSize = <integer>(KB|MB|GB)
* Max size of the in-memory input queue.
* Defaults to 500KB.

persistentQueueSize = <integer>(KB|MB|GB|TB)
* Max size of the persistent queue file on disk.
* Defaults to 0 (no persistent queue).

Example

Here's an example of specifying a persistent queue for a tcp input:

[tcp://9994]
queueSize=50KB
persistentQueueSize=100MB

Persistent queue location

The persistent queue has a hardcoded location, which varies according to the input type.

For network inputs, the persistent queue is located here:

$SPLUNK_HOME/var/run/splunk/[tcpin|udpin]/pq__<port>

Note: There are two underscores in the file name: pq__<port>, not pq_<port>.

For example:

  • The persistent queue for TCP port 2012: $SPLUNK_HOME/var/run/splunk/tcpin/pq__2012
  • The persistent queue for UDP port 2012: $SPLUNK_HOME/var/run/splunk/udpin/pq__2012

For FIFO inputs, the persistent queue resides under $SPLUNK_HOME/var/run/splunk/fifoin/<encoded path>.

For scripted inputs, it resides under $SPLUNK_HOME/var/run/splunk/exec/<encoded path>.

The <encoded path> is derived from the FIFO/scripted input stanza in inputs.conf.

PREVIOUS
Use a test index to test your inputs
  NEXT
Troubleshoot the input process

This documentation applies to the following versions of Splunk® Enterprise: 4.3, 4.3.1, 4.3.2, 4.3.3, 4.3.4, 4.3.5, 4.3.6, 4.3.7, 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters