Splunk® Enterprise

Search Manual

Download manual as PDF

Download topic as PDF

Expected performance and known limitations of real-time searches and reports

Real-time search matches events that have arrived at the port but have not been persisted to disk. The rate of arrival of events and number of matches can determine how much memory is consumed and the impact on the indexing rate.

Indexing throughput

Splunk software performance is expected to be acceptable as long as the indexers are not currently heavily loaded and do not have more than a few concurrent real-time searches.

Real-time searches will have a significant impact on performance in high volume environments and network load when you have many concurrent real-time searches.

When planning your real-time searches, you should consider how it will affect the performance of both:

  • The search peer that must stream the live events.
  • The search head that must process the aggregated stream of live events.

The more work that is done on the search peer, the less that is required on the search head, and vice versa. The search peer is important to the overall system function, so you do not want to burden it with too much filtering of live events. However, if the search peer does not filter at all, the processing power and bandwidth required to send all the live events to the search head may prove costly, especially when you have multiple real-time searches running concurrently.

In cases where the search head cannot keep up with the search peer, the queue on the index processor will cease to flag events for the search. However, the events will have a sequence number that you can use to tell when and how many events were omitted from search consideration.

Concurrent real-time and historical searches

You can run real-time and historical searches concurrently, within the limits of your hardware. There are no restrictions on separate searches for the same or different users.

Concurrent real-time searches

Running multiple real-time searches will negatively impact indexing capacity. The real-time search feature is optimized for real-time alerting on sparse, or rare-term, searches and sacrifices indexing capacity for improved latency and reliability.

Indexed real-time searches

The number of concurrent real-time searches can greatly affect indexing performance. To lessen the impact on the indexer, you can enable indexed real-time search. This will basically run the search like a historical search, but will also continually update it with new events as they appear on disk.

Read more about how to enable indexed real-time search in About real-time searches and reports.

Real-time search windows

Windowed real-time searches are more expensive than non-windowed. The operations required to manage and preview the window contents can result in a windowed real time search not keeping up with a high rate of indexing. If your windowed search does not display the expected number of events, try a non-windowed search. If you are interested only in event counts, try using "timechart count" in your search.

See Specify time ranges for real-time searches.

See also

Real-time searches and reports in the CLI
How to restrict usage of real-time search

This documentation applies to the following versions of Splunk® Enterprise: 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.2.14, 6.2.15, 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.3.14, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.4.9, 6.4.10, 6.4.11, 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.5.6, 6.5.7, 6.5.8, 6.5.9, 6.5.10, 6.6.0, 6.6.1, 6.6.2, 6.6.3, 6.6.4, 6.6.5, 6.6.6, 6.6.7, 6.6.8, 6.6.9, 6.6.10, 6.6.11, 6.6.12, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.3.0, 7.3.1, 7.3.2


Thank you for the followup question. None of the queues mentioned in the wiki pages that you posted in your comment are related to the realtime queue mentioned in the documentation. I sent you an email with the details.

Lstewart splunk, Splunker
November 18, 2016

Lstewart splunk, Splunker

Thank you very much for your document update.
I have one more question, please.
Could you teach me what is actually "the queue on the index processor"?
For example, what stage/queue on <http://wiki.splunk.com/Community:HowIndexingWorks> and <https://wiki.splunk.com/Community:HowDistSearchWorks>?

November 7, 2016

Tkomatsubara and Parkyonsu

The choice of wording is unfortunate. By "drop events" we do not mean the events are removed. What we mean is that some events will not be considered for the search. I have corrected the wording above to this:
In cases where the search head can't keep up with the search peer, the queue on the index processor will cease to flag events for the search. However, the events will have a sequence number that you can use to tell when and how many events were omitted from search consideration.

Lstewart splunk, Splunker
October 31, 2016

Could you answer or update documents to the below questions from "Tkomatsubara splunk, Splunker"?

October 30, 2016

>In cases where the search head can't keep up with the search peer, the queue on the index processor will drop events.

Drop events? What does this mean? Which stage/queue are those events dropped?
Can you make it clear since this is very important.

>Indexed real-time searches

If this "Indexed real-time searches" used, can we avoid this data "drop" issue?

Tkomatsubara splunk, Splunker
October 7, 2016

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters