Expected performance and known limitations of real-time searches and reports

Real-time search matches events that have arrived at the port but have not been persisted to disk. The rate of arrival of events and number of matches can determine how much memory is consumed and the impact on the indexing rate.

Indexing throughput

Splunk software performance is expected to be acceptable as long as the indexers are not currently heavily loaded and do not have more than a few concurrent real-time searches.

Real-time searches will have a significant impact on performance in high volume environments and network load when you have many concurrent real-time searches.

When planning your real-time searches, you should consider how it will affect the performance of both:

• The search peer that must stream the live events.
• The search head that must process the aggregated stream of live events.

The more work that is done on the search peer, the less that is required on the search head, and vice versa. The search peer is important to the overall system function, so you do not want to burden it with too much filtering of live events. However, if the search peer does not filter at all, the processing power and bandwidth required to send all the live events to the search head may prove costly, especially when you have multiple real-time searches running concurrently.

In cases where the search head cannot keep up with the search peer, the queue on the index processor will cease to flag events for the search. However, the events will have a sequence number that you can use to tell when and how many events were omitted from search consideration.

Concurrent real-time and historical searches

You can run real-time and historical searches concurrently, within the limits of your hardware. There are no restrictions on separate searches for the same or different users.

Concurrent real-time searches

Running multiple real-time searches will negatively impact indexing capacity. The real-time search feature is optimized for real-time alerting on sparse, or rare-term, searches and sacrifices indexing capacity for improved latency and reliability.

Indexed real-time searches

The number of concurrent real-time searches can greatly affect indexing performance. To lessen the impact on the indexer, you can enable indexed real-time search. This will basically run the search like a historical search, but will also continually update it with new events as they appear on disk.

Read more about how to enable indexed real-time search in About real-time searches and reports.

Real-time search windows

Windowed real-time searches are more expensive than non-windowed. The operations required to manage and preview the window contents can result in a windowed real time search not keeping up with a high rate of indexing. If your windowed search does not display the expected number of events, try a non-windowed search. If you are interested only in event counts, try using "timechart count" in your search.

See Specify time ranges for real-time searches.

See also

• About real-time searches and reports
• Real-time searches and reports in Splunk Web
• Real-time searches and reports in the CLI
• How to restrict usage of real-time searches