About real-time searches and reports

With real-time searches and reports, you can search events before they are indexed and preview reports as the events stream in.

• You can design alerts based on real-time searches that run continuously in the background. Such real-time alerts can provide timelier notifications than alerts that are based on scheduled reports. For more information, see the Alerting Manual.
• You can also display real-time search results and reports in dashboards. For more information, see the dashboard overview in Dashboards and Visualizations.

The number of concurrent real-time searches can greatly affect indexing performance. To lessen the impact on the indexer, you can enable indexed real-time searches, which is described later in this topic. For more information, see Expected performance and known limitations of real-time searches and reports.

Users must have the Admin role to run and save real-time searches. For more information on managing roles and assigning roles to users, see Create and manage roles with Splunk Web in Securing Splunk Enterprise.

Real-time searches are turned on by default in Splunk Enterprise and Splunk Cloud Platform on Victoria Experience. In Splunk Cloud Platform on Classic Experience, you must open a support ticket to turn on real-time search. For more information, see Differences between Splunk Cloud Platform and Splunk Enterprise in the Splunk Cloud Platform Service Description.

Real-time search mechanics

Real-time searches scan events as the events arrive for indexing. When you kick off a real-time search, Splunk software scans the incoming events. The scan looks for events that contain index-time fields that indicate the event could be a match for your search.

As the real-time search runs, the software periodically evaluates the scanned events against your search criteria to find actual matches within the sliding time range window that you have defined for the search. The number of matching events can fluctuate up or down over time as the search discovers matching events at a faster or slower rate. If you are running the search in Splunk Web, the search timeline also displays the matching events that the search has returned within the chosen time range.

Here is an example of a real-time search with a one minute time range window. At the point that the following screen capture was taken, the search had scanned a total of 436 events since it was launched. The matching event count of 333 represents the number of events matching the search criteria that were identified in the past minute. This number fluctuated between 312 and 357 for the following minute. If the number spiked or dropped dramatically, that could indicate that something interesting was happening that requires a closer look.

As you can see, the newest events are on the right side of the timeline. As time passes, the events move left until the events move off the left side, disappearing from the time range window entirely.

A real-time search should continue running until you or another user stops the search or deletes the search job. The real-time search should not "time out" for any other reason. If your events are stopping it could be a performance-related issue (see "Expected performance and known limitations").

Real-time searches can take advantage of all search functionality, including advanced functionality like lookups, transactions, and so on. There are also search commands that are to be used specifically in conjunction with real-time searches, such as streamstats and rtorder.

Note that real-time searches are resource intensive and can impact the overall health and performance of your searches.

Indexed real-time search

By default, when you run a real-time search, the search runs before events are indexed. Alternatively, you can configure your real-time searches to run after the events are indexed, which can greatly improve indexing performance. This is especially true if you're running a lot of concurrent real-time searches because indexed real-time search decreases the impact on the indexer. Indexed real-time search runs searches like historical searches, but also continually updates the search with new events as the events appear on disk.

Use indexed real-time search when up-to-the-second accuracy is not needed.

Splunk Cloud Platform

For Splunk Cloud Platform on Victoria Experience, indexed real-time search is turned on.

Splunk Enterprise

For Splunk Enterprise, indexed real-time search is turned off by default. To turn on indexed real-time search, follow these steps.

Prerequisites

• Only users with file system access, such as system administrators, can turn on indexed real-time search.
• Review the steps in How to edit a configuration file in the Admin Manual.

Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location. Make the changes in the local directory.

Steps

Open the local limits.conf file for the Search app. For example, $SPLUNK_HOME/etc/apps/<app_name>/local.

Under the [realtime] stanza, set indexed_realtime_use_by_default to true.

Indexed real-time search and newly added search peers

If your deployment includes newly added search peers and you are using indexed real-time search, all events from the new search peers show up by default in historical searches. However, existing indexed real-time searches will not pick up events from the search peers until you kick off another indexed real-time search. As a result, you should restart your indexed real-time searches every time you add new search peers.

If you want to automatically pick up all events in your real-time searches after you've added new search peers to your deployment, use real-time search instead of indexed real-time search. You can do this by changing the indexed_realtime_use_by_default setting in the local limits.conf file from true back to the default, which is false. See How to edit a configuration file in the Splunk Enterprise Admin Manual.

Using real-time search by setting indexed_realtime_use_by_default to false makes events available to searches with lower latency, but reduces indexing throughput.

About the sync delay lag time

The results returned by an indexed real-time search will always lag behind a real-time search. Built into indexed real-time searches is a sync (synchronizing) delay. The sync delay is a precaution so that none of the data is missed.

Indexed data does not necessarily appear on disk in the order that the data is indexed because:

• Multiple threads are used for indexing simultaneously
• The sync delay ordering that is on your operating system

An indexed real-time must remember the latest indexed event that is returned for the current iteration of the time range window. That event is used as the start point for the next iteration of the time range window. If a sync delay is not imposed, some of the events before the latest event might not be searchable yet. These events are not returned during that iteration of the time range window and will never be returned. The likelihood of an unreturned event increases as the indexing and system load increases.

You can control the number of seconds of sync delay lag time with the indexed_realtime_disk_sync_delay = <int> setting. By default, this delay is set to 60 seconds.

The default of 60 seconds is fairly conservative. For most systems a 30 second delay will probably work successfully. If, for your system and usage, it is acceptable for indexed real-time searches to miss some events, you can set a very low or 0 sync delay. However, you will not be able to tell if you are missing events, except for searches that should match all events.

Other indexed real time settings

There are other settings that you can use to configure indexed real-time search behavior, including:

• indexed_realtime_default_span
• indexed_realtime_maximum_span
• indexed_realtime_cluster_update_interval

These settings are described in the limits.conf.spec file.

See also

• Real-time searches and reports in Splunk Web
• Real-time searches and reports in the CLI
• Expected performance and known limitations of real-time searches and reports
• How to restrict usage of real-time searches

Blogs

• Splunk: Limiting Real-Time Searches and Maximizing Performance Gainz by Hurricane Labs