Splunk® Enterprise

Search Manual

Download manual as PDF

Download topic as PDF

About real-time searches and reports

With real-time searches and reports, you can search events before they are indexed and preview reports as the events stream in.

  • You can design alerts based on real-time searches that run continuously in the background. Such real-time alerts can provide timelier notifications than alerts that are based on scheduled reports. For more information, see the Alerting Manual.
  • You can also display real-time search results and reports in dashboards. For more information, see the dashboard overview in Dashboards and Visualizations.


The number of concurrent real-time searches can greatly affect indexing performance. To lessen the impact on the indexer, you can enable indexed real-time searches.

Note: By default, only users with the Admin role can run and save real-time searches. For more information on managing roles and assigning them to users, see "Add and edit roles" in Securing Splunk Enterprise.

Real-time search mechanics

Real-time searches scan events as the events arrive for indexing. When you kick off a real-time search, Splunk software scans the incoming events. The scan looks for events that contain index-time fields that indicate the event could be a match for your search.

As the real-time search runs, the software periodically evaluates the scanned events against your search criteria to find actual matches within the sliding time range window that you have defined for the search. The number of matching events can fluctuate up or down over time as the search discovers matching events at a faster or slower rate. If you are running the search in Splunk Web, the search timeline also displays the matching events that the search has returned within the chosen time range.

Here is an example of a real-time search with a one minute time range window. At the point that this screen capture was taken, the search had scanned a total of 904 events since it was launched. The matching event count of 447 represents the number of events matching the search criteria that had been identified in the past minute. This number fluctuated between 430 and 450 for the following minute. If it had spiked or dropped dramatically, that could have been an indication that something interesting was happening that required a closer look.

6.1 Aboutrealtimesearches.png

As you can see, the newest events are on the right-hand side of the timeline. As time passes, the events move left until the events move off the left-hand side, disappearing from the time range window entirely.

A real-time search should continue running until you or another user stops the search or deletes the search job. The real-time search should not "time out" for any other reason. If your events are stopping it could be a performance-related issue (see "Expected performance and known limitations").

Real-time searches can take advantage of all search functionality, including advanced functionality like lookups, transactions, and so on. There are also search commands that are to be used specifically in conjunction with real-time searches, such as streamstats and rtorder.

Indexed real-time search

Enabling your real-times searches to run after the events are indexed can greatly improve indexing performance. This is especially true if there are a large number of concurrent real-time searches. To lessen the impact on the indexer, you can enable indexed real-time search. This runs searches like historical searches, but also continually updates the search with new events as the events appear on disk.

Use indexed real-time search when up-to-the-second accuracy is not needed.

Prerequisites

  • Only users with the admin role, or a role with equivalent permissions, can enable indexed real-time search.
  • Review the steps in How to edit a configuration file in the Admin Manual.

Steps

  1. Open the local limits.conf file for the Search app. For example, $SPLUNK_HOME/etc/apps/<app_name>/local.
  2. Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location.
  3. Under the [realtime] stanza, set indexed_realtime_use_by_default to true.


If you are using Splunk Cloud and want to change the default to indexed real-time search, open a Support ticket.

Setting the lag time

The results returned by indexed real-time search will always lag behind a real-time search. You can control the number of seconds of lag with the indexed_realtime_disk_sync_delay = <int> setting. By default, this delay is set to 60 seconds.

Other indexed real time settings

There are other settings that you can use to configure indexed real-time search behavior, including:

  • indexed_realtime_default_span
  • indexed_realtime_maximum_span
  • indexed_realtime_cluster_update_interval


These settings are described in the limits.conf.spec file.

PREVIOUS
Open a non-transforming search in Pivot to create tables and charts
  NEXT
Real-time searches and reports in Splunk Web

This documentation applies to the following versions of Splunk® Enterprise: 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.6.0, 6.6.1, 6.6.2, 6.6.3


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters