Splunk® Enterprise

Search Manual

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

About real-time searches and reports

With real-time searches and reports, you can search events before they are indexed and preview reports as the events stream in.

You can design alerts based on real-time searches that run continuously in the background. Such real-time alerts can provide timelier notifications than alerts that are based on scheduled reports. For more information, see the "About alerts" topic, in the Alerting Manual.

You can also display real-time search results and reports in your custom dashboards using the dashboard editor, panel editor, and simple XML. For more information about the visual dashboard editor, see "Create simple dashboards with the UI" in the Splunk Data Visualizations Manual.

For more information about using real-time dashboards with advanced features that go beyond what the visual dashboard editor can provide, see Build a real-time dashboard in the Developer manual.

Note: When Splunk is used out-of-the-box, only users with the Admin role can run and save real-time searches. For more information on managing roles and assigning them to users, see "Add and edit roles" in Securing Splunk.

Real-time search mechanics

Real-time searches search through events as they stream into Splunk for indexing. When you kick off a real-time search, Splunk scans incoming events that contain index-time fields that indicate they could be a match for your search. Splunk identifies these events in the UI as scanned events. This number is cumulative and represents the sum of all events scanned since the search was launched.

As the real-time search runs, Splunk periodically evaluates the scanned events against your search criteria to find actual matches. These events are identified in the UI as matching events. This number represents the number of matching events that currently exist within the sliding time range window that you have defined for the search. As such it can fluctuate up or down over time as Splunk discovers matching events at a faster or slower rate. If you are running the search in Splunk Web, the search timeline also displays the matching events that the search has returned within the chosen time range.

Here's an example of a real-time search with a one minute time range window. At the point that this screen capture was taken, the search had scanned a total of 904 events since it was launched. The matching event count of 447 represents the number of events matching the search criteria that had been identified in the past minute. This number fluctuated between 430 and 450 for the following minute. If it had spiked or dropped dramatically, that could have been an indication that something interesting was happening that required a closer look.

6.1 Aboutrealtimesearches.png

As you can see, the newest events are on the right-hand side of the timeline. As time passes, they move left until they move off the left-hand side, disappearing from the time range window entirely.

A real-time search should continue running until you or another user stops it or deletes the search job; it should not "time out" for any other reason. If your events are stopping it could be a performance-related issue (see "Expected performance and known limitations").

Real-time searches can take advantage of all Splunk search functionality, including advanced functionality like lookups, transactions, and so on. We've also designed search commands that are to be used specifically in conjunction with real-time searches, such as streamstats and rtorder.

Indexed real-time search

The number of concurrent real-time searches can greatly affect Splunk's indexing performance. To lessen the impact on the indexer, you can enable indexed real-time search. This runs the search like a historical search, but also continually updates it with new events as they appear on disk.

To enable indexed real-time search as the default behavior for your real-time searches, edit the limits.conf stanza called realtime and set indexed_realtime_use_by_default = true.

Indexed real-time search is used when up-to-the-second accuracy is not needed. The results returned by indexed real-time search will always lag behind a real-time search. You can control the number of seconds of lag with the indexed_realtime_disk_sync_delay = <int> setting. By default, this delay is 60 seconds.

Other settings you can use to configure indexed real-time search behavior follows.

[realtime]

indexed_realtime_default_span = <int>
* An indexed realtime search is made up of many component historical searches that by default 
* will span this many seconds. If a component search is not completed in this many seconds the
* next historical search will span the extra seconds. To reduce the overhead of running an 
* indexed realtime search you can change this span to delay longer before starting the next 
* component historical search.
* Precendence: Indexers
* Defaults to 1

indexed_realtime_maximum_span = <int>
* While running an indexed realtime search, if the component searches regularly take longer 
* than indexed_realtime_default_span seconds, then indexed realtime search can fall more than 
* indexed_realtime_disk_sync_delay seconds behind realtime. Use this setting to set a limit 
* afterwhich we will drop data to return back to catch back up to the specified delay from 
* realtime, and only search the default span of seconds. 
* Precedence: API overrides SearchHead overrides Indexers
* Defaults to 0 (unlimited) 

indexed_realtime_cluster_update_interval = <int>
* While running an indexed realtime search, if we are on a cluster we need to update the list
* of allowed primary buckets. This controls the interval that we do this. And it must be less 
* than the indexed_realtime_disk_sync_delay. If your buckets transition from Brand New to warm 
* in less than this time indexed realtime will lose data in a clustered environment.
* Precendence: Indexers
* Default: 30
</div>
PREVIOUS
Compare hourly sums across multiple days
  NEXT
Real-time searches and reports in Splunk Web

This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.0.15, 6.1, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.7, 6.1.8, 6.1.9, 6.1.10, 6.1.11, 6.1.12, 6.1.13, 6.1.14


Comments

Bkahlerventer, I've corrected the list of limits.conf arguments. Thank you!

Sophy
June 10, 2014

indexed_realtime_minimum_gap does not exist in current limits.conf maybe it should be indexed_realtime_default_span?

Bkahlerventer
April 3, 2014

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters