Search and report in real time
- The mechanics behind real-time search and time range windows.
- How to invoke real-time search in Splunk Web and the CLI.
- Performance considerations for real-time searches and reports.
- How to disable real-time search in
- How to set search and indexer limits on real-time searches.
You can design alerts based on real-time searches that run continuously in the background. Such real-time alerts can provide timelier notifications than alerts that are based on scheduled searches. For more information, see the "Create an alert" topic, in this manual.
You can also display real-time search results and reports in your custom dashboards using the visual dashboard editor and simple XML. For more information about the visual dashboard editor, see "Create simple dashboards with the visual dashboard editor" in this manual.
For more information about using real-time dashboards with advanced features that go beyond what the visual dashboard editor can provide, see Build a real-time dashboard in the Developer manual.
Note: When Splunk is used out-of-the-box, only users with the Admin role can run and save real-time searches. For more information on managing roles and assigning them to users, see "Add and edit roles" in the Admin Manual.
Real-time search mechanics
Real-time searches search through events as they stream into Splunk for indexing. When you kick off a real-time search, the Splunk scans incoming events that contain index-time fields that indicate they could be a match for your search. Splunk identifies these events in the UI as scanned events. This number is cumulative and represents the sum of all events scanned since the search was launched.
As the real-time search runs, Splunk periodically evaluates the scanned events against your search criteria to find actual matches. These events are identified in the UI as matching events. This number represents the number of matching events that currently exist within the sliding time range window that you have defined for the search. As such it can fluctuate up or down over time as Splunk discovers matching events at a faster or slower rate. If you are running the search in Splunk Web, the search timeline also displays the matching events that the search has returned within the chosen time range.
Here's an example of a real-time search with a one minute time range window. At the point that this screen capture was taken, the search had scanned a total of 3,105 events since it was launched. The matching event count of 558 represents the number of events matching the search criteria that had been identified in the past minute. This number fluctuated between 530 and 560 for the following minute; if it had spiked or dropped dramatically, that could have been an indication that something interesting was happening that required a closer look.
As you can see, the newest events are on the right-hand side of the timeline. As time passes, they move right until they move off the left-hand side, disappearing from the time range window entirely.
A real-time search should continue running until you or another user stops it or deletes the search job; it should not "time out" for any other reason. If your events are stopping it could be a performance-related issue (see the subtopic "Expected performance and known limitations," below).
Real-time searches can take advantage of all Splunk search functionality, including advanced functionality like lookups, transactions, and so on. We've also designed search commands that are to be used specifically in conjunction with real-time searches, such as
Real-time searches in Splunk Web
You run a real-time search and build a real-time report in exactly the same way you run standard searches. However, because you are searching a live and continuous stream of data, the timeline updates as the events stream in and you can only view the report in preview mode. Also, some search commands are more applicable to real-time searches than standard searches. For example, streamstats and rtorder were designed for use in real-time searches.
To kick off a real-time search in Splunk Web, use the time range menu to select a preset Real-time time range window, such as 30 seconds or 1 minute. You can also specify a sliding time range window to apply to your real-time search.
Run this search to see pageview events as they stream in.
The raw events that are streamed from the input pipeline are not time-ordered. You can use the
rtorder command to buffer the events from a real-time search and emit them in ascending time order.
The following example keeps a buffer of the last 5 minutes of pageview events, emitting events in ascending time order once they are more than 5 minutes old. Newly received events that are older than 5 minutes are discarded if an event after that time has already been emitted.
eventtype="pageview" | rtorder discard=t buffer_span=5m
Real-time search relies on a stream of events. Thus, you cannot run a real-time search with any other leading search command, such as
| metadata which does not produce events or
| inputcsv which just reads in a file. Also, if you try to send the search results to
| outputcsv, the CSV file will not be written until the real-time search is Finalized.
Real-time reports in Splunk Web
Run a report to preview the IP addresses that access the most web pages. In this case, the
top command returns a table with three columns: clientip, count, and percent. As the data streams in, the table updates with new values.
eventtype="pageview" | top clientip
For each pageview event, add a
count field that represents the number of events seen so far (but do not include the current event in the count).
eventtype="pageview" | streamstats count current=false
You can also drilldown into real-time reports. However, real-time drilldown does not spawn another real-time search. Instead, it spawns a historic search, as you will drilldown into the events that have already been retrieved and indexed. For more information, see Understand table and chart drilldown actions in the User manual.
Real-time searches and reports in the CLI
To run a real-time search in the CLI, replace the command "search" with "rtsearch":
./splunk rtsearch 'eventtype=pageview'
highlight command to emphasize terms in your search results. The following example highlights "GET" in your page view events:
./splunk rtsearch 'eventtype=pageview | highlight GET'
By default, search results have line wrapping enabled. Use the
-wrap option to turn off line wrapping:
./splunk rtsearch 'eventtype=pageview' -wrap 0
Real-time reports in the CLI will also display in preview mode and update as the data streams in.
./splunk rtsearch 'error | top clientip'
-preview option to suppress the results preview:
./splunk rtsearch 'error | top clientip' -preview false
If you turn off preview, you can still manage (Save, Pause, Finalize, or Delete) the search from the Jobs page in Splunk Web. After you finalize the search, the report table will display. For more information, see "Manage your search jobs" in this manual.
You can view all CLI commands by accessing the CLI help reference. For more information, see "Get help with the CLI" in this manual.
Specify real-time time range windows
Time bounds for historical searches are set at the time the search runs. With real-time searches, the time bounds are constantly updating and by default, the results accumulate from the start of the search. You can also specify a time range that represent a sliding window of data, for example, the last 30 seconds. When you specify a sliding window, Splunk takes that amount of time to accumulate data. For example, if your sliding window is 5 minutes, you will not start to see data until after the first 5 minutes have passed. You can override this behavior so that Splunk backfills the initial window with historical data before running in the normal real-time search mode (see "Real-time backfill," below).
You can specify real-time windows with pre-configured options listed in the time range picker, or by defining a custom real-time window in the time range picker.
Time ranges for real-time search follow the same syntax as for historical searches, except that you precede the relative time specifier with "rt", so that it's rt<time_modifier>.
The syntax for real-time time modifiers is:
rt[+|-]<time_integer><time_unit>@<time_unit>. Read about the syntax for time modifiers in the topic, Change the time range of your search.
These values are not designed to be used from within the search language. They are configuration values that you can specify in the time range picker when you select Custom > Real-time. Also, you can use them in times.conf (to add options to the time range picker), or in the saved search dialog, or if you were directly using the REST API to access the Splunk back end search engine.
When you use time range windows with real-time searches, some of the events that occur within the latest second may not display in Splunk. This is expected behavior and is due to the latency between the timestamps within the events and the time when the event arrives. Because the time range window is with respect to the timestamps within the events and not the time when the event arrives, events that arrive after the time window won't display.
Real-time searches over "all time"
It's important to keep in mind that there is a small difference between real-time searches that take place within a set time window (30 seconds, 1 minute, 2 hours) and real-time searches that are set to "all time."
- In "windowed" real time searches, the events in the search can disappear as they fall outside of the window, and events that are newer than the time the search job was created can appear in the window when they occur.
- In "all-time" real-time searches, the window spans all of your events, so events do not disappear once they appear in the window, but events that are newer than the time the search job was created can appear in the window as they occur.
- In comparison, in standard searches, events never disappear from within the set range of time that you are searching and the latest event is always earlier than the job creation time (with the exception of searches that include events that have future-dated timestamps).
For real-time windowed searches, you can specify that Splunk backfill the initial window with historical data. This is run as a single search, just in two phases: first, a search on historical data to backfill events; then, a normal real-time search. Real-time backfill ensures that real-time dashboards seeded with data on actual visualizations and statistical metrics over time periods are accurate from the start.
You can enable real-time backfill in
limits.conf in the [realtime] stanza:
[realtime] default_backfill = <bool> * Specifies if windowed real-time searches should backfill events * Defaults to true
Expected performance and known limitations
Splunk's performance is expected to be acceptable as long as the indexers are not currently heavily loaded and do not have more than a few concurrent real-time searches. However, real-time searches will have a significant impact on performance in high volume environments and network load when you have many concurrent real-time searches.
You can run multiple real-time and historical searches concurrently, within the limits of your hardware. There are no restrictions on separate searches for the same or different users.
When planning your real-time searches, you should consider how it will affect the performance of both:
- The indexer that must forward the live events
- The searcher that must process the live events
The more work that is done on the indexer, the less that is required on the searcher, and vice versa. The indexer is important to the overall system function, so you do not want to burden it with too much filtering of live events. However, if the indexer does not filter at all, the bandwidth required to send all the live events to the searcher may prove costly, especially when multiple real-time searches running concurrently.
In the case where the searcher can't keep up with the indexer, the queue on the index processor will drop events. However, the events will have a sequence number, so we can tell when and how many events were dropped.
How to disable real-time search
Disable real-time search in indexes.conf
Searching in real time may be very expensive on the indexer. If you want to disable it on an indexer, you can edit a [default] setting in that indexer's
[default] enableRealtimeSearch = <bool>
Note: A search head that connects to multiple indexers will still be able to get real-time search results from the indexers that do have it enabled.
Disable real-time search for a user or role
Real-time search is a capability that you can map to specific users or roles in Splunk Web from Manager > Access Controls. By default, the rtsearch capability is assigned to the Admin and Power roles and not the User role. A role without the rtsearch capability will not be able to run a real-time search on that search head, regardless what indexers that search head is connected to.
Setting search limits on real-time searches
You can use the
[search] stanza in
limits.conf to change the maximum number of real-time searches that can run concurrently on your system.
[search] max_rt_search_multiplier = <decimal number> realtime_buffer = <int>
- A number by which the maximum number of historical searches is multiplied to determine the maximum number of concurrent real-time searches. Defaults to 3.
- Note: The maximum number of real-time searches is computed as:
max_rt_searches = max_rt_search_multiplier x max_hist_searches
- The maximum number of accessible events to keep for real-time searches from the UI. Must be >= 1. Defaults to 10000.
- The real-time buffer acts as a circular buffer once this limit is reached.
Setting indexer limits for real-time search
You can use the
[realtime] stanza in
limits.conf to change the default settings for indexer support of real-time searches. These options can be overridden for individual searches via REST API arguments.
[realtime] queue_size = <int> blocking = [0|1] max_blocking_secs = <int> indexfilter = [0|1]
queue_size = <int>
- The size of queue for each real-time search. Must be > 0.
- Defaults to 10000.
- Specifies whether the indexer should block if a queue is full.
- Defaults to false (0).
max_blocking_secs = <int>
- The maximum time to block if the queue is full. This option is meaningless, if
blocking = false.
- Means "no limit" if set to 0.
- Defaults to 60.
indexfilter = [0|1]
- Specifies whether the indexer should pre-filter events for efficiency.
- Defaults to true (1).
Use the timeline to investigate patterns of events
Specify one or multiple indexes to search