Search and report in real time

Search and report in real time

With real-time searches and reports, you can search events before they are indexed and preview reports as the events stream in. This topic discusses:

• How to invoke real-time search in Splunk Web and the CLI.
• How to specify real-time windows in your searches.
• Performance considerations for real-time searches and reports.
• How to disable real-time search from indexes.conf.
• How to set search and indexer limits on real-time searches.

You can also display real-time search results and reports in your custom dashboards using the visual dashboard editor and simple XML. For more information about the visual dashboard editor, see "Create simple dashboards with the visual dashboard editor" in this manual.

For more information about using real-time dashboards with more advanced features than the visual dashboard editor can provide, see Build a real-time dashboard in the Developer manual.

Real-time searches in Splunk Web

Run a real-time search and build a real-time report exactly the same way you run regular searches. However, because you are searching a live and continuous stream of data, the timeline will update as the events stream in and you can only view the report in preview mode. Also, some search commands will be more applicable (for example, streamstats and rtorder) for use in real-time.

To kick off a real-time search in Splunk Web, use the time range dropdown menu to select Real-time. You can also specify a sliding time range window to apply to your real-time search.

Run this search to see pageview events as they stream in.

The raw events that are streamed from the input pipeline are not time-ordered. You can use the rtorder command to buffer the events from a real-time search and emit them in ascending time order.

The following example keeps a buffer of the last 5 minutes of pageview events, emitting events in ascending time order once they are more than 5 minutes old. Newly received events that are older than 5 minutes are discarded if an event after that time has already been emitted.

Real-time search relies on a stream of events. Thus, you cannot run a real-time search with any other leading search command, such as | metadata which does not produce events or | inputcsv which just reads in a file. Also, if you try to send the search results to | outputcsv, the CSV file will not be written until the real-time search is Finalized.

Real-time reports in Splunk Web

Run a report to preview the IP addresses that access the most web pages. In this case, the top command returns a table with three columns: clientip, count, and percent. As the data streams in, the table updates with new values.

For each pageview event, add a count field that represents the number of events seen so far (but do not include the current event in the count).

You can also drilldown into real-time reports. However, real-time drilldown does not spawn another real-time search. Instead, it spawns a historic search, as you will drilldown into the events that have already been retrieved and indexed. For more information, see Understand table and chart drilldown actions in the User manual.

Real-time searches and reports in the CLI

To run a real-time search in the CLI, replace the command "search" with "rtsearch":

./splunk rtsearch 'eventtype=pageview'

Use the highlight command to emphasize terms in your search results. The following example highlights "GET" in your page view events:

./splunk rtsearch 'eventtype=pageview | highlight GET'

By default, search results have line wrapping enabled. Use the -wrap option to turn off line wrapping:

./splunk rtsearch 'eventtype=pageview' -wrap 0

Real-time reports in the CLI will also display in preview mode and update as the data streams in.

./splunk rtsearch 'error | top clientip'

Use the -preview option to suppress the results preview:

./splunk rtsearch 'error | top clientip' -preview false

If you turn off preview, you can still manage (Save, Pause, Finalize, or Delete) the search from the Jobs page in Splunk Web. After you finalize the search, the report table will display. For more information, see "Manage your search jobs" in this manual.

You can view all CLI commands by accessing the CLI help reference. For more information, see "Get help with the CLI" in this manual.

Specify real-time time range windows

Time bounds for historical searches are set at the time the search runs. With real-time searches, the time bounds are constantly updating and by default, the results accumulate from the start of the search. You can also specify a time range that represent a sliding window of data, for example, the last 30 seconds. When you specify a sliding window, Splunk takes that amount of time to accumulate data. For example, if your sliding window is 5 minutes, you will not start to see data until after the first 5 minutes have passed.

You can specify real-time windows with pre-configured options listed in the time range picker, or by defining a custom real-time window in the time range picker.

Time ranges for real-time search follow the same syntax as for historical searches, except that you precede the relative time specifier with "rt", so that it's rt<time_modifier>.

The syntax for real-time time modifers is: rt[+|-]<time_integer><time_unit>@<time_unit>. Read about the syntax for time modifiers in the topic, Change the time range of your search.

These values are not designed to be used from within the search language. They are configuration values that you can specify in the time range picker when you select Custom > Real-time. Also, you can use them in times.conf (to add options to the time range picker), or in the saved search dialog, or if you were directly using the REST API to access the Splunk back end search engine.

When you use time range windows with real-time searches, some of the events that occur within the latest second may not display in Splunk. This is expected behavior and is due to the latency between the timestamps within the events and the time when the event arrives. Because the time range window is with respect to the timestamps within the events and not the time when the event arrives, events that arrive after the time window won't display.

Expected performance and known limitations

Splunk's performance is expected to be acceptable as long as the indexers are not currently heavily loaded and do not have more than a few concurrent real-time searches. However, real-time searches will have a significant impact on performance in high volume environments and network load when you have many concurrent real-time searches.

You can run multiple real-time and historical searches concurrently, within the limits of your hardware. There are no restrictions on separate searches for the same or different users. However, you cannot run a single search on both real-time data and historical data at the same time.

When planning your real-time searches, you should consider how it will affect the performance of both:

• The indexer that must forward the live events
• The searcher that must process the live events

The more work that is done on the indexer, the less that is required on the searcher, and vice versa. The indexer is important to the overall system function, so you do not want to burden it with too much filtering of live events. However, if the indexer does not filter at all, the bandwidth required to send all the live events to the searcher may prove costly, especially when multiple real-time searches running concurrently.

In the case where the searcher can't keep up with the indexer, the queue on the index processor will drop events. However, the events will have a sequence number, so we can tell when and how many events were dropped.

How to disable real-time search

Disable real-time search in indexes.conf

Searching in real time may be very expensive on the indexer. If you want to disable it on an indexer, you can edit a [default] setting in that indexer's indexes.conf.

[default]
enableRealtimeSearch = <bool>

Note: A search head that connects to multiple indexers will still be able to get real-time search results from the indexers that do have it enabled.

Disable real-time search for a user or role

Real-time search is a capability that you can map to specific users or roles in Splunk Web from Manager > Access Controls. By default, the rtsearch capability is assigned to the Admin and Power roles and not the User role. A role without the rtsearch capability will not be able to run a real-time search on that search head, regardless what indexers that search head is connected to.

Setting search limits on real-time searches

You can use the [search] stanza in limits.conf to change the maximum number of real-time searches that can run concurrently on your system.

[search]
max_rt_search_multiplier = <decimal number>
realtime_buffer = <int>

max_rt_search_multiplier

• A number by which the maximum number of historical searches is multiplied to determine the maximum number of concurrent real-time searches. Defaults to 3.
• Note: The maximum number of real-time searches is computed as: max_rt_searches = max_rt_search_multiplier x max_hist_searches

realtime_buffer

• The maximum number of accessible events to keep for real-time searches from the UI. Must be >= 1. Defaults to 10000.
• The real-time buffer acts as a circular buffer once this limit is reached.

Setting indexer limits for real-time search

You can use the [realtime] stanza in limits.conf to change the default settings for indexer support of real-time searches. These options can be overridden for individual searches via REST API arguments.

[realtime] 
queue_size = <int>
blocking = [0|1] 
max_blocking_secs = <int>
indexfilter = [0|1]

queue_size = <int>

• The size of queue for each real-time search. Must be > 0.
• Defaults to 10000.

blocking =[0|1]

• Specifies whether the indexer should block if a queue is full.
• Defaults to false (0).

max_blocking_secs = <int>

• The maximum time to block if the queue is full. This option is meaningless, if blocking = false.
• Means "no limit" if set to 0.
• Defaults to 60.

indexfilter = [0|1]

• Specifies whether the indexer should pre-filter events for efficiency.
• Defaults to true (1).

• Was this topic useful?
• Post a comment