Splunk® Enterprise

Search Reference

Download manual as PDF

Splunk version 4.x reached its End of Life on October 1, 2013. Please see the migration information.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

About the Search Job Inspector

The search job inspector is a tool which lets you take a closer look at what your search is doing and see where Splunk is spending most of its time. This topic discusses how to use the search job inspector to both troubleshoot the performance of a search and understand the behavior of knowledge objects such as event types, tags, lookups and so on within the search.

Using the search job inspector

You can access the search job inspector for a search job as long as the search artifact still exists (which means that the search has not expired). The search does not still have to be running.

To inspect a search:

1. Run the search.

2. From the Actions dropdown menu, select Inspect search job...

This opens the Search Job Inspector in a new browser window.


To view the properties of a search artifact:

You can use the URL to inspect a search job artifact if you have its search ID (SID). You can find the SID of a search in the job manager (click the Jobs link in the upper right hand corner) or listed in Splunk's dispatch directory, $SPLUNK_HOME/var/run/splunk/dispatch . For more information about the Job Manager, see "Supervise your search jobs" in the User Manual.

If you look at the URI path for the search job inspector window, you will see something like this at the end of the string:

.../inspector?sid=1299600721.22&namespace=search

The sid and namespace are the SID number and the name of the app that it belongs to. Here, the SID is 1299600721.22.

Type the search artifact's SID into the URI path, after sid= and hit return. As long as you have the necessary ownership permissions to view the search, you will be able to inspect it.

Now, what exactly are you looking at?

What the job inspector shows you

While the search is running, the job inspector shows you two different panels. Execution costs lists information about the components of the search and how much impact each component has on the overall performance of the search. Search job properties lists other characteristics of the job. When the search finishes, the job inspector tells you how many results it found and the time it took to complete the search. After the search completes, the job inspector also displays error messages at the top of the screen. Most of the information is self-explanatory, but this section will discuss the panels in more detail.

Execution costs

The Execution costs panel enables you to troubleshoot the efficiency of your search by narrowing it down to the performance impact of specific components that relate to a search-time event processing action. This panel displays a table of the components, their individual durations in seconds, how many times each was invoked while the search ran, and the input and output event counts for each component. The components are listed alphabetically and you will see more or fewer components depending on the search you run. The following table describes the significance of each individual search command and distributed search component. (These are the components you will see if you just run a keyword search.)

Search command component name Description
command.search Once Splunk identifies the events containing the indexed fields matching your search, it looks into the events themselves to identify the ones that match other criteria. These are concurrent operations, not consecutive.
  • command.search.index - tells how long it took to look into the TSIDX files for the location to read in the rawdata.
  • command.search.rawdata - tells how long it took to read the actual events from the rawdata files
  • command.search.typer - tells how long it took identify event types.
  • command.search.kv - tells how long it took to apply field extractions to the events.
  • command.search.fieldalias - tells how long it took to rename fields based according to props.conf.
  • command.search.lookups - tells how long it took to create new fields based on existing fields (perform field lookups).
  • command.search.filter - tells how long it took to filter out events that don't match (for example, fields and phrases).
  • command.search.typer - tells how long it took to assign event types to events.
  • command.search.tags - tells how long it took to assign tags to events.
Distributed search component name Description
dispatch.createProviderQueue The time to connect to all search peers.
dispatch.evaluate The time spent on argument parsing and subsearch evaluation. This is broken down further on each search command that is used.
  • dispatch.evaluate.search - This tells you that the search command was run.
dispatch.fetch The time spent waiting for or fetching events from search peers.
dispatch.preview The time spent generating preview results.
dispatch.process_remote_timeline The time spent decoding timeline information generated by search peers.
dispatch.stream.local The time spent by search head on the streaming part of the search.
dispatch.timeline The time spent generating the timeline and field picker information.

Search job properties

The Search job properties fields are listed in alphabetical order.

Parameter name Description
cursorTime The earliest time from which no events are later scanned. Can be used to indicate progress -- See description for doneProgress.
delegate For saved searches, specifies jobs that were started by the user. Defaults to scheduler.
diskUsage The total amount of disk space used, in bytes.
dispatchState The state of the search. Can be any of QUEUED, PARSING, RUNNING, PAUSED, FINALIZING, FAILED, DONE.
doneProgress A number between 0 and 1.0 that indicates the approximate progress of the search.

doneProgress = (latestTime – cursorTime) / (latestTime – earliestTime)

dropCount For real-time searches only, the number of possible events that were dropped due to the rt_queue_size (default to 100000).
earliestTime The earliest time a search job is configured to start. Can be used to indicate progress -- See description for doneProgress.
eai:acl This describes the app and user-level permissions. For example, is the app shared globally and what users can run or view the search?
eventAvailableCount The number of events that are available for export.
eventCount The number of events returned by the search.
eventFieldCount The number of fields found in the search results.
eventIsStreaming Indicates if the events of this search are being streamed.
eventIsTruncated Indicates if events of the search have not been stored, and thus not available from the events endpoint for the search.
eventSearch Subset of the entire search that is before any transforming commands. The timeline and events endpoint represents the result of this part of the search.
eventSorting Indicates if the events of this search are sorted, and in which order. asc = ascending; desc = descending; none = not sorted
isDone Indicates if the search has completed.
isFailed Indicates if there was a fatal error executing the search. For example, if the search string had invalid syntax.
isFinalized Indicates if the search was finalized (stopped before completion).
isPaused Indicates if the search has been paused.
isPreviewEnabled Indicates if previews are enabled.
isRealTimeSearch Indicates if the search is a real time search.
isRemoteTimeline Indicates if the remote timeline feature is enabled.
isSaved Indicates if the search is saved indefinitely.
isSavedSearch Indicates if this is a saved search run using the scheduler.
isZombie Indicates if the process running the search is dead, but with the search not finished.
keywords All positive keywords used by this search. A positive keyword is a keyword that is not in a NOT clause.
label Custom name created for this search.
latestTime The latest time a search job is configured to start. Can be used to indicate progress -- See description for doneProgress.
numPreviews Number of previews that have been generated so far for this search job.
messages Errors and debug messages.
performance This is another representation of the Execution costs.
remoteSearch The search string that is sent to every search peer.
reportSearch If reporting commands are used, the reporting search.
request GET arguments that the search sends to splunkd.
resultCount The total number of results returned by the search. In other words, this is the subset of scanned events (represented by the scanCount) that actually matches the search terms.
resultIsStreaming Indicates if the final results of the search are available using streaming (for example, no transforming operations).
resultPreviewCount The number of result rows in the latest preview results.
runDuration Time in seconds that the search took to complete.
scanCount The number of events that are scanned or read off disk.
search The search string.
searchProviders A list of all the search peers that were contacted.
sid The search ID number.
statusBuckets Maximum number of timeline buckets.
ttl The time to live, or time before the search job expires after it completes.
Additional info Links to further information about your search. These links may not always be available.
  • timeline
  • field summary
  • search.log


Note: When troubleshooting search performance, it's important to understand the difference between the scanCount and resultCount costs. For dense searches, the scanCount and resultCount are similar (scanCount = resultCount); and for sparse searches, the scanCount is much greater than the result count (scanCount >> resultCount). Search performance should not so much be measured using the resultCount/time rate but scanCount/time instead. Typically, the scanCount/second event rate should hover between 10k and 20k events per second for performance to be deemed good.

Debug messages

If there are errors in your search, these messages (which in previous versions displayed as banners across the dashboard) are presented as DEBUG messages at the top of the Search Job Inspector window. For example, if there are fields missing from your results, the debug messages will say so.

Note: You won't see these messages until the search has completed.

Examples of search job inspector output

Here's an example of the execution costs for a dedup search, run over All time:

* | dedup punct

The search commands component of the Execution costs panel might look something like this:

SearchInspectorExecCostsDedupEx.png

The command.search component and everything under it, gives you the performance impact of the search command portion of your search, which is everything before the pipe character.

Then, command.prededup gives you the performance impact of processing the results of the search command before passing it into the dedup command. The Input count of command.prededup matches the Output count of command.search, and the Input count of command.dedup matches the Output count of command.prededup. In this case, the Output count of command.dedup should match the number of events returned at the completion of the search (which is the value of resultCount, under Search job properties).

Answers

Have questions? Visit Splunk Answers and see what questions and answers the Splunk community has about using the search job inspector.

PREVIOUS
Using the search assistant
  NEXT
Functions for eval and where

This documentation applies to the following versions of Splunk® Enterprise: 4.3, 4.3.1, 4.3.2, 4.3.3, 4.3.4, 4.3.5, 4.3.6, 4.3.7


Comments

What about the "dispatch.stream.remote" field?

Sdlevi27
September 12, 2012

Update docs: There i no longer an actions menu and the job inspector can be found right in the main search interface as the "I" icon.

Sdwilkerson
June 7, 2012

About the Job Inspector documentation: the basic command.search.* and dispatch.* components are mentioned.<br />But what about command.remotetl and dispatch.readEventsInResults.<br /><br />Additionally command.search.typer is mentioned twice.

Pietervi
January 26, 2012

Which unit for the diskUsage ? I assume bytes

Mzorzi
August 23, 2011

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters