Splunk Cloud Platform

Search Manual

Dispatch directory and search artifacts

Each search or alert you run creates a search artifact that must be saved to disk. The artifacts are stored in directories under the dispatch directory. For each search job, there is one search-specific directory. When the job expires, the search-specific directory is deleted.

See About jobs and job management for information about search jobs.

Dispatch directory location

The dispatch directory stores artifacts on nodes where the searches are run. The nodes include search heads, search peers, and standalone Splunk Enterprise instances. The path to the dispatch directory is $SPLUNK_HOME/var/run/splunk/dispatch.

Dispatch directory contents

In the dispatch directory, a search-specific directory is created for each search or alert. Each search-specific directory contains several files including a .srs file of the search results, a search.log file with details about the search execution, and more.

View dispatch directory contents

From a command-line window, or UI window such as Windows Explorer or Finder, you can list the search-specific directories.

For example to view a list in a command-line window, change to the dispatch directory and list the contents in that directory. The following list contains both ad hoc, real-time, and scheduled search-specific directories.

# cd $SPLUNK_HOME/var/run/splunk/dispatch
# ls  

The contents of a search-specific directory includes files and subdirectories. The following example shows the contents of a search-specific directory named 1346978195.13.

#ls 1346978195.13/

Windows users should use dir instead of ls in a command-line window, or Windows Explorer, to see the contents of the dispatch directory.

File descriptions for search-specific directories

The files or subdirectories that appear in a search-specific directory depend on the type of search that you run. The following table lists the files and subdirectories that might appear in your search-specific directories.

File name Contents
args.txt The arguments that are passed to the search process.
alive.token The status of the search process. Specifies if the search is alive or not alive.
audited A flag to indicate the events have been audit signed.
buckets A subdirectory that contains the field picker statistics for each-bucket. The buckets are typically the chunks that are visible in the search histogram UI. This is not related to index buckets.
custom_prop.csv Contains custom job properties, which are arbitrary key values that can be added to search job, retrieved later, are mostly used by UI display goals.
events A subdirectory that contains the events that were used to generate the search results.
generate_preview A flag to indicate that this search has requested preview. This file is used mainly for Splunk Web searches.
info.csv A list of search details that includes the earliest and latest time, and the results count.
metadata.csv The search owner and roles.
peers.csv The list of peers asked to run the search.
pipeline_sets The number of pipeline sets an indexer runs. The default is 1.
remote_events or events_num_num.csv.gz Used for the remote-timeline optimization so that a reporting search that is run with status_buckets>0 can be MapReduced.
request.csv A list of search parameters from the request, including the fields and the text of the search.
results.srs.zst An archive file that contains the search results in a binary serialization format.
rtwindow.czv.gz Events for the latest real-time window when there are more events in the window than can fit in memory. The default limit is 50K.
runtime.csv Pause and cancel settings.
search.log The log from the search process.
sort… A sort temporary file, used by the sort command for large searches.
srtmpfile… A generic search tempfile, used by facilities which did not give a name for their temporary files.
status.csv The current status of the search, for example if the search is still running. Search status can be any of the following: QUEUED, PARSING, RUNNING, PAUSED, FINALIZING, FAILED, DONE.
timeline.csv Event count for each timeline bucket.

The dispatch directory also contains ad hoc data model acceleration summaries. These are different from persistent data model acceleration summaries, which are stored at the index level.

Working with the results.srs.zst file

The results.srs.zst file is an archive file that is created when you run a search and contains the search results in a binary serialization format.

To view search results in the results.srs.zst file you must convert the file into a CSV format. See the toCsv CLI utility in the Troubleshooting Manual for the command line steps to convert the file.

Dispatch directory naming conventions

The names of the search-specific directories in the dispatch directory are based on the type of search. For saved and scheduled searches, the name of a search-specific directory is determined by the following conditions.

  • The name of the search is less than 20 characters and contains only ASCII alphanumeric characters. The search-specific directory name includes the search name.
  • The name of the search is 20 characters or longer, or contains non-alphanumeric characters. A hash is used for the name. This is to ensure a search-specific directory named by the search ID can be created on the filesystem.

A search that contains multiple subsearches might exceed the maximum length for the dispatch directory name. When the maximum length is exceeded, the search fails.

Type of search Naming convention Examples
Local ad hoc search The UNIX time of the search. An ad hoc search.

A real-time ad hoc search.

An ad hoc search that uses a subsearch, which creates two dispatch directories.
Saved search The user requesting the search, the user context the search is run as, the app the search came from, the search string, and the UNIX time.

"count" – run by admin, in user context admin, saved in app search


"Errors in the last 24 hours" – run by somebody, in user context somebody, saved in app search

Scheduled search The user requesting the search, the user context the search is run as, the app the search came from, the search string, UNIX time, and an internal ID added at the end to avoid name collisions. "foo" – run by the scheduler, with no user context, saved in app unix

"foo2" - remote peer search on search head sh01, with admin user context, run by the scheduler, saved in app search

Remote search Searches that are from remote peers start with the word "remote".

"foo2" - remote peer search on search head sh01

Real-time search Searches that are in real-time start with the letters "rt".

Ad hoc real-time search

Replicated search Searches that are replicated search results (artifacts) start with "rsa_".These SIDs occur in search head clusters when a completed search is replicated to a search head other than the search head that originally ran the search
Replicated scheduled search The "rsa_scheduler_" prefix is for replicated results of searches dispatched by the scheduler. The search "foo" is run by the rsa_scheduler, with no user context, saved in app unix.
Report acceleration search These are the probe searches created by datamodel acceleration to retrieve the acceleration percentage from all peers.

Dispatch directory maintenance

The dispatch directory reaper iterates over all of the artifacts every 30 seconds. The reaper deletes artifacts that have expired, based on the last time that the artifacts were accessed and their configured time to live (TTL), or lifetime.

If no search artifacts are eligible to be reaped and the dispatch volume is full, artifacts are not prematurely reaped to recover space. When the dispatch volume is full, new searches cannot be dispatched. You must manually reap the dispatch directory to make space for new search artifacts.

See Extending job lifetimes for information about changing the default lifetime for the search artifact using Splunk Web.

Search artifact lifetime in the dispatch directory

Default lifetime values depend on the type of search. The lifetime countdown begins when the search completes. The following table lists the default lifetime values by type of search.

Search type Default lifetime
Manually run saved search or ad hoc search 10 minutes
Remote search from a peer 10 minutes
Scheduled search

The lifetime varies by the selected alert action, if any. If an alert has multiple actions, the action with the longest lifetime becomes the lifetime for the search artifact. Without an action, the value is determined by the dispatch.ttl attribute in the savedsearches.conf file,. The default value for the dispatch.ttl attribute is twice (2x) the scheduled period.

Alert actions determine the default lifetime of a scheduled search.

  • Email, RSS, and tracking alert actions have a default lifetime of 24 hours.
  • Script alert actions have a default lifetime of 10 minutes.
  • Summary indexing and populate lookup alert actions have a default lifetime of 2 minutes.
Show source scheduled search 30 seconds
Subsearch 5 minutes

Subsearches generate two search-specific directories. There is a search-specific directory for the subsearch and a search-specific directory for the search that uses the subsearch. These directories have different lifetime values.

Change search artifact lifetime

There are a number of ways to change the search artifact lifetime. Modifying the default search behavior affects searches with no other lifetime value or TTL applied.

See How to edit a configuration file.

Search behavior type Process
Global search behavior

In the limits.conf file, set ttl or remote_ttl in the [search] stanza or ttl in the [subsearch] stanza.

Modifying limits.conf provides the default for searches, so it affects searches with no other lifetime value applied.

Search specific behavior

In the savedsearches.conf file, set dispatch.ttl for an individual search.

Or set an individual value for a search when you save the search in Splunk Web.

This overrides the default search behavior.

Searches with alert actions

In the alert_actions.conf, set a ttl value to specify the minimum lifetime of a search artifact if a certain alert action is triggered.

This overrides any shorter lifetime applied to a search.

Clean up the dispatch directory based on the age of directories

As more and more artifacts are added to the dispatch directory, it is possible that the volume of artifacts will cause a adverse effect on search performance or that a warning appears in the UI. The warning threshold is based on the dispatch_dir_warning_size attribute in the limits.conf file.

The default value for the dispatch_dir_warning_size attributes is 5000.

You can move search-specific directories from the dispatch directory to another, destination, directory. You move search-specific directories by using the clean-dispatch command. You must specify a time that is later than the last modification time for the search-specific directories. The destination directory must be on the same file system as the dispatch directory.

The clean-dispatch command is not suitable for production environments and should be used only in certain scenarios as directed by Splunk Support.

Run the command $SPLUNK_HOME/bin/splunk clean-dispatch help to learn how to use the clean-dispatch command.

See Too many search jobs in the Troubleshooting Manual for more information about cleaning up the dispatch directory.

Last modified on 15 December, 2023
View search job properties   Limit search process memory usage

This documentation applies to the following versions of Splunk Cloud Platform: 9.2.2403, 8.2.2112, 8.2.2201, 8.2.2202, 8.2.2203, 9.0.2205, 9.0.2208, 9.0.2209, 9.0.2303, 9.0.2305, 9.1.2308 (latest FedRAMP release), 9.1.2312

Was this topic useful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters