Manage search jobs from the operating system
This topic explains how to manage search jobs from the operating system. It has instructions for doing so on two platforms:
For information on how to manage search jobs in Splunk web, see "Supervise your search jobs with the Job Manager" in the Knowledge Manager Manual.
Manage jobs in *nix
When Splunk is running a job, it will manifest itself as a process in the OS called
splunkd search. You can manage the job's underlying processes at the OS command line.
To see the job's processes and its arguments, type:
> top > c
This will show you all the processes running and all their arguments.
ps -ef | grep "search" will isolate all the splunk search processes within this list. It looks like this:
[pie@fflanda ~]$ ps -ef | grep 'search' 530369338 71126 59262 0 11:19AM ?? 0:01.65 [splunkd pid=59261] search --id=rt_1344449989.64 --maxbuckets=300 --ttl=600 --maxout=10000 --maxtime=0 --lookups=1 --reduce_freq=10 --rf=* --user=admin --pro --roles=admin:power:user AhjH8o/Render TERM_PROGRAM_VERSION=303.2 530369338 71127 71126 0 11:19AM ?? 0:00.00 [splunkd pid=59261] search --id=rt_1344449989.64 --maxbuckets=300 --ttl=600 --maxout=10000 --maxtime=0 --lookups=1 --reduce_freq=10 --rf=* --user=admin --pro --roles=
There will be two processes for each search job; the second one is a "helper" process used by the
splunkd process to do further work as needed. The main job is the one using system resources. The helper process will die on its own if you kill the main process.
The process info includes:
- the search string (search=)
- the job ID for that job (id=)
- the ttl, or length of time that job's artifacts (the output it produces) will remain on disk and available (ttl=)
- the user who is running the job (user=)
- what role(s) that user belongs to (roles=)
When a job is running, its data is being written to
Scheduled jobs (scheduled saved searches) include the saved search name as part of the directory name.
The value of
ttl for a process will determine how long the data remains in this spot, even after you kill a job.
When you kill a job from the OS, you might want to look at its job ID before killing it if you want to also remove its artifacts.
Manage jobs in Windows
Splunk also spawns a separate process for each search job it runs on Windows. Windows does not have a command-line equivalent for the *nix
top command, but there are several ways in which you can view the command line arguments of executing search jobs:
- By using the Process Explorer (http://technet.microsoft.com/en-us/scriptcenter/dd742419.aspx) utility to find the command line of the process performing the search.
- By using the
Get-WMIObjcommands in the Powershell (http://technet.microsoft.com/en-us/scriptcenter/dd742419.aspx) command line environment to get the
CommandLinearguments for a search job.
When Splunk runs a search, it writes data for that search into
%SPLUNK_HOME\var\run\splunk\dispatch\<epoch_time_at_start_of_search>.<number_separator>. Saved searches are written to similar directories that have a naming convention of "
admin__admin__search_" and a randomly-generated hash of numbers in addition to the epoch time.
Use the filesystem to manage jobs
Splunk allows you to manage jobs via creation and deletion of items in that job's artifact directory:
- To cancel a job, go into that job's artifact directory create a file called 'cancel'.
- To preserve that job's artifacts (and ignore its ttl setting), create a file called 'save'.
- To pause a job, create a file called 'pause', and to unpause it, delete the 'pause' file.
Supervise your search jobs with the Job Manager
Design macro searches
This documentation applies to the following versions of Splunk® Enterprise: 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18