Manage jobs in the OS
Manage jobs in the OS
This topic explains how to manage search jobs from the operating system. It has instructions for doing so on two platforms:
- *nix
- Windows
For information on how to manage search jobs in Splunk web, see "Manage Jobs in Splunk Web" in this manual.
Manage jobs in *nix
When Splunk is running a job, it will manifest itself as a process in the OS called splunkd search. You can manage the job's underlying processes at the OS command line.
To see the job's processes and its arguments, type:
> top > c
This will show you all the processes running and all their arguments.
Typing ps -ef | grep "splunkd search" will isolate all the splunk search processes within this list. It looks like this:
[pie@fflanda ~]$ ps -ef | grep "splunkd search" pie 21368 19460 96 13:51 ? 00:01:18 splunkd search --search=search sourcetype="access_combined" --id=1247691084.1188 --maxbuckets=300 --ttl=600 --maxout=500000 --maxtime=0 --lookups=1 --reduce_freq=10 --user=pie --pro --roles=admin:user pie 21371 21368 0 13:51 ? 00:00:00 splunkd search --search=search sourcetype="access_combined" --id=1247691084.1188 --maxbuckets=300 --ttl=600 --maxout=500000 --maxtime=0 --lookups=1 --reduce_freq=10 --user=pie --pro --roles=admin:user pie 22804 20379 0 13:52 pts/9 00:00:00 grep splunk-search
There will be two processes for each search job; the second one is a 'helper' process used by the splunkd process to do further work as needed. The main job is the one using system resources. The helper process will die on its own if you kill the main process.
The process info includes:
- the search string (search=)
- the job ID for that job (id=)
- the ttl, or length of time that job's artifacts (the output it produces) will remain on disk and available (ttl=)
- the user who is running the job (user=)
- what role(s) that user belongs to (roles=)
When a job is running, its data is being written to $SPLUNK_HOME/var/run/splunk/dispatch/<job_id>/
Scheduled jobs (scheduled saved searches) include the saved search name as part of the directory name.
The value of ttl for a process will determine how long the data remains in this spot, even after you kill a job.
When you kill a job from the OS, you might want to look at its job ID before killing it if you want to also remove its artifacts.
Manage jobs in Windows
Splunk also spawns a separate process for each search job it runs on Windows. Windows does not have a command-line equivalent for the *nix top command, but there are several ways in which you can view the command line arguments of executing search jobs:
- By using the Process Explorer (http://technet.microsoft.com/en-us/scriptcenter/dd742419.aspx) utility to find the command line of the process performing the search.
- By using the
TASKLISTandGet-WMIObjcommands in the Powershell (http://technet.microsoft.com/en-us/scriptcenter/dd742419.aspx) command line environment to get theProcessIDandCommandLinearguments for a search job.
When Splunk runs a search, it writes data for that search into %SPLUNK_HOME\var\run\splunk\dispatch\<epoch_time_at_start_of_search>.<number_separator>. Saved searches are written to similar directories that have a naming convention of "admin__admin__search_" and a randomly-generated hash of numbers in addition to the epoch time.
Use the filesystem to manage jobs
Splunk allows you to manage jobs via creation and deletion of items in that job's artifact directory:
- To cancel a job, go into that job's artifact directory create a file called 'cancel'.
- To preserve that job's artifacts (and ignore its ttl setting), create a file called 'save'.
- To pause a job, create a file called 'pause', and to unpause it, delete the 'pause' file.
This documentation applies to the following versions of Splunk: 4.1 , 4.1.1 , 4.1.2 , 4.1.3 , 4.1.4 , 4.1.5 , 4.1.6 , 4.1.7 , 4.1.8 , 4.2 , 4.2.1 , 4.2.2 , 4.2.3 , 4.2.4 , 4.2.5 , 4.3 , 4.3.1 , 4.3.2 View the Article History for its revisions.