Manage Splunk Enterprise jobs from the OS

If you have Splunk Enterprise on Microsoft Windows or *nix, you can manage search jobs from the operating system as described in this topic. For information on how to manage search jobs in Splunk Web, see Manage search jobs in this manual.

Manage jobs in *nix

When a search job runs, it will manifest itself as a process in the OS called splunkd search. You can manage the job's underlying processes at the OS command line.

To see the job's processes and its arguments, type:

> top
> c

This will show you all the processes running and all their arguments.

Typing ps -ef | grep "search" will isolate all the Splunk search processes within this list. It looks like this:

[pie@fflanda ~]$ ps -ef | grep 'search'
530369338 71126 59262   0 11:19AM ??         0:01.65 [splunkd pid=59261] search --id=rt_1344449989.64 --maxbuckets=300 --ttl=600 --maxout=10000 --maxtime=0 --lookups=1 --reduce_freq=10 --rf=* --user=admin --pro --roles=admin:power:user  AhjH8o/Render TERM_PROGRAM_VERSION=303.2
530369338 71127 71126   0 11:19AM ??         0:00.00 [splunkd pid=59261] search --id=rt_1344449989.64 --maxbuckets=300 --ttl=600 --maxout=10000 --maxtime=0 --lookups=1 --reduce_freq=10 --rf=* --user=admin --pro --roles=

There will be two processes for each search job; the second one is a "helper" process used by the splunkd process to do further work as needed. The main job is the one using system resources. The helper process will die on its own if you kill the main process.

The process info includes:

• the search string (search=)
• the job ID for that job (id=)
• the ttl, or length of time that job's artifacts (the output it produces) will remain on disk and available (ttl=)
• the user who is running the job (user=)
• what role(s) that user belongs to (roles=)

When a job is running, its data is being written to $SPLUNK_HOME/var/run/splunk/dispatch/<job_id>/ Scheduled jobs (scheduled saved searches) include the saved search name as part of the directory name.

The value of ttl for a process will determine how long the data remains in this spot, even after you kill a job. When you kill a job from the OS, you might want to look at its job ID before killing it if you want to also remove its artifacts.

Manage jobs in Windows

On Windows, each search likewise runs as a separate process. Windows does not have a command-line equivalent for the *nix top command, but there are several ways in which you can view the command line arguments of executing search jobs.

• Use the Process Explorer utility (http://technet.microsoft.com/en-us/scriptcenter/dd742419.aspx) to find the command line of the process that is performing the search.
• Use the TASKLIST and Get-WMIObj commands in the Powershell (http://technet.microsoft.com/en-us/scriptcenter/dd742419.aspx) command line environment to get the ProcessID and CommandLine arguments for a search job.

When a search runs, the data for that search is written into the %SPLUNK_HOME\var\run\splunk\dispatch\<epoch_time_at_start_of_search>.<number_separator> directory. Saved searches are written to similar directories that have a naming convention of "admin__admin__search_" and a randomly-generated hash of numbers in addition to the UNIX time.

Use the filesystem to manage jobs

You can manage a job by creating and deleting files in the job's artifact directory:

• To cancel a job, go into that job's artifact directory create a file called 'cancel'.
• To preserve that job's artifacts (and ignore its ttl setting), create a file called 'save'.
• To pause a job, create a file called 'pause', and to unpause it, delete the 'pause' file.