Splunk® Enterprise

Knowledge Manager Manual

Download manual as PDF

Splunk Enterprise version 5.0 reached its End of Life on December 1, 2017. Please see the migration information.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Manage search jobs from the operating system

This topic explains how to manage search jobs from the operating system. It has instructions for doing so on two platforms:

  • *nix
  • Windows

For information on how to manage search jobs in Splunk web, see "Supervise your search jobs with the Job Manager" in the Knowledge Manager Manual.

Manage jobs in *nix

When Splunk is running a job, it will manifest itself as a process in the OS called splunkd search. You can manage the job's underlying processes at the OS command line.

To see the job's processes and its arguments, type:

> top
> c

This will show you all the processes running and all their arguments.

Typing ps -ef | grep "search" will isolate all the splunk search processes within this list. It looks like this:

[pie@fflanda ~]$ ps -ef | grep 'search'
530369338 71126 59262   0 11:19AM ??         0:01.65 [splunkd pid=59261] search --id=rt_1344449989.64 --maxbuckets=300 --ttl=600 --maxout=10000 --maxtime=0 --lookups=1 --reduce_freq=10 --rf=* --user=admin --pro --roles=admin:power:user  AhjH8o/Render TERM_PROGRAM_VERSION=303.2
530369338 71127 71126   0 11:19AM ??         0:00.00 [splunkd pid=59261] search --id=rt_1344449989.64 --maxbuckets=300 --ttl=600 --maxout=10000 --maxtime=0 --lookups=1 --reduce_freq=10 --rf=* --user=admin --pro --roles=

There will be two processes for each search job; the second one is a "helper" process used by the splunkd process to do further work as needed. The main job is the one using system resources. The helper process will die on its own if you kill the main process.

The process info includes:

  • the search string (search=)
  • the job ID for that job (id=)
  • the ttl, or length of time that job's artifacts (the output it produces) will remain on disk and available (ttl=)
  • the user who is running the job (user=)
  • what role(s) that user belongs to (roles=)

When a job is running, its data is being written to $SPLUNK_HOME/var/run/splunk/dispatch/<job_id>/ Scheduled jobs (scheduled saved searches) include the saved search name as part of the directory name.

The value of ttl for a process will determine how long the data remains in this spot, even after you kill a job. When you kill a job from the OS, you might want to look at its job ID before killing it if you want to also remove its artifacts.

Manage jobs in Windows

Splunk also spawns a separate process for each search job it runs on Windows. Windows does not have a command-line equivalent for the *nix top command, but there are several ways in which you can view the command line arguments of executing search jobs:

When Splunk runs a search, it writes data for that search into %SPLUNK_HOME\var\run\splunk\dispatch\<epoch_time_at_start_of_search>.<number_separator>. Saved searches are written to similar directories that have a naming convention of "admin__admin__search_" and a randomly-generated hash of numbers in addition to the epoch time.

Use the filesystem to manage jobs

Splunk allows you to manage jobs via creation and deletion of items in that job's artifact directory:

  • To cancel a job, go into that job's artifact directory create a file called 'cancel'.
  • To preserve that job's artifacts (and ignore its ttl setting), create a file called 'save'.
  • To pause a job, create a file called 'pause', and to unpause it, delete the 'pause' file.
PREVIOUS
Supervise your search jobs with the Job Manager
  NEXT
Design macro searches

This documentation applies to the following versions of Splunk® Enterprise: 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters