Splunk® Enterprise

Troubleshooting Manual

Download manual as PDF

Splunk version 4.x reached its End of Life on October 1, 2013. Please see the migration information.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Command line tools for use with Support

This topic contains information on CLI tools to help with troubleshooting Splunk software. Most of these tools are invoked using the Splunk CLI command "cmd".

Do not use these tools without first consulting with Splunk Support.

For general information about using the CLI in Splunk, see "Get help with the CLI" in the Admin Manual.


Runs the specified utility in $SPLUNK_HOME/bin with the required environment variables preset.

To see which environment variables will be set, run "splunk envvars".


  ./splunk cmd btool inputs list
  ./splunk cmd /bin/ls

Syntax: cmd <command> [parameters...]

Objects: None

Required Parameters: None

Optional Parameters: None


View or validate Splunk software configuration files, taking into account configuration file layering and user/app context.


	btool <CONF_FILE> list [options]
        btool check [options]

Objects: None

Required Parameters: None

Optional Parameters:

	--user=SPLUNK_USER  		View the configuration data visible to the given user

	--app=SPLUNK_APP    		View the configuration data visible from the given app

	--dir=DIR           		Read configuration data from the given absolute path instead of $SPLUNK_HOME/etc

	--debug             		Print and log extra debugging information


List: ./splunk cmd btool [--app=app_name] conf_file_prefix list [stanza_prefix]

Add: ./splunk cmd btool [--app=app_name] conf_file_prefix add

Delete: ./splunk cmd btool --app=app_name --user=user_name conf_file_prefix delete stanza_name [attribute_name]

For more information, read "Use btool to troubleshoot configurations."


Queries the fishbucket for checkpoints stored by monitor inputs. Any changes made to the fishbucket using btprobe take effect only after a restart. Shut down your Splunk software before using btprobe on it. For up-to-date usage, run btprobe --help.

You must specify either -d <dir> or --compute-crc <file>.

There are two ways to invoke this tool:

1. btprobe [-h or --help] -d <btree directory> [-k <hex key OR ALL> | --file <filename>] [--salt <salt>] [--validate] [--reset] [--bytes <bytes>] [-r]

This method queries the specified BTree for the given key or file.

	 -d        	 Directory that contains the btree index. (Required)

	 -k        	 Hex crc key or ALL to get all the keys.
	 --file    	 File to compute the crc from.
	 -r        	 Rebuild the btree .dat files (i.e., var/lib/splunk/fishbucket/splunk_private_db/ 
	      (One of -k and --file must be specified.

	 --validate   Validate the btree to look for errors.
         --salt       Salt the crc if --file param is specified.
	 --reset      Reset the fishbucket for the given key or file in the btree. 
                      Resetting the checkpoint for an active monitor input reindexes data, resulting in increased license use.
	 --bytes      Number of bytes to read when calculating CRC (default 256).
	 --sourcetype	 Sourcetype to load configurations and check Indexed Extraction
	                	 and compute CRC accordingly.

2. btprobe [-h or --help] --compute-crc <filename> [--salt <salt>] [--bytes <bytes>]

This method computes a crc from the specified file, using the given salt if any.

  • Example: ./btprobe -d /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db -k 0xe8d117ddba85e714 --validate
  • Example: ./btprobe -d /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db --file /var/log/inputfile --salt SOME_SALT
  • Example: ./btprobe --compute-crc /var/log/inputfile --salt SOME_SALT


$SPLUNK_HOME/bin/splunk cmd classify <path/to/myfile> <mysourcetypename> 


Diagnoses the health of your buckets and can rebuild search data as necessary.

        [--hots]          include hot buckets in scan
	[--warms]         include warm buckets in scan
	[--colds]         include cold buckets in scan
	[--thawed]        include thawed buckets in scan
	[--all]           include all types of buckets
	[--index <index>] only scan specified index (defaults to all)
	[--mode metadata] only supported mode is 'metadata'
	[--verbose]       display diagnostic info while scanning
	[--repair]        attempt to repair buckets if errors found

./splunk --repair works only with buckets created by Splunk Enterprise 4.2 or later.

For more information, read "How Splunk stores indexes" in the Managing Indexers and Clusters Manual.


./splunk cmd locktest


./splunk cmd locktool

Usage :

lock : [-l | --lock ] [dirToLock] <timeOutSecs>

unlock [-u | --unlock ] [dirToUnlock] <timeOutSecs>

Acquires and releases locks in the same manner as splunkd. If you were to write an external script to copy db buckets in and out of indexes you should acqure locks on the db colddb and thaweddb directories as you are modifying them and release the locks when you are done.


./splunk cmd parsetest

	parsetest "<string>" ["<sourcetype>|source::<filename>|host::<hostname>"]
	parsetest file <filename> ["<sourcetype>|host::<hostname>"]
	parsetest "10/11/2009 12:11:13" "syslog"
	parsetest file "foo.log" "syslog"


Simple utility tool for testing modular regular expressions.

./splunk cmd pcregextest mregex=<regex>

Usage: pcregextest mregex="query_regex" (name="subregex_value")* (test_str="string to test regex")?

Example: pcregextest mregex="[[ip:src_]] [[ip:dst_]]" ip="(?<ip>\d+[http://docs.splunk.com/Documentation/Splunk/4.3.1/Troubleshooting/dotnum dotnum]{3})" dotnum="\.\d+" test_str=""

That is, define modular regex in the 'mregex' parameter. Then define all the subregexes referenced in 'mregex'. Finally you can provide a sample string to test the resulting regex against, in 'test_str'.



./splunk cmd searchtest search



./splunk cmd signtool [-s | --sign] [<dir to sign>]


./splunk cmd signtool [-v | --verify] [<dir to verify>]

Using logging configuration at /Applications/splunk/etc/log-cmdline.cfg.

Allows verification and signing splunk index buckets. If you have signing set up in a cold to frozen script. Signtool allows you to verify the signatures of your archives.


This will take a look at your time-series index files (or "tsidx files"; they are appended with .tsidx) and verify that they meet the necessary format requirements. It should also identify any files that are potentially causing a problem

go to the $SPLUNK_HOME/bin directory. Do "source setSplunkEnv".

Then use tsidxprobe to look at each of your index files with this little script you can run from your shell (this works with bash):

  • for i in `find $SPLUNK_DB -name '*.tsidx'`; do tsidxprobe $i >> tsidxprobeout.txt; done

(If you've changed the default datastore path, then this should be in the new location.)

The file tsidxprobeout.txt will contain the results from your index files. You should be able to gzip this and attach it to an email and send it to Splunk Support.


This utility script searches for tsidx files at a specified starting location, runs tsidxprobe for each one, and outputs the results to a file.

From $SPLUNK_HOME/bin, call it like this:

splunk cmd python tsidx_scan.py [path]


splunk cmd python tsidx_scan.py /opt/splunk/var/lib/splunk

If you omit the optional path, the scan starts at $SPLUNK_DB

The output is written to the file tsidxprobe.YYYY-MM-DD.txt in the current directory.


This tool "walks the lexicon" to tell you which terms exist in a given index. For example, with some search commands (like tstat), the field is in the index; for other terms it is not. Walklex can be useful for debugging.

Walklex outputs a line with three pieces of information:

  • term ID (a unique identifier)
  • number of occurrences of the term
  • term


From $SPLUNK_HOME/bin, type

./splunk cmd walklex </path/to/tsidx_file.tsidx> "<key>::<value>"

It recognizes wildcards:

./splunk cmd walklex </path/to/tsidx_file.tsidx> ""

./splunk cmd walklex </path/to/tsidx_file.tsidx> "*::*"

Empty quotes return all results, and asterisks return all keys or all values (or both, as in the example above).


./splunk cmd walklex </path/to/tsidx_file.tsidx> "token"

Anonymize data samples to send to Support
I can't find my data!

This documentation applies to the following versions of Splunk® Enterprise: 4.3, 4.3.1, 4.3.2, 4.3.3, 4.3.4, 4.3.5, 4.3.6, 4.3.7, 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18, 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.1, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.7, 6.1.8, 6.1.9, 6.1.10, 6.1.11, 6.1.12, 6.1.13, 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.2.14


Thanks, Skawasaki. Added!

Jlaw splunk, Splunker
July 16, 2015

We're missing the walklex command to "walk the lexicon" in tsdix file:<br /><br />./splunk cmd walklex /path/to/tsidx_file.tsidx "key::value"<br />./splunk cmd walklex /path/to/tsidx_file.tsidx "token"<br />./splunk cmd walklex /path/to/tsidx_file.tsidx ""<br />./splunk cmd walklex /path/to/tsidx_file.tsidx "*::*"<br /><br />"" for everything, and "*::*" for all key-values

Skawasaki splunk
April 28, 2014

Thanks, Sowings! I added some info to the fsck section. For some good guidance about using fsck, see http://docs.splunk.com/Documentation/Splunk/5.0.2/Indexer/HowSplunkstoresindexes#Troubleshoot_your_buckets

Jlaw splunk, Splunker
March 8, 2013

Requesting a sample invocation for fsck.

Sowings splunk, Splunker
March 8, 2013

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters