Troubleshooting

Troubleshooting

composite.xml and compositelibmap.xml

If you hand-edit the Splunk Server's XML configuration files, bad syntax can cause unexpected problems. To debug them, you can see the complete XML configuration of the running splunkd process by looking at these files. They contain the server's internal XML configuration after it has loaded all modules and performed all substitutions. You'll see the composite configuration created from your local .xml files. Even the comments from your files are there, as a debugging aid.

# more $SPLUNK_HOME/var/run/splunk/composite.xml

# more $SPLUNK_HOME/var/run/splunk/compositelibmap.xml

Test Configuration Changes

To put configuration file changes into effect and test for errors, restart the server with this command.

# splunk restart

If everything is configured correctly, the Splunk Server will immediately begin loading data from its configured sources. To see if any data has been indexed, load the Splunk search UI into your browser and click on the Saved Splunk labeled "All".

If no results appear after the server has been running for a minute, your setup may be configured incorrectly.

The most basic way to check up on the server is to watch its log file with this command:

# tail -f $SPLUNK_HOME/var/log/splunk/splunkd.log

If you see a message like the one below, it means there's an error in your configuration files.

1119904858 ERROR loader : Error initializing component config

Server crashes are most often caused by bad syntax in configuration files. Please report crashes to Splunk Support.

Core Files

To collect a useable core file, use ulimit to remove any maximum file size setting before starting Splunk.

# ulimit -c unlimited

# splunk restart

This setting only affects the processes you start in a particular shell, so you may wish to do it in a new session. For Linux, start Splunk with the --nodaemon option and then start the web interface manually with "splunk start splunkweb" in another shell.)

Depending on your system, it may be named something like core.1234, where the number indicates the process id and be the same location as the splunkd executable.

Debug mode

Starting splunk in debug mode provides significantly more assistance for the Splunk Support team in identifying a problem. Here is how to do this:

• Stop Splunk, if it is running.
• Save your existing splunkd.log file by moving it to a new filename, like splunkd.log.old.
• Restart Splunk in debug mode with "splunk start --debug"
• When you notice the problem, stop Splunk.
• Move the new splunkd.log file elsewhere and replace your old one.
• Restart Splunk normally.

Built-In Tools

Statistics Page

The Splunk Server displays a summary of server statistics on its Admin page tab labeled Splunk Watch or Splunk Professional Watch

Splunklogger Index

The Splunk Server creates its own log messages and saves them in a separate index called splunklogger that appears on the index menu. To check the progress of a specific file, try a search like this in the Splunk box at the top of the page. (The wildcard character is easier than typing the full pathname.)

domain::splunklogger *20050515.log

If you have a custom module installed, search for splunklogger events that also include your module's name.

domain::splunklogger mysql

Splunk Test Tool

This command is designed to help admins who have problems getting splunk up and running. Run this command to start a test instance of the Splunk Server, gather system information while it runs, and then shut it down.

# splunk test

The test tool gathers the following information and logs it to $SPLUNK_HOME/var/log/splunk for use in customer support and debugging. Splunk Support staff may ask you to run the test tool and send them a tarball of the log directory.

All Systems

• df
• uname -a
• netstat -a

Linux

• getconf GNU_LIBPTHREAD_VERSION
• free
• strace ./splunk start --nodaemon

Solaris

• prtconf
• vmstat
• truss ./splunk start --nodaemon

Darwin (Mac OS X)

• vmstat
• ktrace

Splunk Server Log Files

The most basic way to watch the server in action is to type this command.

# tail -f $SPLUNK_HOME/var/log/splunk/splunkd.log

If the Splunk Server is working normally, you'll see new messages written to splunkd.log every few seconds. If the server does not write to the log every few seconds, the last entry may indicate a problem. If you're having trouble with the server, look for ERROR messages such as these (in this case, the problem is incorrect XML syntax in a module configuration file.)

05-08-2005 13:11:27.533 ERROR loader - Parsing error in config file for module /opt/splunk/etc/modules/tailingprocessor - skipping install 05-08-2005 13:11:27.537 ERROR loader - /opt/splunk/etc/modules/tailingprocessor/config.xml:StartTag: invalid element name

Log File Directory

The splunkd process writes all of its log files to the directory $SPLUNK_HOME/var/log/splunk . By default it should contain the log files shown below. If you install or activate other modules, you will see additional log files from each one.

searchhistory.log

A log of all searches performed on the server since installation or the most recent "splunk clean" command. The historyprocessor module indexes this file into the built-in splunklogger index for self-splunking through the Web interface.

splunkd_stdout.log

The Unix standard output device for the server.

splunkd_stderr.log

The Unix standard error device for the server.

splunklogger.log

A subset of the Splunk Server's own log events since installation or the most recent "splunk clean" command. The statsprocessor module indexes this file into the built-in splunklogger index for self-splunking through the Web interface.

splunkd.log

A record of actions made by the Splunk Server. May be requested by Splunk Support for troubleshooting purposes.

splunkSearch.log

Events logged by the Web search interface.

 !DirMonCompleted.log

A log of files successfully processed by the directory monitor.

 !DirMonDuplicate.log

A log of files rejected by the directory monitor as already loaded.

 !DirMonError.log

A log of files the directory monitor failed to open and load successfully.

Alerts (Splunk Professional)

Live Splunks are a Splunk Professional feature that let you schedule a Saved Splunk to run on a regular schedule and alert you via the Splunk interface, email, a shell command or RSS if results meet certain conditions. For instructions see the Splunk User Manual section on Saved Splunks and Live Splunks.

Indexing Performance

If Splunk falls behind indexing your data, try these tricks to speed it up.

Hardwire sources as single-line events

The default value of the property AUTO_LINEMERGE is True , which it should be. But if you know a specific source or sourcetype consists of only single-line events, you can turn off the linemerge function on it for faster processing. See the section on Properties.

Use FIFO instead of files

If Splunk falls behind tailing your syslog file(s), reconfigure syslog to write to a FIFO and use the syslogFIFO module to load them. FIFOs live in RAM rather than on disk, so they're much faster for reading and writing.

splunkd Monitor

The monitor is a separate process that watches for the splunkd pid file (in $SPLUNK_HOME/var/run/splunk) and restarts splunkd if it crashes. It checks once per minute and will attempt to restart splunkd three times before it gives up. To start splunkd with the monitor:

splunk start monitor

The monitor also logs status messages to $SPLUNK_HOME/var/log/splunk/splunkmon.log for each status check.

To stop the monitor:

splunk stop monitor

• Was this topic useful?
• Post a comment