Troubleshooting Manual

 


Intro to troubleshooting Splunk Enterprise

NOTE - Splunk version 4.x reached its End of Life on October 1, 2013. Please see the migration information.

Intro to troubleshooting Splunk Enterprise

This topic is intended as a first step in either diagnosing your Splunk Enterprise problem yourself or asking for help.

Narrow down the problem

For example, if the error occurs in a dashboard or alert, check the underlying search first to see whether the error appears there. When troubleshooting searches, it's almost always best to remove the dashboard layer as soon as possible.

For another example, does the problem exist in one app but not the other? With one user but not admins?

Basically, is there any case for which this does work?

Did the error start occurring after the product was functioning normally?

Yes! So what has changed? Remember to think of both Splunk and non-Splunk factors. Was there a server outage? Network problems? Has any configuration or topology changed?

No, it never functioned normally. Check the operating environment and installation. Start with the system requirements in the Installation Manual.

Resources to help you

Configurations

Splunk has configuration files in several locations, with rules about which files take precedence over each other. Use btool to check which settings your Splunk instance is using. Read about btool in this manual.

The *.conf files are case-sensitive. Check settings and values against the spec and example configuration files in the Admin manual.

There are also a lot of settings in the .conf files that aren't exposed in Splunk Web. It's best to leave these alone unless you know what changing these settings might do.

Splunk log files

Splunk has various internal log files that can help you diagnose problems. Read about the log files in this manual.

Understand how your data gets into Splunk

The Distributed Deployment Manual has a high-level overview of the Splunk data pipeline, breaking it into input, parsing, indexing, and search segments.

For more detail on each segment, see this Community Wiki article about how indexing works.

I've figured out exactly where the problem is

Hey, well done!

Check the (continuously growing) chapter in this manual on some of the most common symptoms and solutions.

If you need additional help or opinions, ask the Splunk community! The Community Wiki, Splunk Answers, and the #splunk IRC channel on efnet are available to everyone and provide a great resource.

Test potential fixes or workarounds

Once you've found a way to fix the problem, test it! Test any noninvasive changes first. Then, test any changes that would create minor interruptions. Make sure no new issues arise from your tested solution.

Always test invasive or major changes in a sandbox environment before moving them to your production system! Your sandbox should be an independent system that mirrors the affected environment.

Stuck?

If you get stuck at any point, contact Splunk Support. Don't forget to send a diag! Read about making a diag in this manual.

This documentation applies to the following versions of Splunk: 4.2.3 , 4.2.4 , 4.2.5 , 4.3 , 4.3.1 , 4.3.2 , 4.3.3 , 4.3.4 , 4.3.5 , 4.3.6 , 4.3.7 , 5.0 , 5.0.1 , 5.0.2 , 5.0.3 , 5.0.4 , 5.0.5 , 5.0.6 , 5.0.7 , 5.0.8 , 5.0.9 , 5.0.10 , 6.0 , 6.0.1 , 6.0.2 , 6.0.3 , 6.0.4 , 6.0.5 , 6.0.6 , 6.1 , 6.1.1 , 6.1.2 , 6.1.3 , 6.1.4 , 6.2.0 View the Article History for its revisions.


You must be logged into splunk.com in order to post comments. Log in now.

Was this documentation topic helpful?

If you'd like to hear back from us, please provide your email address:

We'd love to hear what you think about this topic or the documentation as a whole. Feedback you enter here will be delivered to the documentation team.

Feedback submitted, thanks!