Introduction to troubleshooting Splunk Enterprise

This topic is intended as a first step in either diagnosing your Splunk Enterprise problem yourself or asking for help.

Narrow down the problem

For example, if the error occurs in a dashboard or alert, check the underlying search first to see whether the error appears there. When troubleshooting searches, it's almost always best to remove the dashboard layer as soon as possible.

For another example, does the problem exist in one app but not the other? With one user but not admins?

Basically, is there any case for which this does work?

Did the error start occurring after the product was functioning normally?

Yes! So what has changed? Remember to think of both Splunk and non-Splunk factors. Was there a server outage? Network problems? Has any configuration or topology changed?

No, it never functioned normally. Check the operating environment and installation. Start with the system requirements in the Installation Manual.

Resources to help you

Configurations

Splunk has configuration files in several locations, with rules about which files take precedence over each other. Use btool to check which settings your Splunk instance is using. Read about btool in this manual.

The *.conf files are case-sensitive. Check settings and values against the spec and example configuration files in the Admin manual.

There are also a lot of settings in the .conf files that aren't exposed in Splunk Web. It's best to leave these alone unless you know what changing these settings might do.

Splunk log files

Splunk has various internal log files that can help you diagnose problems. Read about the log files in this manual.

Understand how your data gets into Splunk

The Distributed Deployment Manual has a high-level overview of the Splunk data pipeline, breaking it into input, parsing, indexing, and search segments.

For more detail on each segment, see this Community Wiki article about how indexing works.

I've figured out exactly where the problem is

Hey, well done!

Check the (continuously growing) chapter in this manual on some of the most common symptoms and solutions.

If you need additional help or opinions, ask the Splunk community! The Splunk Answers is available to everyone, and is a great resource.

Test potential fixes or workarounds

Once you've found a way to fix the problem, test it! Test any noninvasive changes first. Then, test any changes that would create minor interruptions. Make sure no new issues arise from your tested solution.

Always test invasive or major changes in a sandbox environment before moving them to your production system! Your sandbox should be an independent system that mirrors the affected environment.

Stuck?

If you get stuck at any point, contact Splunk Support. Don't forget to send a diag! Read about making a diag in this manual.