Splunk® Enterprise

Troubleshooting Manual

Acrobat logo Download manual as PDF


Splunk Enterprise version 7.1 is no longer supported as of October 31, 2020. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see How to upgrade Splunk Enterprise.
Acrobat logo Download topic as PDF

Introduction to troubleshooting Splunk Enterprise

This topic is intended as a first step in either diagnosing your Splunk Enterprise problem yourself or asking for help.

Narrow down the problem

For example, if the error occurs in a dashboard or alert, check the underlying search first to see whether the error appears there. When troubleshooting searches, it's almost always best to remove the dashboard layer as soon as possible.

For another example, does the problem exist in one app but not the other? With one user but not admins?

Basically, is there any case for which this does work?

Did the error start occurring after the product was functioning normally?

Yes! So what has changed? Remember to think of both Splunk and non-Splunk factors. Was there a server outage? Network problems? Has any configuration or topology changed?

No, it never functioned normally. Check the operating environment and installation. Start with the system requirements in the Installation Manual.

Resources to help you

Configurations

Splunk has configuration files in several locations, with rules about which files take precedence over each other. Use btool to check which settings your Splunk instance is using. Read about btool in this manual.

The *.conf files are case-sensitive. Check settings and values against the spec and example configuration files in the Admin manual.

There are also a lot of settings in the .conf files that aren't exposed in Splunk Web. It's best to leave these alone unless you know what changing these settings might do.

Splunk log files

Splunk has various internal log files that can help you diagnose problems. Read about the log files in this manual.

Understand how your data gets into Splunk

The Distributed Deployment Manual has a high-level overview of the Splunk data pipeline, breaking it into input, parsing, indexing, and search segments.

For more detail on each segment, see this Community Wiki article about how indexing works.

I've figured out exactly where the problem is

Hey, well done!

Check the (continuously growing) chapter in this manual on some of the most common symptoms and solutions.

If you need additional help or opinions, ask the Splunk community! The Splunk Answers is available to everyone, and is a great resource.

Test potential fixes or workarounds

Once you've found a way to fix the problem, test it! Test any noninvasive changes first. Then, test any changes that would create minor interruptions. Make sure no new issues arise from your tested solution.

Always test invasive or major changes in a sandbox environment before moving them to your production system! Your sandbox should be an independent system that mirrors the affected environment.

Stuck?

If you get stuck at any point, contact Splunk Support. Don't forget to send a diag! Read about making a diag in this manual.

Last modified on 19 March, 2021
PREVIOUS
What's in the Troubleshooting Manual?
  NEXT
Determine which version of Splunk Enterprise you're running

This documentation applies to the following versions of Splunk® Enterprise: 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.0.13, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.1.10, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8, 7.3.9, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.0.5, 8.0.6, 8.0.7, 8.0.8, 8.0.9, 8.0.10, 8.1.0, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.10, 8.1.11, 8.1.12, 8.1.13, 8.1.14, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.0.7, 9.0.8, 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.2.0


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters