Splunk® Enterprise

Troubleshooting Manual

Splunk Enterprise version 9.0 will no longer be supported as of June 14, 2024. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see How to upgrade Splunk Enterprise.

How to file a great Support case

When you contact Support, you can save time by starting out with everything we'll need!

Here are some ideas to get you started.

Describe the issue

Where does the issue occur? On a forwarder? On an indexer?

What elements are present for the issue? What's the timeline leading to the error? What processes are running when the error appears?

What behavior do you observe, compared to what you expect? Be specific: for example, how late is "late"?

Try to classify the problem:

  • Is it a searching issue? These include Splunk Web, management, roles, apps, views and dashboards, search language.
  • Is it a back end issue? These problems could include crashing, OS issues, REST API, or SDK.
  • Is it a configuration issue? These include extractions, input configurations, forwarding, apps disabling, or authentication.
  • Is it a performance problem?

Send diagnosic files

Most Support cases are for functional problems: the software has been configured to do something, but it is behaving in an unexpected way. Splunk Support needs both the context of the problem and insight into the instance that is not performing as expected. That insight comes in the form of a "diag" or diagnostic file, which is essentially a snapshot of the configuration of the Splunk platform instance and the recent logs from that instance.

You can make a diag on any instance type: forwarder, indexer, search head, or deployment server. If you have a forwarder and a receiver that are not working together correctly, send us diags of both. Label the diags so it's clear which instance each is from. If you have many forwarders, send only one representative forwarder diag.

The diag tarball does not contain any of your indexed data, but you can examine its contents before sending it. Read about what you can include or exclude from diags in Generate a diagnostic file in this manual.

Splunk Support might request another diag after recommending a change or update to the instance. This diag can verify that the change has been applied and examine its effect. It is not unusual to have multiple updated diags for a single case. If you send multiple diags, label each one clearly.

Collecting core files

Splunk Support might ask you to collect a core file.

Begin by setting the ulimit to remove any maximum file size setting and restarting Splunk.

# ulimit -c unlimited
# splunk restart

This setting only affects the processes you start from the shell where you ran the ulimit command. To find out where core files land in your particular UNIX flavor and version, consult the system documentation. The below text includes some general rules that may or may not apply.

On UNIX, if you start Splunk with the --nodaemon option (splunk start --nodaemon), it may write the core file to the current directory. Without the flag the expected location is / (the root of the filesystem tree). However, various platforms have various rules about where core files go with or without this setting. Consult your system documentation. If you do start splunk with --nodaemon, you will need to, in another shell, start the web interface manually with splunk start splunkweb.

Depending on your system, the core may be named something like core.1234, where '1234' is the process ID of the crashing program.

Working with LDAP configurations

If you are having trouble setting up LDAP, Support will typically need the following information:

  • The authentication.conf file from $SPLUNK_HOME/etc/system/local/.
  • An ldif for a group you are trying to map roles for.
  • An ldif for a user you are trying to authenticate as.

In some instances, a debug splunkd.log or web_service.log is helpful.

Last modified on 27 January, 2023
Contact Support   Generate a diagnostic file

This documentation applies to the following versions of Splunk® Enterprise: 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.0.13, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.1.10, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8, 7.3.9, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.0.5, 8.0.6, 8.0.7, 8.0.8, 8.0.9, 8.0.10, 8.1.0, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.10, 8.1.11, 8.1.12, 8.1.13, 8.1.14, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.0.7, 9.0.8, 9.0.9, 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.1.4, 9.2.0, 9.2.1

Was this topic useful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters