Splunk® Enterprise

Managing Indexers and Clusters of Indexers

Splunk Enterprise version 9.0 will no longer be supported as of June 14, 2024. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see How to upgrade Splunk Enterprise.

Troubleshoot Hadoop Data Roll

Issue: Splunk user permission issues disrupt the ability to copy data to HDFS from the Indexers

Data archiving is performed from the Indexers. However, all the archiving setup is done from the Search Head. Permissions issues between the search head and indexer will prevent archiving.

For example, if the indexer permissions are not identical to the search head you might see this exception:

java.io.IOException: Login failure for null from keytab /etc/security/keytabs/splun.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user

You can test persmissions by copying files from the Indexers as the Splunk user to HDFS: hadoop fs -put somefile /user/splunk/archive

Check the following to make sure the Splunk user is configured consistently with the HDFS permissions:

  • Kerberos Keytab path
  • Hadoop Client library path
  • Java library path
  • Splunk user exists on the indexers.
  • Splunk user has permission to write to HDFS.

Issue: You need to collect archiving errors

Trap all archiving errors in a generic way and then raise an alert

To open the archiving dashboard for debugging, access: Settings > Virtual Index > Archived Indexes > View Dashboards

Splunk Web uses the following query:

index=_internal source=*splunk_archiver.log* earliest=-1d | rex max_match=1000 "\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d+ -\d{4} (?<severity>\w+) " | where severity="ERROR"
Last modified on 09 December, 2016
Archive cold buckets to frozen in Hadoop  

This documentation applies to the following versions of Splunk® Enterprise: 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.0.13, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.1.10, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8, 7.3.9, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.0.5, 8.0.6, 8.0.7, 8.0.8, 8.0.9, 8.0.10, 8.1.0, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.10, 8.1.11, 8.1.12, 8.1.13, 8.1.14, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.0.7, 9.0.8, 9.0.9, 9.0.10, 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.1.4, 9.1.5, 9.1.6, 9.2.0, 9.2.1, 9.2.2, 9.2.3, 9.3.0, 9.3.1


Was this topic useful?







You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters