Splunk® Enterprise

Distributed Search

Download manual as PDF

Download topic as PDF

Handle Raft issues

If the Raft metadata that underlies search head clustering gets into a bad state on a member, you can often correct the problem by cleaning the member's var/run/splunk/_raft folder. See Fix Raft issues on a member.

If the cluster is unable to elect a captain and maintain a healthy state due to Raft issues, you can clean the Raft folder on all members and then bootstrap the cluster. See Fix the entire cluster.

Fix Raft issues on a member

The primary symptom of a Raft issue is that the member's status appears as "down" when you run splunk show shcluster-status on the captain. To confirm the Raft issue, look in the member's splunkd.log file for an error message that starts with the string "ERROR SHCRaftConsensus".

File corruption in a member's _raft folder is a common cause of Raft issues. You can fix the problem by cleaning the folder on the member. The folder then repopulates from the captain.

To fix a Raft issue, clean the member's _raft folder. Run the splunk clean raft command on the member:

  1. Stop the member:
    splunk stop
    
  2. Clean the member's raft folder:
    splunk clean raft
    
  3. Start the member:
    splunk start
    

The _raft folder will be repopulated from the captain.

Fix the entire cluster

If captain election fails even though a majority of members are available, raft metadata corruption is a likely cause. To confirm, you can examine the members' splunkd.log files for errors that start with the string "ERROR SHCRaftConsensus".

You can resolve the issue by cleaning the folder on all members and then bootstrapping the cluster:

  1. Stop all members.
  2. Run splunk clean raft on each member:
    splunk clean raft
    
  3. Start all members.
  4. Select one member to be captain and bootstrap it:
    splunk bootstrap shcluster-captain -servers_list "<URI>:<management_port>,<URI>:<management_port>,..." -auth <username>:<password>
    
  5. If you are using search peer replication, you must re-add the search peers to one member. See Replicate the search peers across the cluster.
PREVIOUS
Runtime considerations
  NEXT
Overview of search head pooling

This documentation applies to the following versions of Splunk® Enterprise: 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.6.0, 6.6.1, 6.6.2


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters