Handle Raft issues

If the Raft metadata that underlies search head clustering gets into a bad state on a member, you can often correct the problem by cleaning the member's var/run/splunk/_raft folder. See Fix Raft issues on a member.

If the cluster is unable to elect a captain and maintain a healthy state due to Raft issues, you can clean the Raft folder on all members and then bootstrap the cluster. See Fix the entire cluster.

Fix Raft issues on a member

The primary symptom of a Raft issue is that the member's status appears as "down" when you run splunk show shcluster-status on the captain. To confirm the Raft issue, look in the member's splunkd.log file for an error message that starts with the string "ERROR SHCRaftConsensus".

File corruption in a member's _raft folder is a common cause of Raft issues. You can fix the problem by cleaning the folder on the member. The folder then repopulates from the captain.

To fix a Raft issue, clean the member's _raft folder. Run the splunk clean raft command on the member:

1. Stop the member:

   splunk stop
   

2. Clean the member's raft folder:

   splunk clean raft
   

3. Start the member:

   splunk start
   

The _raft folder will be repopulated from the captain.

Fix the entire cluster

If captain election fails even though a majority of members are available, raft metadata corruption is a likely cause. To confirm, you can examine the members' splunkd.log files for errors that start with the string "ERROR SHCRaftConsensus".

You can resolve the issue by cleaning the folder on all members and then bootstrapping the cluster:

1. Stop all members.
2. Run splunk clean raft on each member:

   splunk clean raft
   

3. Start all members.
4. Select one member to be captain and bootstrap it:

   splunk bootstrap shcluster-captain -servers_list "<URI>:<management_port>,<URI>:<management_port>,..." -auth <username>:<password>
   

   The -servers_list parameter contains a comma-separated list of the cluster members, including the member that you are running the command on. The members are identified by URI and management port.

5. If you are using search peer replication, you must re-add the search peers to one member. See Replicate the search peers across the cluster.