Use maintenance mode
Certain conditions can generate errors during hot bucket replication and cause the source peer to roll the bucket. While this behavior is generally beneficial to the health of the cluster, it can result in many small buckets across the cluster, if errors occur frequently. Situations that can generate an unacceptable number of small buckets include persistent network problems or repeated offlining of peers.
To stop this behavior, you can temporarily put the cluster into maintenance mode. This can be useful for system maintenance work that generates repeated network errors, such as network reconfiguration. Similarly, if you need to upgrade your peers or otherwise temporarily offline several peers, you can invoke maintenance mode to forestall bucket rolling during that time.
Warning: While the cluster is in maintenance mode, the master will not enforce replication factor or search factor policies. The only bucket fix-up that occurs during maintenance mode is that the master will attempt, when necessary, to reassign primaries to available searchable bucket copies. So, the cluster can be operating under a valid but incomplete status. See Cluster states to understand the implications of this.
Note: The CLI commands
apply cluster-bundle and
rolling-restart incorporate maintenance mode functionality into their behavior by default, so there's no reason to invoke it explicitly when running those commands. A message stating that maintenance mode is on will appear on the master dashboard when you invoke these actions; you can ignore the message.
Enable maintenance mode
Put the cluster into maintenance mode before starting maintenance activity. Once you've finished with maintenance, you should disable maintenance mode.
To invoke maintenance mode, run this CLI command on the master node:
splunk enable maintenance-mode
When you run the
enable command, a message warning of the effects of maintenance mode appears and requires confirmation that you want to continue.
To return to the standard bucket-rolling behavior, run:
splunk disable maintenance-mode
To determine whether maintenance-mode is turned on, run:
splunk show maintenance-mode
A returned value of
1 indicates that maintenance mode is on;
0 indicates that maintenance mode is off.
Take a peer offline
Restart the entire cluster or a single peer node
This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.0.15