Use maintenance mode
Certain conditions can generate errors during hot bucket replication and cause the source peer to roll the bucket. While this behavior is generally beneficial to the health of the indexer cluster, it can result in many small buckets across the cluster, if errors occur frequently. Situations that can generate an unacceptable number of small buckets include persistent network problems or repeated offlining of peers.
To stop this behavior, you can temporarily put the cluster into maintenance mode. This can be useful for system maintenance work that generates repeated network errors, such as network reconfiguration. Similarly, if you need to upgrade your peers or otherwise temporarily offline several peers, you can invoke maintenance mode to forestall bucket rolling during that time.
Maintenance mode works the same for single-site and multisite clusters. It has no notion of sites.
Warning: While the cluster is in maintenance mode, the master will not enforce replication factor or search factor policies. The only bucket fix-up that occurs during maintenance mode is that the master will attempt, when necessary, to reassign primaries to available searchable bucket copies. So, the cluster can be operating under a valid but incomplete status. See Indexer cluster states to understand the implications of this.
Note: The CLI commands
apply cluster-bundle and
rolling-restart incorporate maintenance mode functionality into their behavior by default, so there's no reason to invoke it explicitly when running those commands. A message stating that maintenance mode is on will appear on the master dashboard when you invoke these actions.
Enable maintenance mode
Put the cluster into maintenance mode before starting maintenance activity. Once you have finished with maintenance, you should disable maintenance mode.
To invoke maintenance mode, run this CLI command on the master node:
splunk enable maintenance-mode
When you run the
enable command, a message warning of the effects of maintenance mode appears and requires confirmation that you want to continue.
Important: Maintenance mode does not persist across master restarts. If the master restarts, you need to re-enable maintenance mode.
Disable maintenance mode
To return to the standard bucket-rolling behavior, run:
splunk disable maintenance-mode
Determine maintenance mode status
To determine whether maintenance mode is on, run:
splunk show maintenance-mode
Maintenance mode : 0 indicates maintenance mode is OFF.
Maintenance mode : 1 indicates maintenance mode is ON.
Take a peer offline
Restart the entire indexer cluster or a single peer node
This documentation applies to the following versions of Splunk® Enterprise: 6.1, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.7, 6.1.8, 6.1.9, 6.1.10, 6.1.11, 6.1.12, 6.1.13, 6.1.14, 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.2.14, 6.2.15