Splunk® Enterprise

Managing Indexers and Clusters of Indexers

Download manual as PDF

Download topic as PDF

Put a peer into detention

When a peer is in the state of detention, its functionality is reduced. It stops replicating data from other peer nodes and, depending on the type of detention, stops indexing most or all data. It continues to participate in searches.

A peer can enter detention either automatically, in response to a low level of free disk space, or manually.

Automatic detention

When a peer enters the detention state automatically, it

  • stops indexing all data, internal and external.
  • stops replicating data from other peer nodes.
  • continues to participate in searches.

The peer node enters the detention state automatically when it runs low on disk space. The setting that controls automatic detention is minFreeSpace in server.conf. The default value is 5000, or 5GB, meaning that the peer enters detention when it has less than 5GB of free disk space.

The peer automatically leaves the detention state when its free disk space grows to exceed minFreeSpace.

Manual detention

When a peer enters the detention state manually, it

  • stops replicating data from other peer nodes.
  • optionally disables external data ports, causing it to stop indexing most types of external data.
  • continues to index internal data.
  • continues to participate in searches.

When you manually put a peer into the detention state, it remains in detention until you remove it from detention. Manual detention persists through peer restart.

The effect of disabling external data ports

You can optionally disable external data ports when setting a peer to manual detention. This brings a halt to the indexing of most external data, including

  • TCP inputs
  • UDP inputs
  • HTTP event collector data inputs
  • data sent from a forwarder to the peer through its receiving port

However, external data can continue to enter the peer through these methods:

  • scripted inputs
  • file and folder monitoring
  • the receivers/stream endpoint

In addition, the indexer can continue to route incoming data to another Splunk Enterprise instance or to a third-party system.

Use cases

Here are some of the key use cases for manual detention:

  • To bring a near halt to the growth of disk usage on the peer, for example, if the peer is close to running out of space.
  • To effectively decommission an old peer, making it available only for searches on existing data.
  • To stop a troublesome peer from handling external or replicated data, while keeping the peer available for diagnostics.
  • To force new data to go to the new peers, when you add new peers to a cluster.
Note: You can also use data rebalancing to move data to new peers. See Rebalance the cluster.
  • To slow the growth of disk usage on a peer that belongs to a pre-approved firewall exception list and needs to continue receiving incoming data. For this use case, you can configure the peer to stop replication activity but continue to consume external data.

Put a peer into manual detention

To put a peer into detention, run the CLI command splunk edit cluster-config with the -manual_detention parameter.

You can set the -manual_detention parameter to one of several values:

  • on. The peer enters detention and closes the ports that consume incoming data. These ports are the receiving TCP, UDP, and HTTP event collector ports.The effect of closing the ports is to halt indexing of most external data. The peer continues to index internal data. The peer stops replicating data from other peer nodes.
  • on_ports_enabled. The peer enters detention and the ports stay open to accept incoming data. The peer continues to index both external and internal data. The peer stops replicating data from other peer nodes.
  • off. The peer is not in detention. This is the default.

You can run this command from the peer itself or from the master node.

Caution: The peer must be in the Up state, or "status," before you put it in detention. For information on how to determine the status of a peer, see View the master dashboard.

To run the command from the peer:

splunk edit cluster-config -auth <username>:<password> -manual_detention [off|on|on_ports_enabled]

To run the command from the master node:

splunk edit cluster-config -auth <username>:<password> -peers <peer_guid1>,<peer_guid2>,... -manual_detention [off|on|on_ports_enabled]

Note the following:

  • -peers specifies the set of peers that you want to put in detention. Identify each peer by its GUID. When you run the command from the master node, you must include this parameter.

Take a peer out of manual detention

To take a peer out of detention:

splunk edit cluster-config -auth <username>:<password> -manual_detention off 

Use a REST endpoint to put the peer into manual detention

You can use the REST endpoint cluster/slave/control/control/set_manual_detention to put a peer into manual detention.

Note: A previous endpoint, cluster/slave/control/control/set_detention_override, has been deprecated. Use cluster/slave/control/control/set_manual_detention in its place.

See the REST API documentation for cluster/slave/control/control/set_manual_detention.

View the detention state

You can view the states, detention-related or otherwise, of all peers from the master dashboard. See View the master dashboard.

These are the possible detention states:

  • AutomaticDetention. Peer entered detention automatically.
  • ManualDetention. Peer entered detention manually and no longer consumes external data.
  • ManualDetention-PortsEnabled. Peer entered detention manually and continues to consume external data.


You can also use the DMC to view the state of the peers.

In addition, some CLI commands also provide peer state information:

  • To view the state of all peers, run this command on the master:

splunk list cluster-peers
  • To view the state of a single peer, run this command on the peer:

splunk list cluster-config
PREVIOUS
Remove excess bucket copies from the indexer cluster
  NEXT
Remove a peer from the master's list

This documentation applies to the following versions of Splunk® Enterprise: 6.5.1612 (Splunk Cloud only), 6.6.0, 6.6.1, 6.6.2


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters