Splunk® Enterprise

Managing Indexers and Clusters of Indexers

Download manual as PDF

Splunk Enterprise version 5.0 reached its End of Life on December 1, 2017. Please see the migration information.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

What happens when a peer node goes down

A peer node can go down either intentionally (by invoking the CLI offline command, as described in "Take a peer offline") or unintentionally (for example, by a server crashing).

No matter how a peer goes down, the master coordinates remedial activities to recreate a full complement of bucket copies. This process is called bucket fixing. The master keeps track of which bucket copies are on each node and what their states are (primacy, searchability). When a peer goes down, the master can therefore instruct the remaining peers to fix the cluster's set of buckets, with the aim of returning to the state where the cluster has:

  • Exactly one primary copy of each bucket (the valid state).
  • A full set of searchable copies for each bucket, matching the search factor.
  • A full set of copies (searchable and non-searchable combined) for each bucket, matching the replication factor (the complete state).

Barring catastrophic failure of multiple nodes, the master can usually recreate a valid cluster. If the cluster has a search factor of at least 2, it can do so almost immediately. Whether or not a master can also recreate a complete cluster depends on the number of peers still standing compared to the replication factor. At least replication factor number of peers must remain standing for a cluster to be made complete.

The time that the cluster takes to return to a complete state can be significant, because it must first stream buckets from one peer to another and make non-searchable bucket copies searchable. See "Estimate the cluster recovery time when a peer goes offline" for more information.

Besides the remedial steps to fix the buckets, a few other key events happen when a node goes down:

  • The master rolls the generation and creates a new generation ID, which it communicates to both peers and search heads when needed.
  • Any peers with copies of the downed node's hot buckets roll those copies to warm.

When a peer node gets taken offline intentionally

The offline command removes a peer from the cluster and then stops the peer. It takes the peer down gracefully, allowing most in-progress searches to complete while quickly returning the cluster to a fully searchable state.

Note: The peer goes down after a maximum of 5-10 minutes, even if searches or remedial activities are still in progress.

For the basics on how to take a peer offline, read the topic "Take a peer offline."

The offline command

The offline command has this syntax:

splunk offline

This command causes the following sequence of actions to occur:

1. Partial shutdown. The peer immediately undergoes a partial shutdown. The peer stops accepting both external inputs and replicated data. It continues to participate in searches for the time being.

2. Primacy reassignment. The master reassigns primacy from any primary bucket copies on the peer to available searchable copies of those buckets on other peers. At the end of this step, which should take just a few moments if the cluster has a search factor of at least 2, the cluster returns to the valid state. During this step, the peer's status is ReassigningPrimaries.

3. Generation ID rolling. The master rolls the cluster's generation ID. At the end of this step, the peer no longer joins in new searches, but it continues to participate in any in-progress searches.

4. Full shutdown. The peer shuts down completely after a maximum of 5-10 minutes, or when in-progress searches and primacy re-assignment activities complete - whichever comes first. It no longer sends heartbeats to the master.

5. Restart count. After the peer shuts down, the master waits the length of the restart_timeout attribute (10 minutes by default), set in server.conf. If the peer comes back online within this period, no further remedial activities occur. During this step, the peer's status is Restarting. If the peer does not come back up within the timeout period, its status changes to Down.

6. Remedial activities. If the peer does not restart within the restart_timeout period, the master initiates actions to fix the cluster buckets. It tells the remaining peers to replicate copies of the buckets on the offline peer to other peers. It also compensates for any searchable copies of buckets on the offline peer by instructing other peers to make non-searchable copies of those buckets searchable. At the end of this step, the cluster returns to the complete state. If taking the peer offline results in less than replication factor number of remaining peers, the cluster cannot finish this step and cannot return to the complete state. For details on how bucket-fixing works, see "Bucket-fixing scenarios."

When a peer node goes down unintentionally

When a peer node goes down for any reason besides the offline command, it stops sending the periodic heartbeat to the master. This causes the master to detect the loss and initiate remedial action. The master coordinates essentially the same actions as when the peer gets taken offline intentionally, except for the following:

  • The downed peer does not continue to participate in ongoing searches.
  • The master waits only for the length of the heartbeat timeout (by default, 60 seconds) before reassigning primacy and initiating bucket-fixing actions.

Searches can continue across the cluster after a node goes down; however, searches will provide only partial results until the cluster regains its valid state.

Bucket-fixing scenarios

Bucket fixing involves three activities:

  • Reassigning primary status to other searchable copies of buckets.
  • Making non-searchable copies of buckets searchable.
  • Streaming replicated copies of buckets to peers that don't already have a copy of that bucket.

Converting a searchable copy of a bucket from non-primary to primary happens very quickly, because searchable bucket copies already have index files and so there's virtually no processing involved. (This assumes that there's a spare searchable copy available, which requires the search factor to be at least 2. If not, a non-searchable copy will first have to be made searchable before it can be designated as primary.) However, converting a non-searchable copy of a bucket to searchable takes some time, because the peer must first construct the bucket's index files. For help estimating the time needed to make non-searchable copies searchable, read "Estimate the cluster recovery time when a peer when a peer goes offline".

The two examples in this topic show how a master 1) recreates a valid and complete cluster and 2) creates a valid but incomplete cluster when insufficient nodes remain standing. The process operates the same whether the peer goes down intentionally or unintentionally.

Remember these points:

  • A cluster is "valid" when there is one primary searchable copy of every bucket. Any search across a valid cluster provides a full set of search results.
  • A cluster is "complete" when the cluster has replication factor number of copies of buckets with search factor number of searchable copies.
  • A cluster can be valid but incomplete if there are searchable copies of all buckets, but there are less than replication factor number of copies of buckets. So, if a cluster with a replication factor of 3 has exactly three peer nodes and one of those peers goes down, the cluster can be made valid but it cannot be made complete, since, with just two standing nodes, it's not possible to fulfill the replication factor by maintaining three sets of buckets.

Example: Fixing buckets to create a valid and complete cluster

Assume:

  • The downed peer is part of a cluster with these characteristics:
    • 5 peers, including the downed peer
    • replication factor = 3
    • search factor = 2
  • The downed peer holds these bucket copies:
    • 3 primary copies of buckets
    • 10 searchable copies (including the primary copies)
    • 20 total bucket copies (searchable and non-searchable combined)

When the peer goes down, the master sends messages to various of the remaining peers as follows:

1. For each of the three primary bucket copies on the downed node, the master identifies a node with another searchable copy of that bucket and tells the node to mark the copy as primary.

When this step finishes, the cluster regains its valid state, and any subsequent searches provide a full set of results.

2. For each of the 10 searchable bucket copies on the downed node, the master identifies a node with a non-searchable copy and tells the node to make the copy searchable. The node then begins building index files for that copy.

3. For each of the 20 total bucket copies on the downed node, the master identifies 1) a node with a copy of that bucket and 2) a node that does not have a copy of that bucket. It then tells the node with the copy to stream the bucket data to the second node.

When this last step finishes, the cluster regains its complete state.

Example: Fixing buckets to create a valid but incomplete cluster

Assume:

  • The downed peer is part of a cluster with these characteristics:
    • 3 peers, including the downed peer
    • replication factor = 3
    • search factor = 1
  • The downed peer holds these bucket copies:
    • 5 primary copies of buckets
    • 5 searchable copies (the same as the number of primary copies; because the search factor = 1, all searchable copies must also be primary)
    • 20 total bucket copies (searchable and non-searchable combined)

Since the cluster has just three nodes and a replication factor of 3, the downed node means that the cluster can no longer fulfill the replication factor and therefore cannot be made complete.

When the peer goes down, the master sends messages to various of the remaining peers as follows:

1. For each of the five searchable, primary bucket copies on the downed node, the master first identifies a node with a non-searchable copy and tells the node to make the copy searchable. The node then begins building index files for that copy.

2. The master then tells the node(s) from step 1 to mark the five, newly searchable copies as primary. Unlike the previous example, the step of designating other bucket copies as primary cannot occur until non-searchable copies have been made searchable, because the cluster's search factor = 1.

Once step 2 completes, the cluster regains its valid state. Any subsequent searches provide a full set of results.

3. For the 20 total bucket copies on the downed node, the master cannot initiate any action to make replacement copies (so that the cluster again has three copies of each bucket), because there aren't enough nodes remaining to hold the copies.

Since the cluster cannot recreate a full set of replication factor number of copies of buckets, the cluster remains in an incomplete state.

Pictures, too

The following diagram shows a cluster of five peers, with a replication factor of 3 and a search factor of 2. The primary bucket copies reside on the source peer that's receiving data from a forwarder, with searchable and non-searchable copies of that data spread across the other peers.

Bucket searchability primacy new.png

Note: This is a highly simplified diagram. To reduce complexity, it shows only the buckets for data originating on one peer. In a real-life scenario, most, if not all, of the other peers would also be originating data and replicating it to other peers on the cluster.

The next diagram shows the same simplified version of the cluster, after the source node holding all the primary copies has gone down and the master has finished directing the remaining peers in fixing the buckets:

Bucket generation 2.png

The master directed the cluster in a number of activities to recover from the downed peer:

1. The master reassigned primacy from bucket copies on the downed peer to searchable copies on the remaining peers. When this step finished, it rolled the generation ID.

2. It directed the peers in making a set of non-searchable copies searchable, to make up for the missing set of searchable copies.

3. It directed the replication of a new set of non-searchable copies (1D, 2D, etc.), spread among the remaining peers.

Even though the source node went down, the cluster was able to fully recover both its complete and valid states, with replication factor number of total buckets, search factor number of searchable bucket copies, and exactly one primary copy of each bucket. This represents a different generation from the previous diagram, because primary copies have moved to different peers.

PREVIOUS
How cluster nodes start up
  NEXT
What happens when a peer node comes back up

This documentation applies to the following versions of Splunk® Enterprise: 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters