SplitBrainProblem

Split Brain Problem :

 Split brain is an issue that can occur in distributed systems or computer networks where a group of nodes or servers are unable to communicate with each other properly, resulting in a loss of coordination and potentially inconsistent behavior.

In a distributed environment with a central (or leader) server, if the central server dies, the system must quickly find a substitute, otherwise, the system can quickly deteriorate. The scenario in which a distributed system has two or more active leaders is called split-brain. 

In a split brain scenario, the network or communication link between nodes becomes compromised, leading to a division of the cluster into multiple subgroups or partitions. Each subgroup may mistakenly believe that it is the only active and functioning part of the system. This can cause nodes within each subgroup to make independent decisions or perform conflicting actions, leading to data inconsistencies.

For example, in a distributed database system, each subgroup continues to process write operations independently, data inconsistencies can arise. Each subgroup might have its own version of data, leading to conflicts when the network connectivity is restored and the subgroups try to synchronize their data.

----------------------------------

PROBLEM

One of the problems is that we cannot truly know if the leader has stopped for good or has experienced an intermittent failure like a stop-the-world GC pause or a temporary network disruption. Nevertheless, the cluster has to move on and pick a new leader. If the original leader had an intermittent failure, we now find ourselves with a so-called zombie leader. A zombie leader can be defined as a leader node that had been deemed dead by the system and has since come back online. Another node has taken its place, but the zombie leader might not know that yet. The system now has two active leaders that could be issuing conflicting commands. How can a system detect such a scenario, so that all nodes in the system can ignore requests from the old leader and the old leader itself can detect that it is no longer the leader?

Solution

Every time a new leader is elected, the generation number gets incremented. This means if the old leader had a generation number of ‘1’, the new one will have ‘2’. This generation number is included in every request that is sent from the leader to other nodes. This way, nodes can now easily differentiate the real leader by simply trusting the leader with the highest number. The generation number should be persisted on disk, so that it remains available after a server reboot. One way is to store it with every entry in the Write-ahead Log.

Examples

Kafka: To handle Split-brain (where we could have multiple active controller brokers), Kafka uses ‘Epoch number,’ which is simply a monotonically increasing number to indicate a server’s generation.

HDFS: ZooKeeper is used to ensure that only one NameNode is active at any time. An epoch number is maintained as part of every transaction ID to reflect the NameNode generation.

----------------------------------------------------------

How to resolve?

Quorum to rescue.

Lets say two set of servers, which are disconnected from each other, are not allowed to make independent decision.

An action will be considered successful, when a majority of server confirm or agree on the value/action. So, the number of servers in the majority are called Quorum. As an expression, we say 2n+1/2 is the quorum, where n is the no. of failure with which we can sustain.

Quorum ensures that we have enough data copies to withstand server failures, but it doesn't provide strong consistency guarantees to clients.

For instance, if a client writes to the quorum and it only succeeds on one server, the other servers will still have outdated values. When a client reads from the quorum, it might get the latest value if the server with the latest value is available, but it could also get an old value if that server is not accessible at the time of reading.

To prevent such scenarios, it's necessary to track if the quorum agrees on an operation and only provide clients with values that are guaranteed to be available on all servers. In this context, a leader and followers model is used. One server is elected as the leader and the others act as followers. The leader controls and coordinates replication on the followers. The leader's responsibility is to decide which changes should be visible to clients. To achieve this, a high-water mark is used to track the entry in the write ahead log that has been successfully replicated to a quorum of followers. All entries up to the high-water mark are then made visible to the clients. Additionally, the leader shares the high-water mark with the followers to ensure consistency in the event of a leader failure and the ascension of a new leader from the followers.

credit : Martin Fowler

-----------------------------

Here are some ways to solve a split-brain problem:

• Use a consensus algorithm

  Ensure all nodes in the cluster agree on a single course of action. For example, the Paxos or Raft algorithm can be used to ensure that all nodes agree on which node should be the master node.
• Install a Quorum node

  Also referred to as a "Witness", this node should be installed within the cluster to avoid a split brain scenario.
• Use mathematical calculations

  Clusters with an odd number of nodes can use mathematical calculations to prevent split-brain and keep running. They do this by reaching a quorum.
• Proper network configuration

  Nodes should be connected through reliable and redundant network connections. Network partitions should be detected and resolved quickly.
• Automatic or manual reconciliation

  Once the problem has ended, automatic or manual reconciliation might be required in order to have the cluster in a consistent state.

K8s uses the RAFT consensus algorithm for quorum. In order to maintain quorum, you will need floor(n/2)+1 healthy master nodes.

Practicaly this means:

• 1 master node: you will require 1 healthy master node for quorum, the loss of the master node will render the cluster headless.
• 2 master nodes: you will require 2 healthy master nodes for quorum, the loss of either master node will render the cluster headless.
• 3 master nodes: you will require 2 healty master nodes for quorum, the loss of one of the master nodes can be compensated.
• 4 master nodes: you will require 3 healty master nodes for quorum, the loss of one on the master nodes can be compensated. A setup with 4 master nodes has no advantage over a 3 master nodes setup.
• 5 master nodes: you will require 3 healthy master nodes for quorum, the loss of up to two master nodes can be compsensated.
• 6 master nodes: you will require 4 healty master nodes for quorum, the loss of up to two master nodes can be compensated. No advantage compared to 5 master nodes.
• 7 master nodes: you will require 4 healthy master nodes for quorum, the loss of up to three master nodes can be compsensated.

This is the reason why it is recommended to use an odd number of master nodes for the control plane. More then 7 master nodes will result in a overhead for determining cluster membership and quorum, it is not recommended. Depending on your needs, you typically end up with 3 or 5 master nodes.