Across the rugged cliffs of Black Stone Gorge stood three ancient sister fortresses: White Rock, Green Rock, and Black Rock. Perched between sheer precipices and rushing rapids, the three strongholds guarded a colossal watergate controlling the flow of the entire river basin.

For a hundred years, the confederacy was ruled by an unbreakable covenant: the opening and closing of the watergate was determined solely by the Council of the Three Elders. Each fortress chieftain bore a bronze seal. In all matters of state, so long as two of the three pressed their seals (a simple majority), the decree was etched into stone, and the river pilots obeyed without question. For generations, this rule of simple majority protected the basin through flood and drought.

Until one year of brutal drought, two downstream strongholds—Red Rock and Yellow Rock—sought to join the league. The three elder chieftains welcomed them with open arms, agreeing to expand the council from the Three Elders to the Council of the Five Sages. Under the charter of the new grand alliance, all future decrees would require three seals out of five (a three-fifths majority).

Amending the covenant seemed a cause for celebration. The grand chieftain penned a proclamation: “From this day forth, the Council of Three is dissolved; the Council of Five assumes the seals.” Fast couriers were dispatched to deliver the edict to all five mountain fortresses.

Yet the catastrophe began the moment the couriers galloped into the wilderness.

The mountain trails were treacherous. Couriers traveling by light skiff downstream reached Red Rock and Yellow Rock in half an hour; others, navigating sheer cliffside ledges, were halted by rockslides, taking a full day and night to scale the heights of Black Rock.

This gave birth to a terrifying temporal dislocation: Green Rock, Red Rock, and Yellow Rock received the edict early. Their three chieftains assembled joyfully in the lower gorge hall. In their eyes, the world was now governed by the Council of Five, and their three seals formed an undeniable three-fifths majority! With beating drums, they stamped a decree: “Raise the watergate immediately to irrigate the downstream plains.”

Upstream, however, the chieftains of White Rock and Black Rock had seen neither courier nor proclamation. In their universe, the Council of Three remained supreme law. The two elders conferred over the drought, resolved to hold back the waters to protect the highland reservoirs, and pressed their bronze seals—two against one, a perfect two-thirds majority! With sounding gongs, they issued a contradictory decree: “Bar the gates shut; release not a drop.”

One single watergate; two completely opposite, irreconcilable decrees. Armed guards under rival banners drew steel at the control winches. Each accused the other of treason, yet both brandished decrees stamped by legitimate, lawful majorities under the charters in their possession!

The alliance stood on the razor’s edge of civil war.

At the final moment, the white-haired elder grand marshal ascended the gatehouse with his staff, roaring at the factions to halt. He picked up the two contradictory, seal-covered scrolls from the stone table and sighed heavily: “Neither of you has betrayed the oath. The fault lies in the proclamation itself! Minds awaken at different hours; mountain paths run with different speeds. Two governing covenants cannot be swapped in the blink of an eye!”

The old marshal drew his sword, carving a deep furrow across the bluestone floor of the council chamber, declaring the law that would govern the gorge for centuries: The Decree of the Conjoined Halls.

“Henceforth, whenever our league admits or expels a fortress, it shall not be accomplished by a single stroke. There must be an intermediate Conjoined Council Period. During this passage, neither the Old Three nor the New Five rule alone. Two halls sit together, and two sets of seals must unite! For any decree to take effect, it must cross two iron thresholds simultaneously: First, a majority of the Old Council of Three must set their seals; Second, a majority of the New Council of Five must also set their seals. Neither may be missing; both thresholds must be satisfied.”

The commanders looked on in stunned realization. Under this conjoined rule, even if messages flew fast or lagged behind, even if some thought the league was transitioning while others thought it complete, no single splinter faction could ever assemble a quorum alone: Attempt to hijack the league under the old charter? Without majority support from the new hall, the decree dies in the chamber; Attempt to rush ahead under the new charter? Without majority approval from the elder hall, the law has no standing! The two overlapping majorities locked together like intertwined iron teeth, mathematically exterminating any possibility of two rival majorities coexisting.

The grand marshal laid down the final decree: Only after this Conjoined Council decree was engraved on the stone stelae of all five fortresses and affirmed across the river could the master issue the final Charter of the Single Hall, smoothly decommissioning the old seals and inaugurating the era of the Five Sages.

Across the centuries that followed, the league expanded and contracted many times—from three to five, and from five back to four. Yet no matter how storms choked the canyon trails, no matter how long couriers were delayed in the drifts, never again did two rival banners clash above the watergate.

— — —

What it is

This is the most subtle and mathematically rigorous safeguard in distributed consensus protocols (such as Raft and etcd): Raft Joint Consensus and Dynamic Cluster Membership Changes.

In distributed consensus engines (such as etcd, Consul, and TiKV), cluster membership (e.g., expanding from 3 nodes to 5 nodes) must be adjusted dynamically as workloads grow. However, performing a direct, single-step cutover from old configuration $C_{\text{old}}$ to new configuration $C_{\text{new}}$ without downtime is hazardous: Because distributed networks inevitably suffer asynchronous latency, network partitions, and message reordering, individual nodes receive and apply configuration updates at different times.

Allowing a single-step cutover leads directly to Split-Brain (Dual Disjoint Majorities):

• Nodes A and B have not yet received the reconfiguration log entry and still believe the cluster is the 3-node set $C_{\text{old}} = {A, B, C}$. Together (2/3), they form a valid majority under $C_{\text{old}}$, electing Leader 1 and committing logs;
• Nodes C, D, and E have already applied the 5-node configuration $C_{\text{new}} = {A, B, C, D, E}$. Together (3/5), they form a valid majority under $C_{\text{new}}$, electing Leader 2 and committing conflicting logs! At that instant, two independent majorities exist simultaneously within the same cluster, causing fatal split-brain and permanent data divergence.

In 2014, Diego Ongaro and John Ousterhout introduced the Joint Consensus mechanism in their seminal Raft thesis:

Cluster reconfiguration must traverse a two-phase transition, where the transitional state is treated as a first-class replicated log entry in the state machine:

1. Enter Joint Consensus ($C_{\text{old,new}}$): Upon receiving a membership change request, the Leader appends a special joint configuration entry, $C_{\text{old,new}}$, to its log; While $C_{\text{old,new}}$ is active, all log commit decisions and Leader elections must obey the Joint Majority Rule:
   • Must secure agreement from a majority of nodes in $C_{\text{old}}$;
   • AND must secure agreement from a majority of nodes in $C_{\text{new}}$.
2. Finalize to $C_{\text{new}}$: Once the $C_{\text{old,new}}$ entry is committed (meaning it has been durably stored across majorities of both $C_{\text{old}}$ and $C_{\text{new}}$, guaranteeing that any future leader must possess this entry), the Leader safely proposes a second entry: the pure $C_{\text{new}}$ configuration; Once $C_{\text{new}}$ is committed, the cluster fully enters the new topology, and decommissioned nodes can be powered down safely.

Why it matters

• Mathematical Prevention of Split-Brain: By the pigeonhole principle, any two majorities of $C_{\text{old}}$ overlap, and any two majorities of $C_{\text{new}}$ overlap. Requiring a joint majority across both configurations guarantees that no two disjoint quorums can ever form at any point in time;
• True Zero-Downtime Operations: Expansions, contractions, and cross-availability-zone migrations occur seamlessly while the cluster continues servicing high-throughput client traffic;
• Single-Server Changes vs. Joint Consensus: Raft also supports an alternative: “Single-Server Membership Changes” (adding or removing one server at a time, where majorities naturally overlap without a joint state). However, production implementations (such as HashiCorp Raft and Apache Ratis) recognize that concurrent network reorderings during back-to-back single-server changes can create subtle corner cases. Therefore, mature enterprise engines embrace Joint Consensus for its uncompromising determinism when modifying arbitrary subsets of nodes.

Metaphor mapping

• Watergate at Black Stone Gorge → Replicated state machine / Shared resource (The core data and operations managed by the consensus cluster)
• Chieftains’ bronze seals → Voting member nodes in Raft cluster (Nodes with active voting rights (Voter status))
• Two seals out of three needed to open → Quorum / Majority rule (Simple majority required to commit logs or elect leaders ($N/2 + 1$))
• Couriers delayed along cliff trails → Asynchronous network delay & partition (Unpredictable network timing causing nodes to receive configs at different moments)
• Three lower fortresses claiming 3-of-5 majority → Disjoint quorum formed under $C_{\text{new}}$ (Updated nodes forming a majority under new config and acting independently)
• Two upper fortresses claiming 2-of-3 majority → Disjoint quorum formed under $C_{\text{old}}$ (Stale nodes forming a majority under old config, causing dual leaders)
• The Decree of the Conjoined Halls → Joint Consensus configuration ($C_{\text{old,new}}$) (A transitional log entry binding both old and new memberships together)
• Must have both old and new majorities → Joint Majority rule (Commits require separate majority approval from both $C_{\text{old}}$ and $C_{\text{new}}$)
• Carving stone stelae before single-hall charter → Committing $C_{\text{old,new}}$ before appending $C_{\text{new}}$ (Ensuring transition is immutable across cluster before shedding old configuration)
• Decades of additions/removals without split banners → Zero-downtime dynamic membership change (Provably safe online reconfiguration without stopping production services)