▶ Cinematic fable · Watch on YouTube ▶ 影片版寓言 · 在 YouTube 观看
大衍王朝的疆域由五座雄关共同守望:除了坐镇关中平原的中央京畿,东南西北四方各设一座重镇,合称“五岳磐石”。
王朝立国百余年,天下之所以政令通达、井然有序,全赖太祖定下的一套“金钟晨传”律法。
每日破晓,中央京畿的钟鼓楼上便敲响重达万斤的盘龙金钟。三声洪亮沉雄的钟鸣荡过云霄,守钟司随即派出四队背负朱漆铜筒的快马信使,快马加鞭赶往四方重镇。铜筒内装着盖有中枢相印的红泥木札,上面烙着当朝年号与今日干支。
按太祖律例,天下大事不必五关尽数赞同,只要有三座要塞每日如期收到相印木札,中枢政令便具法统。于是,商旅过关可凭验印文书纳税放行,粮仓可依令开仓赈济,钱庄票号亦能兑付汇票。百余年来,哪怕偶尔有一路信使在暴雨泥泞中折损马匹,其余三关与京畿依旧钟声相闻,天下大势坚如磐石。
唯有北境极高处的“绝顶玄冰关”,孤悬于万仞雪峰之上。
那里常年罡风呼啸,冻云低垂,驻守关隘的慕容统领麾下仅有数百精锐。每当入冬,千丈悬崖上的栈道便覆满坚冰,信使往往要历经数日艰险方能攀上峰顶。但无论风雪多大,慕容统领每日清晨必登上烽火台,引颈眺望南方云海,等候来自京畿的信使。
那一年寒露方过,一场罕见的万古暴雪骤然席卷北溟。
狂风卷着如斗大雪,一夜之间将绝壁栈道齐腰斩断。暴雪连绵三月不绝,积雪深达数丈,飞鸟绝迹,信鸽在风暴中折翅坠崖。通往绝顶关的咽喉要道彻底化作一片白茫茫的死域。
山峰之上,慕容统领与部众陷入了死一般的沉寂。
十日过去,南方没有信使。三十日过去,云海茫茫,依然没有一骑飞驰而来。
到了冬至,慕容统领独自坐在冰封的石室内,面色凝重如铁。按照太祖法典《天符典则》中的古训:“若三十昼夜不闻中枢金钟、不见相印木札,则必是京畿遭逢大难、社稷有倒悬之危。守关之将不可坐待乾坤失序,当即起‘易元更始’之意,将本关历数自进一纪,自刻年号,代行代望。”
慕容统领深吸一口长气,自炭炉中拔出烧红的铁凿,在一块平整的枣木牌上用力刻下:“承天改元,大寒元年”。
他将木牌插上关门,然而悬崖下唯有怒号的狂风,无人回应。
又过了三个月,春汛迟迟未至,冰雪依旧封锁万壑。慕容统领再次拔出凿子,将木牌拔出削平,咬牙刻上:“大寒二年”。
一年、两年、三年……
在极北孤峰的漫长极夜与酷寒中,老将的心志被冻成了偏执的坚冰。他深信南方的四座关隘早已在中原大乱中化作焦土,天下唯有自己这一座孤峰尚存忠骨。每一次沙漏见底、每一次严冬更迭,他便固执地挥动铁凿,将年号一步步往上翻跳:
“大寒三年”、“大寒四年”……直至“大寒十年”!
而在群山之南、云海之下,世界究竟是何光景?
那里没有战火,亦没有崩塌。那场万古暴雪仅仅锁住了北境绝顶的局部山隘,关中平原与东南西三关依旧四季分明、岁稔年丰。京畿的金钟每日清晨如常敲响,内阁首辅与文武百官端坐奉天殿,批阅着天下三关呈送的万千奏章。首辅的印札每日如期送达,商旅络绎不绝,百姓安居乐业。在所有人眼中,王朝依然安详地运转在平稳的治世之中。
直到第四年暮春,骄阳化尽了千山积雪。
北境古道破冰复通。慕容统领顶盔贯甲,带领残存的百名精兵顺着泥泞的栈道疾驰而下。老将腰间悬挂着一柄沉重的纯铁大印,战马两侧高悬着巨大的白布长幡,幡面上以焦墨狂草写着四个大字——“大寒十年”!
残骑一路疾驰,撞入京畿城门,直闯奉天殿大理石玉阶之下。
老将翻身下马,捧着那块烙着“大寒十年”的木牌,高声长喝:“塞北孤军血战十载,不见京畿音信!今遵太祖成宪,以‘大寒十年’更始大历,请中枢勘验天命!”
此言一出,金銮殿上一片死寂。文武百官面面相觑,内阁首辅更是脸色惨白。
为何?因为太祖在《天符典则》的卷首,白纸黑字写下过一条不容任何人违背的最高铁律:
“天下诸关,凡见有携‘更高年号’之使者入关呈文,不论因由、不论来历,当朝执政与坐朝首辅必须即刻交卸相印、退避阶下,朝政立时封驳停摆,以待重选大统;违者视为欺天僭越,天下共诛之。”
这条铁律最初的用意本极高尚:为了防止篡位者霸占中枢死不放权,只要天下有任何一处发现了更具合法性的新纪元,旧的执政者必须立刻无条件退位认降。
然而此刻,灾难降临了。
“大寒十年”赫然压过了朝廷现行的一切案卷。按照祖宗成宪,首辅长叹一声,只得颤抖着解下腰间黄金相印,置于御案之上,躬身退入侧廊。
京畿的行政机器瞬间瘫痪。
六部衙门封存大印,天下钱庄关闭兑房,原本在关口排队入关的数万车队被拦在护城河外,通商关牒一律作废。整个大衍王朝的商贸与赋税,在正午时分戛然而止。
朝廷停摆之后,依律当场开启大选,推选新朝领袖。
群臣取来慕容统领随身携带的铁箱,当众撬开封条,欲查验他这十年来治理塞北的卷宗与功勋。
然而,箱盖掀开,满殿皆惊。
铁箱里除了一层层发黄的风干草料与几本空无一字的白纸簿册,竟没有半卷像样的文书!在这孤绝的三年里,慕容统领未曾收取过一文钱粮,未曾安抚过一名流民,甚至连关外有何敌情变化都一无所知。他的卷宗底账,依然停留在四年前大雪封山的那一日!
群臣哗然。按大选之规,没有功勋底册的将领,根本无法获得三关镇将的赞同票。慕容统领不仅当不上首辅,甚至连参选的资格都极为勉强。
但首辅已经退位,玉玺已经封存。为了收拾残局,四方重镇只得被迫重新推举新官,文书往来耗费数日,商贾滞塞损银巨万。
满朝文臣武将皆顿足长叹:一个与世隔绝、毫无治国功绩的雪山遗老,仅凭在冰窟里胡乱拨动自己的算盘刻下的虚妄年号,竟能仅凭一纸空文,就把天下运转了数载的盛世中枢逼入绝境!
这一场无妄之灾,史称“雪岭扰朝”。
事后,朝廷痛定思痛,召集三山鸿儒与司天监智囊,在《天符典则》中正式增设了一道保命修正令——“预询之策”。
新令昭告天下五关:
往后凡有孤悬绝地之关将,若察觉中枢音讯断绝、意欲改元更始,严禁私自刻凿更高年号!
该将必须先派遣一名轻骑斥候,手持白羽,向邻近各关进行“预先垂询”:
斥候到了邻关城下,先问两件事:第一,“你关案头近三十日内,可曾听闻中央京畿的金钟晨鸣?”第二,“我关所存卷宗底账,比诸公如何?”
邻关守将若隔着护城河答道:“京畿晨钟日日不绝,我等案头朱印崭新,天命未改,尔等安敢生乱?”斥候便必须即刻收回白羽,回山安抚将士,绝不准将年号往上跳动哪怕半格!
唯有当四方重镇中有多半关隘尽皆回应“京畿确实钟息鼓绝、中枢已倾”,且该斥候出示的功勋卷宗确实比其余各关更为详实丰厚时,关将方准正式加刻年号、兴师入京开启大选。
自此之后,冰峰纵使再度被大雪封禁十年、百年,雪山孤客也休想仅凭虚妄的空头年号,惊扰平原之上的朗朗乾坤。
——到这儿你大概已经认出来了:这便是分布式共识基石 Raft 协议中至关重要的安全扩展机制——Pre-Vote 协议与扰乱节点(Disruptive Server)防御。
这是什么
在标准 Raft 共识协议(Diego Ongaro & John Ousterhout, 2014)中,任期(Term)充当着分布式系统的逻辑时钟(Logical Clock),用于识别并废弃过期的陈旧信息。
标准 Raft 协议有一条至高无上、硬编码在每个节点事件循环中的核心铁律:
“若任何 RPC 请求或响应中包含的任期 $T > currentTerm$,则节点必须立即将自身 $currentTerm$ 更新为 $T$,并无条件退化为跟随者(Follower)状态。”
这条规则本意是为了确保当集群选举出新任 Leader 时,旧的僵尸 Leader 能够在收到新消息的第一时间主动让位,防止发生双主脑裂。
然而,这条看似纯粹优雅的数学不变量,却在真实的工业级网络世界中催生了一个致命缺陷——扰乱节点问题(Disruptive Server Problem):
- 瞬态隔离与任期膨胀:
考虑一个 5 节点的 Raft 集群 ${N_1, N_2, N_3, N_4, N_5}$,其中 $N_1$ 是当前任期为 1 的合法 Leader。此时,节点 $N_5$ 因机架交换机抖动、短暂网络分区或长 GC 停顿,与集群中其余 4 个节点失联。
$N_5$ 收不到来自 $N_1$ 的周期性心跳广播(
AppendEntries),其本地的选举超时计时器(electionTimeout)随即触发。$N_5$ 将自身转换为候选人(Candidate),把自身任期加 1(变为 2),并向外广播RequestVote。 由于处于隔离状态,$N_5$ 永远收不到半数以上的赞同票。超时后,$N_5$ 再次重试:任期变为 3、4、5……甚至由于网络长期单向断开,其任期一路膨胀到了 100。 - 重连后的致命扰乱(The Disruptive Rejoin):
当网络分区自愈、$N_5$ 重新连入集群时,灾难降临了。
$N_5$ 携带自身高达 100 的任期,向全集群广播
RequestVote。 此时正稳定处理海量业务读写的合法 Leader $N_1$(此时任期仍为 1)收到了这条请求。由于 $100 > 1$,$N_1$ 必须严格遵从 Raft 铁律,瞬间卸任 Leader,退化为 Follower! 此时,整个集群失去了 Leader,所有正在排队的客户端写请求瞬间阻塞、超时,集群服务发生剧烈抖动。 - 荒谬的结局:
$N_1$ 退位后,集群进入重新选举。那么 $N_5$ 能成为新 Leader 吗?绝无可能!
因为在 Raft 协议中,为了保证领袖完备性(Leader Completeness),选民在投票时会严格比对候选人的日志:若候选人日志的最后一条条目(
lastLogIndex与lastLogTerm)没有选民自身的新,选民将拒绝投票。$N_5$ 在隔离期间与世隔绝,日志极其陈旧,根本拿不到多数派选票。 最终的结果极其讽刺:一个日志落后、毫无可能当选的故障节点,仅仅凭借自己在小黑屋里狂刷出来的虚假高 Term,白白废黜了健康的 Leader,平白无故制造了一场集群不可用风暴!
为了根治这个顽疾,Diego Ongaro 在其斯坦福博士论文《Consensus: Bridging Theory and Practice》(Chapter 9.6)中提出了 Pre-Vote 扩展算法:
在进入真正的选举之前,增加一个投石问路的预候选人(PreCandidate)状态:
- 第一步:发起 Pre-Vote 预探
当节点的
electionTimeout超时后,它不能直接增加自身的currentTerm,也不进入正式的Candidate状态,而是切换为PreCandidate。 它向集群广播PreVote请求,携带的假设任期是 $currentTerm + 1$。 - 第二步:选民双重门禁审核
接收到
PreVote请求的节点,只有在同时满足以下两个严苛条件时,才投出赞成票:- 日志完备性(Log Completeness):候选人的日志至少和自己一样新;
- 领袖租约有效性(Active Leader Lease / Heartbeat Check):在自身的最小选举超时时间(
minElectionTimeout)内,没有收到过来自当前合法 Leader 的心跳消息。
- 第三步:按需晋升
如果集群中的合法 Leader 依然健在,其余健康节点都在持续接收心跳,它们会果断拒绝给 $N_5$ 投赞成票。
$N_5$ 拿不到多数派的 Pre-Vote 响应,其
currentTerm便永远不会真正递增!它只能作为一个安静的旁观者,默默等待网络恢复后被 Leader 的心跳重新同化。 唯有当多数派节点确实也认为 Leader 已经失联、且候选人日志足够新时,该节点才获准正式将currentTerm加 1,蜕变为真正的Candidate发起真正的选举。
为什么重要
在现代大规模云原生基础设施中,网络不是理想的同轴电缆,而是充满瞬态抖动、丢包、灰度部署重启与垃圾回收毛刺的混乱深渊:
- 杜绝单节点毛刺引发集群级雪崩: 在 Kubernetes、etcd 以及分布式数据库集群中,云主机或容器的短暂网络闪断(Network Flap)、vMotion 热迁移、虚拟机暂停等极为普遍。如果没有 Pre-Vote,任何一台宿主机的单网卡微秒级丢包或长达数秒的 Java/Go GC STW 停顿,都会导致其醒来后以超高 Term 摧毁现有的健康 Leader,引发频繁的无意义重新选主与业务 TPS 断崖式下跌。
- 工业级生产环境的标准基线:
- etcd 与 Kubernetes:etcd 在 3.4 版本中正式将
--experimental-enable-prevote提升为稳定默认配置。Kubernetes 控制平面的稳定性因此获得了飞跃,彻底告别了因个别边缘 Worker 节点网络波动引发的 API Server 写入阻塞。 - 分布式存储与 NewSQL(TiKV / CockroachDB):在大规模 Multi-Raft 架构中,单集群可能划分出数十万个轻量级 Raft Group(Region)。如果单台物理机因网卡溢出丢包,其上成千上万个 Region 的 Follower 将集体超时并狂刷 Term;若无 Pre-Vote,这台机器网络恢复的瞬间将同时废黜全集群数万个 Leader,直接导致全分布式数据库发生链式雪崩。
- etcd 与 Kubernetes:etcd 在 3.4 版本中正式将
- 与 CheckQuorum 机制的互补闭环: Pre-Vote 解决了“外围游离节点试图扰乱健康中枢”的问题;而 Raft 的另一大扩展 CheckQuorum(领袖仲裁检查) 则解决了相反方向的难题——“被局部隔离的旧 Leader 误以为自己依然在位”(旧 Leader 若在半数租约时间内收不到多数派的心跳确认,主动退位为 Follower)。两者珠联璧合,才构筑起了分布式系统在严苛异步网络下的终极韧性。
隐喻对应表
- 绝顶玄冰关与慕容统领麾下的残卒 → 遭遇网络隔离、长 GC 或瞬态抖动的分布式节点(Partitioned / Flapping Node)
- 京畿平原与其余三关的通畅商贸与政务 → 维持多数派仲裁的健康活跃集群(Healthy Majority Quorum)
- 京畿钟鼓楼每日破晓的盘龙金钟与朱漆木札 → Leader 定期向全集群广播的保活心跳(Heartbeats / AppendEntries RPC)
- 孤峰绝壁上三十昼夜不闻钟声的沙漏古训 → 节点本地触发选举的超时计时器(Election Timeout)
- 慕容统领每逢寒冬自刻枣木牌将历数不断上翻(大寒元年→十年) → 隔离节点在无法联通集群时单调递增的虚假任期(Incrementing Term)
- 冰雪融化后手持“大寒十年”铁幡疾驰入京 → 隔离恢复后携带超高 Term 发起投票请求的扰乱节点(Disruptive Server with High Term)
- 见更高年号必须即刻交印退位的太祖至高铁律 → Raft 核心规则:收到任意 $T > currentTerm$ 必须无条件退化为 Follower(Step down on Term > currentTerm)
- 首辅退位导致全境六部封印、商贸停摆的混乱空窗 → 集群合法 Leader 被无辜废黜导致的服务阻塞与写入不可用(Leader Deposition & Write Outage)
- 开箱发现慕容统领三年卷宗全空、绝无登基可能 → 故障节点因日志过旧(Stale Log)在正式选举中无法获得多数票(Leader Completeness Rejection)
- 预询之策(遣轻骑执白羽先向外关打探京畿钟声) → 预投票阶段(Pre-Vote Phase / PreCandidate State)
- 外关守将因案头朱印崭新、晨钟未歇而断然拒关 → 追随者感知健康 Leader 活跃租约拒绝 PreVote 请求(Reject PreVote under Active Leader Lease)
- 斥候无功而返、严禁私进年号刻度 → 预投票失败遏制任期虚高,保护中枢政令平稳无虞(Suppress Term Bump & Preserve Stability)
The domain of the Great Yan Empire was guarded by five mighty mountain fortresses. In addition to the imperial citadel stationed in the central Guanzhong plains, four regional bastion cities stood sentinel to the east, south, west, and north, collectively known as the “Five Anchors of Granite.”
For more than a century since the dynasty’s founding, imperial decrees flowed without obstruction, and the realm flourished in disciplined tranquility, all owing to an ancestral statute instituted by the First Emperor: the “Law of the Golden Dawn Bell.”
At daybreak each morning, the Bell and Drum Pavilion at the imperial center struck the ten-thousand-pound Coiled Dragon Bell. Three resonant, earth-shaking chimes echoed across the clouds, and the Ministry of Timetables immediately dispatched four contingents of courier riders, carrying cinnabar-lacquered brass tubes, speeding toward the four regional bastions. Within each brass tube lay a red-clay-sealed pine tally stamped with the Prime Minister’s personal seal, inscribed with the current imperial reign era and the day’s heavenly stems and earthly branches.
According to the First Emperor’s charter, affairs of state did not require the unanimous assent of all five bastions. So long as any three garrisons received the sealed tallies on schedule each day, the central decree commanded sovereign legitimacy. Merchants passed through frontier gates, paying duties upon verified seals; granaries unlocked their reserves for relief; and merchant guilds redeemed bills of exchange. Over a hundred winters, even if a torrential downpour claimed the mounts of one courier route in the mire, the remaining three bastions and the capital remained in mutual chime, and the foundations of the realm stayed as unyielding as iron.
Only the solitary outpost perched upon the northern rim—the “Summit of Black Glaciers”—hung isolated upon a sheer cliff of ten thousand fathoms.
There, freezing gales howled year-round, and heavy frost clouds hung low. General Murong, who commanded the garrison, led only a few hundred hardened veterans. Whenever winter descended, the wooden plank roads pinned to the vertical cliffs froze under three feet of solid glaze; courier riders often braved days of lethal peril merely to ascend the peak. Yet no matter how savage the tempest, Murong climbed the beacon ramparts each dawn, craning his neck toward the southern sea of clouds, waiting for the rider from the capital.
That year, just after the frost fell, an unprecedented century-blizzard engulfed the northern wastes.
Winds whipped up snowdrifts the size of millstones, snapping the cliffside plank bridges in twain overnight. For three months the gale raged without respite; snow accumulated fathoms deep, wild fowl vanished from the skies, and carrier falcons plummeted with frozen wings into the abyss. The narrow gullet leading up to the Summit of Black Glaciers was completely transformed into a blinding, frozen desert.
Atop the isolated pinnacle, General Murong and his men sank into tomb-like silence.
Ten days passed; no courier arrived from the south. Thirty days passed; across the mist-shrouded horizon, not a single rider galloped through the snow.
By the winter solstice, Murong sat alone within his frozen stone quarters, his face grim as tempered iron. According to the ancient ancestral codex, the Canon of Celestial Tallies: “Should thirty sunrises elapse without hearing the golden bell of the capital or receiving the imperial tally, it must be presumed that catastrophe has befallen the throne and the realm hangs in mortal peril. The commanding general of the frontier shall not wait idly as heaven falls into chaos; he shall forthwith initiate the ‘Restoration of the Epoch,’ advancing the local calendar by one cycle, carving his own reign title to govern in proxy.”
Murong took a long, resolute breath. Drawing a red-hot iron chisel from the brazier, he carved forcefully upon a polished slab of jujube wood: “By Mandate of Heaven, First Year of the Great Frost.”
He affixed the wooden tablet to the fortress gate. Yet beneath the precipice, only the furious gales howled; none responded.
Another three months went by. Spring thaw failed to arrive, and ice remained locked across a thousand ravines. Murong drew his chisel once more, planed the tablet clean, and etched with gritted teeth: “Second Year of the Great Frost.”
One year, two years, three years…
Through the protracted polar night and biting frost of the solitary peak, the old general’s discipline crystallized into an unshakeable obsession. He became thoroughly convinced that the four southern fortresses had been reduced to ashes in a great civil war, and that within the four seas, only his icy summit retained loyal soldiers of the empire. With every emptying of the hourglass, with every turn of the biting winter, he stubbornly wielded his iron chisel, bumping the reign title ever higher:
“Third Year of the Great Frost,” “Fourth Year of the Great Frost”… all the way to the “Tenth Year of the Great Frost!”
Yet south of the mountains, beneath the blanket of clouds, what was the true state of the realm?
There had been no warfare, nor had the dynasty fallen. That historic blizzard had merely severed the extreme, localized high-altitude pass of the northern peak; across the central plains and the eastern, southern, and western garrisons, the seasons progressed in peaceful succession, and harvests were bountiful. The Golden Dawn Bell of the imperial citadel chimed faithfully every morning. The Prime Minister and the civil and military ministers sat within the Hall of Imperial Governance, reviewing the thousands of memorials submitted by the three active bastions. The Prime Minister’s tallies arrived on time each day; merchants thronged the high roads, and commoners lived in peace. In the eyes of all, the empire was operating smoothly in the heart of a golden age.
Until the late spring of the fourth year, when a blazing sun finally dissolved the mountain glaciers.
The ancient northern trail broke free from the ice. Clad in heavy iron armor, General Murong led his remaining hundred frostbitten warriors galloping down the thawing mountain pass. From the old general’s waist hung a heavy iron seal, and tall white banners fluttered beside their steeds, bearing four jagged characters written in thick black ink: “Tenth Year of the Great Frost!”
The battered cavalry galloped through the capital gates, riding straight up to the marble balustrades of the imperial audience hall.
The old warrior dismounted, held aloft the wooden tablet inscribed with “Tenth Year of the Great Frost,” and bellowed into the hall: “The frontier outpost has bled and stood sentinel for ten harsh years without word from the capital! Now, obeying the First Emperor’s sacred statute, we inaugurate a new era under the Tenth Year of the Great Frost. We bid the court to submit to the mandate of heaven!”
At these words, absolute silence fell upon the imperial court. The gathered ministers looked at one another in sheer bewilderment, while the Prime Minister’s face drained of all color.
Why? Because upon the first scroll of the Canon of Celestial Tallies, the First Emperor had set down an inviolable supreme statute in indelible ink:
“Throughout all bastions of the realm, whenever an envoy arrives bearing a ‘higher reign year,’ regardless of circumstance or origin, the sitting ruler and Prime Minister must immediately surrender the seal of office, step down from the dais, and suspend all state governance to await a general election of the throne. Whosoever disobeys shall be deemed an usurper against heaven, and all under the sky shall unite to strike them down.”
The original wisdom behind this statute had been noble: to prevent corrupt usurpers from clinging to the central throne, the moment any corner of the empire produced a newly verified sovereign era, the old regime was mandated to step down without resistance.
Yet at this moment, catastrophe struck.
The “Tenth Year of the Great Frost” arbitrarily towered over all active records of the reigning government. In accordance with ancestral law, the Prime Minister sighed deeply, tremblingly unfastened the golden seal from his waist, placed it upon the imperial desk, and bowed low as he stepped into the side corridor.
The administrative machinery of the empire ground to an instant halt.
The six ministries locked their official seals; merchant banks shuttered their teller windows; tens of thousands of trade caravans queuing outside the gates were stranded at the moat, their transit permits voided on the spot. All commerce and tax collection across the Great Yan Empire ceased dead in its tracks at high noon.
With the government paralyzed, a general election was convened on the spot to choose the new leader of the realm.
The council of ministers brought forth the iron chests carried by General Murong, prying open the seals in public to inspect the archives, ledgers, and deeds he had accumulated over the past ten years of frontier rule.
Yet when the lids were thrown open, the entire hall gasped in disbelief.
Inside the iron chests lay nothing but dried mountain straw and stacks of completely blank white vellum scrolls! Across those three secluded years, General Murong had collected not a single copper in revenue, sheltered not a single refugee, and remained utterly ignorant of troop movements beyond the wall. His administrative ledgers remained frozen on the very day the mountain pass had been snowed in four years prior!
The court erupted in fury. By the rules of imperial election, any general whose archives were obsolete could never obtain the confirmatory votes of the other regional commanders. Murong could not ascend the throne; he was scarcely even qualified to stand as a candidate.
Yet the Prime Minister had already abdicated, and the imperial seal was locked away. To resolve the vacuum, the four regional garrisons were forced to convene emergency conclaves to elect a new administration, wasting days in frantic message relays while merchant losses mounted into mountains of silver.
The scholars and generals of the court beat their breasts in lament: a hermit isolated upon a frozen rock, who had merely advanced his own abacus inside an ice cave, had paralyzed the entire prosperous empire through a single sheet of empty paper!
This unprovoked disaster came to be recorded in history as the “Turmoil of the Frost Ridge.”
In the aftermath, the imperial court reformed the laws, convening master scholars and grand astronomers to officially institute a lifesaving amendment into the Canon of Celestial Tallies—the “Policy of Pre-Inquiry.”
The new decree was proclaimed across the five fortresses:
Henceforth, should any isolated garrison commander find communication severed and wish to inaugurate a new reign era, he is strictly forbidden from carving higher year-tablets in isolation!
The commander must first dispatch a lone light scout bearing a white feather to conduct a “preliminary inquiry” among neighboring garrisons:
Upon arriving outside a neighboring fortress, the scout must inquire into two matters: First, “Within the past thirty sunrises, has your post heard the daily dawn bell from the imperial capital?” Second, “How do our frontier ledgers compare against the records upon your desks?”
If the garrison commander across the moat replies: “The dawn bell of the capital tolls without fail, the cinnabar seals upon our desks are fresh, and the mandate of heaven remains unbroken. How dare you stir up sedition?”—the scout must immediately withdraw his white feather, return to the peak, and keep the garrison calm, forbidden from advancing the calendar by even a single stroke!
Only when a true majority among the regional bastions responds that “the capital bell has indeed fallen silent and the center has fallen,” and only when the scout’s archives are proven to be as complete and up-to-date as their own, is the frontier commander permitted to officially carve a new reign tablet, march to the capital, and trigger an election.
From that day forth, even if the icy pinnacle were locked away by blizzards for ten years or a century, no isolated hermit playing with empty reign titles could ever again disrupt the peace of the realm beneath the clouds.
—By now you have probably recognized it: this is the essential safety extension in the Raft consensus protocol—the Pre-Vote Protocol and the defense against Disruptive Servers.
What it is
In the standard Raft consensus protocol (Diego Ongaro & John Ousterhout, 2014), Terms act as logical clocks used to detect and discard stale information.
Standard Raft enforces a foundational invariant hardcoded into every node’s event loop:
“If an RPC request or response contains term $T > currentTerm$: set $currentTerm = T$, and convert to follower.”
This rule was originally designed to ensure that whenever a new Leader is elected, any lingering zombie leader from an older term steps down immediately upon hearing from the new regime, preventing split-brain scenarios.
However, in real-world asynchronous cloud networks, this elegant mathematical rule introduces a fatal vulnerability known as the Disruptive Server Problem:
- Transient Partition and Term Inflation:
Consider a 5-node Raft cluster ${N_1, N_2, N_3, N_4, N_5}$, where $N_1$ is the healthy Leader at term 1. Node $N_5$ becomes temporarily partitioned from the other 4 nodes due to a top-of-rack switch reboot, asymmetric packet loss, or a prolonged GC pause.
$N_5$ stops receiving periodic heartbeats (
AppendEntries) from $N_1$. Its localelectionTimeoutexpires. $N_5$ increments its term from 1 to 2, transitions toCandidate, and broadcastsRequestVote. Because it is isolated, $N_5$ receives no replies. The timeout fires again, and $N_5$ increments its term to 3, 4, 5… eventually inflating to term 100 while completely cut off. - The Disruptive Rejoin:
When the network heals and $N_5$ reconnects, disaster strikes.
$N_5$ broadcasts
RequestVotecarrying its inflated term of 100. When Leader $N_1$ (still operating peacefully at term 1) receives this message, because $100 > 1$, $N_1$ must immediately step down and revert to Follower status! The cluster loses its active Leader. All ongoing client write operations stall and time out, inducing severe latency spikes and service unavailability. - The Absurd Outcome:
Once $N_1$ steps down, the cluster begins a new election. Can $N_5$ become the new Leader? Absolutely not!
In Raft, the Leader Completeness invariant dictates that voters only grant votes if the candidate’s log is at least as up-to-date as their own (
lastLogTermandlastLogIndex). Having been partitioned, $N_5$’s log is hopelessly stale. It cannot gather a majority quorum of votes. The outcome is profoundly disruptive: an obsolete node with zero chance of winning has needlessly unseated a perfectly healthy leader, forcing the cluster into an unnecessary election cycle simply because its term counter kept ticking in isolation!
To cure this vulnerability, Diego Ongaro introduced the Pre-Vote extension in Chapter 9.6 of his Stanford doctoral dissertation, Consensus: Bridging Theory and Practice:
Before entering a real election, a node enters an exploratory PreCandidate state:
- Step 1: The Pre-Vote Exploration
When a node’s
electionTimeoutfires, it does not increment itscurrentTerm. Instead, it transitions toPreCandidateand sends speculativePreVoteRPCs carrying an anticipated term of $currentTerm + 1$. - Step 2: Dual-Gate Voter Verification
A recipient node grants a
PreVoteonly if two conditions are met simultaneously:- Log Up-To-Date Criterion: The candidate’s log is at least as up-to-date as the voter’s own log;
- Active Leader Lease / Heartbeat Criterion: The voter has not received a heartbeat from a valid Leader within its minimum election timeout (
minElectionTimeout).
- Step 3: Promotion on Demand
If a healthy Leader is still sending regular heartbeats, majority followers will reject the
PreVote. $N_5$ fails to collect a majority of Pre-Votes, and itscurrentTermnever increments! It remains harmless until normal heartbeats from the Leader bring it back in sync. Only if a majority of nodes agree that the Leader has truly vanished and the candidate’s log is sufficiently fresh does the node actually increment itscurrentTerm, transition toCandidate, and initiate a genuineRequestVoteelection.
Why it matters
In modern cloud environments, networks are fraught with transient anomalies, noisy neighbors, and garbage collection hiccups:
- Preventing Single-Node Glitches from Triggering Cluster Outages: In Kubernetes, etcd, and distributed databases, container restarts, VM live migrations, or brief packet loss are routine occurrences. Without Pre-Vote, any transient hiccup on a single node causes that node to re-enter the cluster with an inflated term, immediately deposing the active Leader and triggering cascading write failures across all connected applications.
- The Standard Baseline in Production Systems:
- etcd and Kubernetes: etcd introduced Pre-Vote as an experimental flag and made it the default in version 3.4 (
--experimental-enable-prevote=true). This dramatically stabilized large-scale Kubernetes control planes by immunizing them against flaky worker or master nodes. - Distributed NewSQL (TiKV and CockroachDB): In Multi-Raft architectures managing hundreds of thousands of Raft ranges across physical machines, a single flapped network interface without Pre-Vote would trigger rolling elections across thousands of regions simultaneously, sparking a cluster-wide death spiral.
- HashiCorp Consul, Apache Ratis, and SOFAJRaft: All implement Pre-Vote as a fundamental stability prerequisite.
- etcd and Kubernetes: etcd introduced Pre-Vote as an experimental flag and made it the default in version 3.4 (
- Synergy with CheckQuorum: Pre-Vote solves the problem of “outlying partitioned nodes disrupting a healthy center”; Raft’s complementary CheckQuorum mechanism solves the inverse problem—“a partitioned leader unaware it has lost its majority” (forcing an isolated leader to step down if it cannot reach a majority within an election timeout). Together, they fortify Raft into a production-grade consensus engine resilient against asynchronous network chaos.
Metaphor mapping
- The Summit of Black Glaciers and General Murong’s isolated veterans → A partitioned, lagging, or GC-stalled node (Partitioned / Flapping Node)
- The prosperous trade across the capital and remaining regional fortresses → A healthy majority quorum maintaining consensus (Healthy Majority Quorum)
- The morning chime of the Coiled Dragon Bell and daily sealed tallies → Periodic heartbeats sent by the active Leader (Heartbeats / AppendEntries RPC)
- The thirty-day hourglass rule upon the snowy peak → The local election timeout timer (Election Timeout)
- Murong carving higher reign titles onto jujube wood (Year 1 through 10) → Monotonically incrementing the term counter in isolation (Incrementing Term)
- Galloping into the capital under the banner of “Tenth Year of the Great Frost” → Rejoining the cluster with an inflated term (Disruptive Server with High Term)
- The ancestral statute forcing the Prime Minister to surrender the seal upon seeing a higher reign year → Raft’s core invariant: step down to Follower whenever $T > currentTerm$ (Step down on Term > currentTerm)
- The paralysis of commerce and state affairs upon the Prime Minister’s abdication → Service write outage caused by premature leader deposition (Leader Deposition & Write Outage)
- The discovery that Murong’s chests contained blank ledgers, disqualifying him from ruling → An isolated node failing the Leader Completeness log check (Cannot win election due to Stale Log)
- The Policy of Pre-Inquiry (sending a scout with a white feather to check the capital’s status) → The Pre-Vote exploratory phase (Pre-Vote Phase / PreCandidate State)
- The regional commander rejecting the scout because the capital bell chimes faithfully → Followers rejecting Pre-Vote due to active Leader lease (Reject PreVote under Active Leader Lease)
- The scout returning empty-handed, forbidding any change to the reign calendar → Suppressing term inflation and safeguarding cluster stability (Suppress Term Bump & Preserve Stability)