▶ Cinematic fable · Watch on YouTube ▶ 影片版寓言 · 在 YouTube 观看
柳溪镇有一座戏台。
戏台是老柯祖上传下来的。四面墙,一道顶,台板是楠木的,踩上去不响。老柯不唱戏,他租台。哪家班子来镇上演出,就把台租下来,自己搭布景、挂灯、调弦、试身段。天不亮进园子,等第一声锣。
搭台要四个时辰。
镇上逢五逢十有集。集上人多,人来得急,走得也急。辰时末,四乡的人挤满长街;到了巳时初,散得只剩几个挑担的。这四个时辰的差,正好吃掉最热闹的那半个上午。
老柯收租不贵。可来过的班子回去都说:柳溪的台好,就是等不起。
第二年,镇东的老郑起了个园子,台比老柯的矮,用松木,一面墙是篱笆。他那边搭台两个时辰。人都去了他那边。那年冬天空着台,老柯坐在台板上听风穿过去。
女儿阿卷说:爹,搭得快不一定赢。他们是在比谁先开锣。
老柯说:那我把台做成死的——布景不拆,灯不落,琴不松,班子进门直接唱。
阿卷说:那不一样。班子不同,戏不同。
老柯还是去试了。
— —
他挑了个班子,叫小满班。四个人,唱的是一出《月下打更》。老柯让他们在园子里住了一个月,天天练。练到后来,四个人上台不用看人,抬手就是那个位置;琴师的手一直悬在弦上;连灯芯都修到了那个高度——风一吹,火苗不晃。
那天午后,四个人在台上唱到一半,老柯把园门关了。
他从怀里掏出一沓纸,一支笔。
阿卷问:你干什么?
老柯说:抄。
他绕着台走。第一排第三块台板,缝里嵌着半粒谷子;左边第二根柱子的绳结是双套,因为那根柱子比别的歪半寸;第三盏灯,灯芯剪的是斜口,火苗朝东南偏;琴师右手小指压在第四弦上,弦还震着;打更的人左脚在台沿,右脚悬空,鼓槌停在离鼓面一寸的地方。
他从晌午抄到掌灯。抄完,卷起来,塞进一个竹筒。
第二天来了一班外地人,问:台要搭多久?
老柯把竹筒递了过去。
— —
头一回成了。
外地班子照着纸上念,一节一节把台摆出来:这块板,这个绳结,这盏斜口灯。最后琴师把指头按上第四弦,打更的抬起鼓槌。
台上那四个人忽然就”在”了。
不是慢慢热起来的。是一瞬间,那口气接上了。
老柯站在台下,看见看客愣了一下——原本要等的那半个上午,没有了。戏是从中间开始的,可他们不知道中间在哪儿,因为从来没有人在这座台子上这么唱过。
那一夜,全镇都听说柳溪的台开锣最快。第二天来了三班。
第三班就不行了。
他要唱的不是《月下打更》,是《白蛇》。第一句就不是那么起的。阿卷照着抄本摆,摆到一半,台上的人说:不对,这里我要转身。
阿卷说:纸上没写。
台就散在那儿了。
— —
老柯又坐了一个冬天。
第二年春天,他做了第二件事。他不再抄一个班子,他抄整座园子。
不只是一出戏。台板、柱子、绳、灯、瓦、地砖、后廊的水缸、门轴的松紧,一样不落。他抄了整整七天,二十七卷。
来一班人,就把二十七卷全摊在台上,让他们照着摆。
结果是:摆得出来,但摆完天黑了。
一卷一卷摊开,跟从头搭一遍差不了多少。
阿卷那天说了一句:爹,你不是抄得太多,你是摊得太早。
老柯问:什么意思?
阿卷说:谁一进门就把整座园子摊开?他先踩台板,再抬头看柱子,然后再看灯。你等他叫,他要哪一卷,你再摊哪一卷。
— —
老柯就把二十七卷全挂在门口,一卷一格,绳头垂下来。谁要哪一格,门口的小六就抽哪一格,摊到台上。
这一改,快得吓人。第一班人进门,小六抽了三卷,锣就响了。
多了个抽卷的人,反倒少了四个时辰。
因为头三卷正是每班人第一步要看的:台板、柱子、灯。剩下的二十四卷里,有一半一年也用不上一次。老柯把常用的那几卷挂在最外头,剩下的往后排。
到那年秋天,柳溪的台已经快到”客人进门,茶还没凉”。老柯觉得这事算完了。
然后到了十月初五,镇上一百二十年一次的大集。
那天来的不是三班,是一百二十班。
— —
小六站在门口。头一刻钟,他抽得比平常快三倍。
第二刻钟,门口堵住了。
因为一百二十班唱的是一百二十出戏。头三卷抽完,后面各不相同,每一卷在屋里只有一个位置——谁抽走了,别人就抽不到。小六开始跑,跑到后来,胳膊抬不起来。
那天下午有一半的班子没开成锣。老柯站在廊下,看见门口的队比他当年搭台时还长。
阿卷蹲在旁边说:我们没比原来快。我们只是把”等”从台上搬到了门口。
老柯没接话。他那天夜里发现了两件更坏的事。
第一件,账错了。
两班人拿着同一张收条来领钱。收条上的印是”柳溪园”,底下一个号,两班人的号一模一样。
老柯翻出抄本才明白:他连门房的印、账房的沙漏一起抄了进去。他给每班人的,是一座”已经盖过印、已经走过半格”的园子。两班人进门时,抄本上都是同一时刻。
第二件,台塌了一角。
十六班的人共用一卷台板图。第十三班嫌第三块板矮,垫了一片瓦。瓦留在了板上——第十六班踩上去,人歪了半寸,从台沿摔了下来。
阿卷说:会改的东西,不能共着看。
— —
那年冬天,老柯做了四件事。
第一,他把园子裁了。原来的园子有正门、偏门、后门、马道、水房、账房、门房、柴房、耳房,四十七处出入口。他拆掉四十四处,只留三个洞:一个送信,一个送饭,一个传话。园子小了,人进去就快;更要紧的是,外人想钻进来,也只能从这三个洞钻。
第二,门口永远摊着三卷。不等人叫,先摊。那三卷是每班进门头一步都要看的。
第三,抄本第一页写了一行字,进门先看:抄本进门,先换新印,先翻新沙漏。
第四,另起一行:只读的卷子,一百班共着看;动过笔的卷子,一班一份。
第二年,柳溪的台能在一刻钟里开一百班。同一份抄本,同一座园子,谁也不占谁的。
— —
后来有人问阿卷:那个竹筒里到底装了什么?
阿卷说:装的不是戏。装的是一座”已经热起来的”园子。
园子搭起来不难,难的是热。台板踩顺了,绳结受了力,灯芯修到了那个高度,琴弦刚校过还在震,四个人的气已经连成一条。这些东西不是造出来的,是养出来的。养要时间。
所以老柯抄的不是一个空园子。他抄的是那个”养到了”的瞬间。
到这里你可能已经认出来了:这就是 MicroVM Snapshot-Restore(微虚机快照恢复)。
这是什么
它是 serverless 平台把 cold start 从几百毫秒压到个位数毫秒的核心手法。三个东西叠起来:
- microVM。一台”只留必要器官”的虚拟机。设备模型砍到只剩 virtio-net、virtio-block、vsock 和串口;VMM 本体在几万行量级(Firecracker 约五万行,QEMU 是百万行)。每个租户一台 KVM 虚拟机,共享的只有硬件虚拟化层,不是内核本身——这是它和容器最本质的差别。外面再套 jailer:chroot、seccomp、cgroup、网络命名空间,把”三个洞”钉死。
- snapshot。把一台已经在跑、已经热了的 microVM 的完整状态写成文件:全部内存页、CPU 寄存器、设备状态。关键不在”能不能存”,在在哪个瞬间冻结——语言运行时初始化完、JIT 编译完、连接池建好、依赖都在内存里的那一刻。冻的不是”一台空机器”,是”一台刚好能立刻干活的机器”。
- restore。恢复 = 把内存文件 mmap 进来,把寄存器和设备状态写回去,继续执行。不重放启动过程,所以不付启动的钱。
真正让它从”能用”变成”便宜”的是后面两招:
- 按需缺页(userfaultfd)。恢复时不读整个内存文件。内存区域挂到 userfaultfd 上,guest 第一次碰到某个还没填的页时陷入 host,host 才去快照文件里读那一页。所以恢复延迟正比于”实例立刻用到多少内存”,而不正比于”快照有多大”。
- 共享只读页(page cache 复用)。同一个快照恢复一千次,那一千台 microVM 读的是同一份 host page cache。没被写过的页在物理内存里只有一份。这就是为什么”一千个并发的相同函数”不需要一千份内存。谁写了,谁触发 copy-on-write,拿自己的一页。
配套还有两条容易忘、但必须做的规矩:
- 熵与时钟。恢复出来的机器,RNG 状态和原机一模一样。不再播种,两个克隆就会生成同一批随机数、同一批 UUID、同一批密钥。内核时钟也得重对齐,否则新实例一睁眼就以为自己活在快照那一刻。所以恢复后要 reseed 熵池、重同步时间。
- 状态是有毒的。快照里的 TCP 连接、TLS 会话、DNS 缓存、token、文件锁,全都指向一个”当时还在、现在可能已经不在”的对端。所以能快照的不是任意程序,而是能把外部状态重建的程序:恢复时重连、重解析、重新取凭证。这条约束决定了 serverless 函数为什么必须无状态。
区别要说清:cold boot 是”从零走一遍启动”;warm pool 是”提前启动若干台,等人来领”,拿钱买时间;fork 是”从一个进程复制子进程”,共享的是内核对象,隔离强度不同;live migration pre-copy 是”把一台正在跑的搬走”,一换一;而 snapshot-restore 是”把一台冻住,再复制很多台”,一变多。
为什么重要
它的重要性是一条经济学不等式。
一台通用虚拟机跑一个函数,启动要几百毫秒到几秒,空转还占着内存和 CPU 时间片。一台物理机能塞的通用 VM 是几十台量级。同样的主机改用 microVM 能塞几千台——每台 VMM 只有几 MB,设备模型几乎没有,不跑的时候还能换出去。这个密度差,就是 serverless 能按”每毫秒”计费的前提。
它同时解了一个更难的矛盾:隔离强度与密度。容器共享内核,一个内核漏洞就是所有租户的漏洞,多租户跑不可信代码时它不够;虚拟机隔离强,但启动贵、占内存,密度不够。microVM 的做法是承认”隔离的成本主要来自设备模型和启动路径”,然后只砍这两样,把硬件虚拟化的边界留着。快照恢复则把”启动”这一项从账单上抹掉。
它的失败模式也很典型,几乎全部来自”快照里带着过去”:
- 缺页风暴。恢复把延迟摊到了第一次访问上。一千个实例同时恢复、同时访问不同页,host 的缺页处理先饱和,恢复延迟反而比冷启动更差。所以要预热工作集、限制并发恢复数、保证热集能装进 page cache。
- 快照膨胀。内存快照等于内存大小。依赖越多、运行时越重,快照越大,恢复时缺页越多。这反过来推动运行时变轻——serverless 偏爱小运行时不是审美。
- 陈旧状态。过期的 DNS、失效的凭证、指向已死对端的连接、被别处改过的配置。恢复出来的实例”记得”一个已经不存在的世界。
- 同源克隆。忘了重新播种熵,一百个实例生成同一个随机数。这在密钥、幂等键、采样决策上是灾难。
- 版本漂移。快照绑定了内核、VMM、宿主机的某些特性。平台升级后旧快照不能恢复,或者恢复出诡异行为。快照库得跟着版本一起管。
所以它的工程重心不在”怎么快照”,而在怎么定义一台”可快照的”机器:工作集要小,外部状态要能重建,身份要在恢复后重新生成,只读的东西要真的只读。把这些定下来,快照恢复才是能长期跑在生产上的机制,而不是一个漂亮的 demo。
隐喻对应表
- 柳溪镇的戏台与园子 → 一台 microVM;楠木台板、柱子、灯 → 完整的运行时与内存状态
- 搭台四个时辰 → cold boot:走一遍内核、init、运行时初始化
- 集日人来得急走得也急 → serverless 的请求潮;等待就是丢掉的生意
- 老郑的矮台、篱笆墙、两个时辰 → 更轻的方案:更快的启动,换来更弱的保障
- 小满班练了一个月、气连成一条 → 运行时被”养热”:JIT 编译完、连接池建好、依赖已驻留
- 唱到一半关门 → 在热的、静止的瞬间冻结
- 一沓纸、一支笔,绕着台抄 → snapshot:内存页 + 寄存器 + 设备状态
- 竹筒里不是戏,是”已经热起来的”园子 → 快照存的是”刚好能立刻干活”的状态,不是程序本身
- 照着抄本摆,一瞬间”在”了 → restore:映射内存、写回寄存器、继续执行
- 缺了”我要转身”那一句 → 快照只覆盖已走过的路径;新请求需要没被冻结过的分支
- 二十七卷全摊开,摆完天黑了 → 恢复时把整个快照读进来,代价等同重建
- 卷子挂在门口,要哪格抽哪格 → userfaultfd 按需缺页
- 小六一个人抽卷 → host 上的缺页处理线程
- 头三卷先摊着 → 预热工作集 / prefetch
- 常用卷挂最外头 → 热页排布与预取顺序
- 一百班共读同一卷,谁都不许在上面写 → 只读页在 page cache 里共享
- 第十三班垫了一片瓦 → 写操作;污染共享页
- 动过笔的卷子一班一份 → copy-on-write
- 一百二十班堵在门口,一条卷只有一个位置 → 缺页风暴;恢复延迟摊到首次访问上
- 门口排的队比搭台还长 → 恢复并发过高,比冷启动更慢
- 两班人拿着同一个号的收条 → 忘记 reseed 熵:克隆出同一批随机数与密钥
- 抄本上”已经走过半格”的沙漏 → 陈旧时钟;恢复后必须重同步时间
- 四十七处出入口拆到只剩三个洞 → 最小设备模型 + jailer(chroot、seccomp、cgroup、网络命名空间)
- 园子小了,人进去也就快了 → 攻击面和启动路径同时变短
- 门口永远摊着三卷 → 用小工作集换低缺页
- 编排太重的班子摆不平 → 快照膨胀:运行时越重,恢复越贵
- 一换一 vs 一变多 → live migration pre-copy 与 snapshot-restore 的目的不同
Willow Creek had one theatre.
It came down to Old Ke from his grandfather. Four walls, one roof, nanmu floorboards that did not creak. Old Ke did not sing. He rented the stage. Whatever troupe came through town rented it, built its own set, hung its lamps, tuned its strings, and tried its postures — into the yard before dawn, waiting for the first gong.
Building the stage took four double-hours.
The town held a market on every fifth and tenth day. The market was crowded, and people came in a hurry and left in a hurry. By the end of the morning watch, four villages’ worth of people packed the long street; by the beginning of the next watch, only a few pole-carriers were left. Those four double-hours ate the busiest half of the morning exactly.
Old Ke’s rent was not high. But every troupe that came went home saying: Willow Creek’s stage is fine, you just can’t wait for it.
The next year Old Zheng opened a yard east of town. His stage was lower, made of pine, one wall no more than a fence. Building there took two double-hours. Everyone went to him. That winter Old Ke’s stage stood empty, and he sat on the boards listening to the wind cross them.
His daughter Ah Juan said: Father, being quick to build is not how you win. They are competing over who rings the gong first.
Old Ke said: Then I’ll make the stage dead — set left standing, lamps left burning, strings left taut. A troupe walks in and sings.
Ah Juan said: That doesn’t work. Different troupes, different plays.
Old Ke went and tried anyway.
— —
He picked a troupe called Little Full. Four people, one play: The Night Watch Under the Moon. He put them up in the yard for a month and had them rehearse daily. In the end the four of them could take the stage without looking at each other, raise a hand and find the position already there. The lute player’s hand never left the string. Even the lamp wicks had been trimmed to that exact height — a gust came, and the flame did not waver.
One afternoon, with the four of them halfway through the play, Old Ke shut the yard gate.
He took a stack of paper and a brush out of his coat.
Ah Juan asked: What are you doing?
Old Ke said: Copying.
He walked the stage. Third board in the front row, half a grain of rice caught in the seam. The rope knot on the second pillar on the left was a double hitch, because that pillar leaned half an inch. The third lamp had its wick cut at a slant, so the flame leaned southeast. The lute player’s little finger rested on the fourth string, and the string was still vibrating. The watchman had his left foot on the stage lip, his right foot in the air, the drumstick stopped an inch above the drumhead.
He copied from noon until lamps were lit. Then he rolled it up and put it in a bamboo tube.
The next day a troupe from out of town asked: How long to build the stage?
Old Ke handed them the tube.
— —
The first time, it worked.
The outsiders read off the paper and set the stage piece by piece: this board, this knot, this slanted wick. At the end the lute player put his finger on the fourth string, and the watchman lifted the stick.
And the four of them were suddenly there.
Not warming up. In one breath, the thread was joined.
Standing below, Old Ke watched the audience blink — because the half morning they were supposed to wait for was gone. The play began in the middle, and they could not tell where the middle was, because no one had ever sung on that stage that way.
That night the whole town heard that Willow Creek rang its gong fastest. Three troupes came the next day.
The third one could not do it.
It was not The Night Watch they wanted to sing. It was The White Snake. The first line did not start that way. Ah Juan set the stage from the copy, and halfway through, the singer said: No — here I turn.
Ah Juan said: The paper doesn’t say.
And the stage came apart where it stood.
— —
Old Ke sat through another winter.
In the spring he did a second thing. He stopped copying one troupe and copied the whole yard.
Not one play. The boards, the pillars, the ropes, the lamps, the tiles, the paving, the water jar in the back corridor, the tightness of the door hinge — everything. He copied for seven days. Twenty-seven scrolls.
A troupe would arrive, and he would lay all twenty-seven out on the stage for them to follow.
The result: they could build it, but by the time they finished it was dark.
Laying out scroll after scroll cost about what building from scratch cost.
That day Ah Juan said: Father, you didn’t copy too much. You unrolled too early.
Old Ke asked: What does that mean?
Ah Juan said: Who unrolls a whole yard at the door? First he steps on the boards. Then he looks up at the pillars. Then he looks at the lamps. Wait until he asks. He asks for a scroll, you unroll that scroll.
— —
So Old Ke hung all twenty-seven at the doorway, one to a slot, the cord ends dangling. Whichever slot was called for, Little Six at the door pulled that scroll and laid it on the stage.
The change was frighteningly fast. The first troupe walked in, Little Six pulled three scrolls, and the gong rang.
One extra boy pulling scrolls, and four double-hours gone.
Because those first three scrolls were exactly what every troupe looked at first: boards, pillars, lamps. Of the other twenty-four, half went unused in a year. Old Ke moved the frequently used ones to the outer row and put the rest behind.
By autumn, Willow Creek could nearly promise “tea still warm when the guest arrives.” Old Ke thought the matter was finished.
Then came the fifth of the tenth month, and the great market that comes once in a hundred and twenty years.
That day it was not three troupes. It was a hundred and twenty.
— —
Little Six stood at the door. For the first quarter hour he pulled three times faster than usual.
In the second quarter hour, the doorway jammed.
Because a hundred and twenty troupes wanted a hundred and twenty different plays. The first three scrolls were pulled; after that, nothing matched. And each scroll had only one place in the house — whoever pulled it out, the next troupe could not have it. Little Six started running. By the end his arms would not come up.
Half the troupes never rang a gong that afternoon. Old Ke stood under the eaves and saw a line at his door longer than the one he used to have at the stage.
Ah Juan crouched beside him and said: We aren’t faster than before. We only moved the waiting from the stage to the door.
Old Ke said nothing. That night he found two worse things.
The first: the accounts were wrong.
Two troupes came to collect with the same receipt. The seal read “Willow Creek Yard,” and beneath it a number. The two numbers were identical.
Old Ke dug out the copy and understood. He had copied the gatekeeper’s seal and the accountant’s hourglass along with everything else. What he handed each troupe was a yard that had already been sealed, with half the sand already run. Both troupes walked in at the same moment on the paper.
The second: a corner of the stage collapsed.
Sixteen troupes shared the same floorboard scroll. The thirteenth thought the third board sat low and wedged a roof tile under it. The tile stayed. The sixteenth stepped on it, came down half an inch off, and fell from the stage lip.
Ah Juan said: Anything that gets changed can’t be shared.
— —
That winter Old Ke did four things.
First, he cut the yard down. The old yard had a front gate, a side gate, a back gate, a horse run, a water room, an accounts room, a gatehouse, a wood store, an ear room — forty-seven ways in and out. He tore out forty-four. Three holes remained: one for letters, one for food, one for word of mouth. A smaller yard is quicker to enter. More to the point, anyone trying to get in from outside now had only those three holes to try.
Second, three scrolls always lay open at the door. Nobody had to ask. Those three were what every troupe looked at first.
Third, on the first page of every copy he wrote a line, to be read before entering: A copy that comes in takes a new seal first, and turns a new hourglass first.
Fourth, a line beneath it: A scroll that is only read, a hundred troupes share. A scroll that has been written on, one troupe, one copy.
The next year, Willow Creek could open a hundred stages in a quarter hour. One set of copies, one yard, and no one took anyone else’s.
— —
Later someone asked Ah Juan what had actually been in that bamboo tube.
Ah Juan said: Not a play. A yard that had already grown warm.
Building a yard is not hard. Warm is hard. The boards have been walked smooth, the knots have taken their load, the wicks are trimmed to that height, the strings were tuned a moment ago and are still ringing, and four people’s breath has joined into one line. None of that is built. It is raised, and raising takes time.
So what Old Ke copied was not an empty yard. He copied the instant it had finished being raised.
By now you’ve probably recognized it: this is MicroVM Snapshot-Restore.
What it is
It is the core technique serverless platforms use to push cold start from hundreds of milliseconds down to single-digit milliseconds. Three things stacked:
- A microVM. A virtual machine with only the organs it needs. The device model is cut back to virtio-net, virtio-block, vsock, and a serial port; the VMM itself sits in the tens of thousands of lines (Firecracker about fifty thousand; QEMU is over a million). Each tenant gets its own KVM virtual machine, and what is shared is the hardware virtualization layer, not the kernel — that is the essential difference from a container. Around it sits a jailer: chroot, seccomp, cgroup, network namespace, nailing down the three holes.
- A snapshot. The complete state of a microVM that is already running and already warm, written to a file: every memory page, the CPU registers, the device state. The hard part is not whether you can save it. The hard part is which instant you freeze — the moment after the language runtime has initialized, the JIT has compiled, the connection pool has been built, the dependencies are resident in memory. You are not freezing an empty machine. You are freezing a machine that is exactly ready to work.
- A restore. Restore is mmap the memory file, write back the registers and device state, and continue executing. The boot path is not replayed, so the cost of booting is not paid.
What turns this from workable into cheap is the next two moves:
- Demand paging (userfaultfd). A restore does not read the whole memory file. The memory region is registered with userfaultfd; the first time the guest touches a page that has not been filled, it traps to the host, and only then does the host read that page from the snapshot file. So restore latency scales with how much memory the instance touches immediately, not with how large the snapshot is.
- Shared read-only pages (page cache reuse). Restore the same snapshot a thousand times and all thousand microVMs read one host page cache. A page nobody has written exists once in physical memory. That is why a thousand concurrent copies of the same function do not need a thousand copies of its memory. Whoever writes takes a copy-on-write and gets a page of their own.
Two rules come with this, easy to forget and mandatory:
- Entropy and clock. A restored machine has exactly the RNG state of the original. Without reseeding, two clones generate the same random numbers, the same UUIDs, the same keys. The kernel clock has to be realigned too, or the new instance opens its eyes believing it lives at the instant of the snapshot. So a restore reseeds the entropy pool and resynchronizes time.
- State is toxic. The TCP connections, TLS sessions, DNS cache, tokens, and file locks inside a snapshot all point at a peer that existed then and may not exist now. So the thing you can snapshot is not an arbitrary program but a program that can rebuild its external state: reconnect, re-resolve, re-fetch credentials on restore. This constraint is why serverless functions have to be stateless.
The distinctions matter. Cold boot walks the startup path from zero. A warm pool starts instances ahead of time and lets someone claim one — money buying time. Fork copies a process into a child, sharing kernel objects, with a different isolation strength. Live migration pre-copy moves one running machine somewhere else, one for one. Snapshot-restore freezes one machine and copies it into many: one becomes many.
Why it matters
Its importance is an economic inequality.
A general-purpose VM running one function takes hundreds of milliseconds to seconds to start, and while idle still holds memory and CPU shares. A physical host fits tens of general-purpose VMs. The same host fits thousands of microVMs — each VMM is a few megabytes, the device model is nearly nothing, and an idle one can be swapped out. That density gap is the precondition for billing serverless by the millisecond.
It also resolves a harder tension: isolation strength versus density. Containers share a kernel, so one kernel bug is every tenant’s bug; for multi-tenant untrusted code that is not enough. Virtual machines isolate properly but start expensively and occupy memory, so density suffers. The microVM approach admits that the cost of isolation comes mostly from the device model and the boot path, cuts only those two, and keeps the hardware virtualization boundary. Snapshot-restore then erases boot from the bill entirely.
Its failure modes are equally characteristic, and nearly all of them come from the snapshot carrying the past:
- Page fault storms. Restore defers its latency to the first access. A thousand instances restoring at once, each touching different pages, saturates the host’s fault handling, and restore latency ends up worse than cold boot. Hence: warm the working set, cap concurrent restores, make sure the hot set fits in page cache.
- Snapshot bloat. A memory snapshot is the size of memory. More dependencies and a heavier runtime mean a bigger snapshot and more faults on restore. This is what pushes runtimes to get lighter — serverless’s preference for small runtimes is not aesthetics.
- Stale state. Expired DNS, revoked credentials, connections to a dead peer, config changed elsewhere. The restored instance remembers a world that no longer exists.
- Identical clones. Forget to reseed entropy and a hundred instances produce the same random number. In keys, idempotency tokens, and sampling decisions, that is a disaster.
- Version drift. A snapshot is bound to particular kernel, VMM, and host features. After a platform upgrade, old snapshots either will not restore or restore into strange behaviour. The snapshot library has to be managed alongside platform versions.
So the engineering weight is not in “how do we snapshot.” It is in defining what counts as a snapshottable machine: small working set, rebuildable external state, identity regenerated after restore, and read-only things that are genuinely read-only. Settle those and snapshot-restore becomes a mechanism you can run in production for years, not a handsome demo.
Metaphor mapping
- The Willow Creek stage and yard → one microVM; the nanmu boards, pillars, lamps → the full runtime and memory state
- Four double-hours to build → cold boot: kernel, init, runtime initialization
- Market crowds that come and go fast → a serverless request surge; waiting is lost business
- Old Zheng’s low pine stage with a fence wall → a lighter approach: faster to start, weaker guarantees
- Little Full rehearsing a month until their breath joins → the runtime grown warm: JIT compiled, pool connected, dependencies resident
- Shutting the gate mid-play → freezing at a warm, quiescent instant
- Paper and brush, walking the stage → snapshot: memory pages + registers + device state
- The tube holds not a play but a yard already grown warm → the snapshot stores a ready-to-work state, not the program
- Setting the stage from the copy, suddenly there → restore: map memory, write registers, continue
- The missing line “here I turn” → the snapshot only covers paths already walked; new requests need branches that were never frozen
- Unrolling all twenty-seven, finishing after dark → reading the whole snapshot at restore, costing the same as rebuilding
- Scrolls hung at the door, pulled only when called for → userfaultfd demand paging
- Little Six pulling scrolls alone → the host’s fault-handling thread
- Three scrolls always lying open → warming the working set / prefetch
- Frequent scrolls moved to the outer row → hot-page placement and prefetch order
- A hundred troupes reading one scroll, none allowed to write on it → read-only pages shared through page cache
- The thirteenth troupe wedging in a roof tile → a write; poisoning a shared page
- A scroll that has been written on gets one copy per troupe → copy-on-write
- A hundred and twenty troupes jammed at the door, one slot per scroll → a page fault storm; restore latency deferred to first touch
- The door line longer than the old build queue → restore concurrency too high, worse than cold boot
- Two troupes holding receipts with the same number → forgetting to reseed entropy; identical random numbers and keys
- The hourglass on the copy already half run → a stale clock; time must be resynchronized after restore
- Forty-seven doorways cut down to three holes → a minimal device model plus the jailer (chroot, seccomp, cgroup, network namespace)
- A smaller yard is quicker to enter → a shorter attack surface and a shorter boot path at once
- Three scrolls always open at the door → trading a small working set for few faults
- A troupe too heavy to set up → snapshot bloat: the heavier the runtime, the more expensive the restore
- One for one versus one becoming many → live migration pre-copy and snapshot-restore aim at different things