Spring in Wangchuan begins with the night drum.

The river runs along the east wall. It has one sharp bend called Broken Neck Bay. Every spring, when the water climbs to the third mark on the city wall, the watch must do three things: strike the drum, open the south granary, and send a fast rider downstream with the flood warning for the three towns.

Every year they rehearse all three.

The rule is this: pick a clear day, put everyone at their post, strike the drum for real, open the granary for real, ride the horse for real. One day of rehearsal, one ledger written, one ledger filed in the ancestral hall.

The trouble is that on a clear day, Broken Neck Bay never bursts.

The night it did burst was the nineteenth of the third month. The drum missed one beat. The granary’s grain count was short by two measures. The rider was stopped by water at Willow Ford. By the time the warning arrived downstream, all three towns were under.

When the water fell, the city gathered in the ancestral hall and read the ledger. The ledger said: drum in order, grain count correct, rider arrived.

The truth of that night went out with the water. No one could return to the nineteenth of the third month and see which part gave way first.

That winter a young woman came from the west of the city. Her name was Yan. She did not look at the river. She looked at the drum tower.

She told the magistrate: give me an empty hall, a troupe, and a year of lamp oil I cannot exhaust, and I will give you an April.

That was the shadow city.

The shadow city was a windowless hall. Inside, they built a small Wangchuan to scale: the bend at Broken Neck Bay, the beams of the south granary, the pontoon at Willow Ford — even the drum from the tower, though only the size of a palm.

In the center of the hall sat one person holding a small drum. The city called her the keeper of hours.

The shadow city had only two rules.

The first: no sky indoors.

No looking at the real sun. No listening to the real water clock. No casting a real die. When dawn comes is set by the keeper’s drum. When the river rises is set by the channel she opens. When the rider arrives is set by the sand glass in her hand. Even which fork a watchman takes is drawn from a wax-sealed book of lots.

The second rule: every lot drawn, every channel opened, every beam that falls must be written down with that day’s lamp number.

A lamp number is a single number. The same number is the same rain, the same late rider, the same rotted plank.

— —

The first year of the shadow city fell apart badly.

Not because the play was poor, but because people kept peeking at the sky.

The old granary clerk found the sand glass slow and began counting his own heartbeat. The ferryman at Willow Ford grew impatient with the keeper’s waves and turned to look at the real river through the window. The drummer followed the real water clock, and his beats ran ahead of the book of lots. One day, by dusk, the play had broken twice — same book, same place the first two times, and the third time somewhere else entirely.

Yan called everyone to the center of the hall and said five words.

Only one hand.

From that day every hand in the hall belonged to the keeper of hours: time, water, wind, roads, and the turn of a person’s mind. If you needed time, you asked the drum. If you needed to cast a die, you took it from the book.

The play ran smoothly.

And once it ran smoothly, time inside the hall became something you could buy and sell. One quarter hour outside, three months inside. By the end, they could live thirty years of Wangchuan in a single night.

The next spring, Yan called out the lamp number: the nineteenth of the third month.

Rain began inside the hall. The drum missed one beat. The granary count came up two measures short. The pontoon at Willow Ford lifted half a foot. On the four thousand one hundred and twelfth beat, the keeper stopped her hand.

The play stopped there.

Anyone could walk into the scene. Open the granary ledger and see which column the two measures went missing from. Take apart the water clock and see which gear swallowed the beat. Send the rider back and let him choose a different ford.

This is what real events can never give you. The real nineteenth of the third month happens once. The shadow city’s nineteenth can be held in mid-air while people walk around it and look.

They found the cause. The bronze vessel of the water clock contracted in the spring cold and lost one beat a night, and the beat it lost was a flood signal. The two measures were missing because the clerk recorded what was received while the granary opened against what was due. The rider was stopped because the pontoon’s cable had soaked for three days and no one had changed it.

Fix the vessel. Change the accounting. Replace the cable.

Then call the same lamp number again.

The same rain. The same missed beat. The same late rider. But this time the granary opened against what was due, and the two extra measures were exactly enough to feed three towns until the water fell. The bridge did not lift.

The same disaster, the second time, and the city stood.

— —

In the third year, a new person joined the city.

His job was called troublemaking. He did not defend anything. He only made trouble inside the play: cut one cable, jam one granary door, make a messenger turn his ankle on the road, make a rain fall half an hour longer than forecast.

The rule was the same. Every piece of trouble had to be drawn from the book of lots, and every piece had to be written down with its lamp number.

So trouble got numbers. The city’s young people shook the lots all night, ten thousand lamp numbers in one night. Whichever number brought the city down was copied onto the wall; the next day they called that number and replayed the night, and where it broke was plain to see.

In time, the children of Wangchuan could recite the numbers on the wall the way they recited the solar terms.

— —

But Yan kept a wooden board by the door of the hall, and on it were two lines.

The shadow city is not Wangchuan.

The rain jar in the hall is round; real rain is not. The planks in the hall were measured from last year’s grain; real wood grows worms. The shadow city can tell you that within this set of rules, this is how it falls. It cannot tell you that the real sky will rain this way.

Below those lines was a smaller line.

No sky indoors.

If anyone looks up at the sky, the shadow city comes apart.

By now you’ve probably recognized it: this is Deterministic Simulation Testing.

What it is

Deterministic Simulation Testing (DST) runs an entire distributed system inside a fully reproducible simulated environment. Its most famous practitioner is FoundationDB: a whole database cluster — network, disk, clock, randomness — runs inside a single-threaded simulator, with the same interfaces serving both production and simulation.

The method is a set of interlocking rules:

• One event loop, a virtual clock. No real sleep, no real time.Now(). Time is an integer advanced by an event queue; timeouts, retries, and scheduling all execute in virtual-time order.
• Every source of nondeterminism is replaced. The network becomes a simulated network that can delay, reorder, duplicate, drop, and partition at will. The disk becomes a simulated disk that can lose writes, lie about fsync, tear a write, or crash and restart. The clock becomes a simulated clock that can jump. Threads become simulated threads. The only randomness left is a seeded PRNG.
• The seed is the reproduction. One run is a pure function of f(seed, binary, config). Keep the seed and the same failure replays in a second — no “run it again and hope it lands.”
• Fault injection. The simulator can crash any node and cut any link at any moment. FoundationDB adds BUGGIFY, which fires faults at specific code sites with a fixed probability (about one in four), only under simulation — so the same seed always fires in the same places.
• Time is compressed. Thousands of simulated nodes in one single-threaded process; an hour of wall clock can cover hundreds of thousands of cluster-hours, and seeds can be swept all night.

The critical difference from ordinary integration testing is not the simulation. It is the determinism. In a normal test, a rare timing bug lands once and you cannot catch or reproduce it; you can only pray it stays away. In DST, that bug has a seed, and the seed is its address.

Why it matters

Most distributed-systems bugs live in the seams of timing: a message arriving three milliseconds late, an fsync silently swallowed, two nodes both convinced they are leader at the same instant. Such bugs have vanishing probability in small tests, and when they strike production they often leave nothing in the logs.

DST turns rare timing into a named input. Failures need not be waited for; you specify them. And after you find one you need not “work out how to reproduce it” — you write down the seed. This moves distributed testing from a game of probability to a game of enumeration, and an enumeration you can pause, rewind, and inspect frame by frame.

It changes more than testing. It changes the engineering loop. FoundationDB wired simulation into day-to-day development: a change comes in, the seed swarm runs, and a failure hands you a seed directly. TigerBeetle built the same idea into VOPR (Viewstamped Operation Replicator), running real consensus and storage-engine code against a controllable world. Antithesis pushed determinism down into a custom hypervisor, The Determinator, so an entire machine’s virtual hardware, network, and time are deterministic and the system under test barely changes. In the Rust ecosystem, madsim packages the same approach as a library.

It also draws its own boundary, and the boundary is hard:

• Fidelity. A simulator is a model, not the world. Whatever it does not model — real NIC queueing, kernel page-cache semantics, GC pauses, real hardware failure rates — it cannot test. Model it wrong and the test passes quietly, then production falls over.
• Discipline. One hidden read of the real clock, one background thread, one third-party library not wired into the simulator, and determinism is broken; replay forks. This is not a rule you can mostly follow. It is all or nothing.
• Not a performance test. A virtual clock measures logic, not latency. Throughput and tail latency still belong on real hardware.

So DST does not replace integration tests, chaos engineering, canaries, or formal verification. It replaces the practice of running integration tests over and over in the hope of hitting a timing bug — and it lifts the hardest class of bug, rare, fatal, and irreproducible, to the place where a unit test already lives: named, and replayable.

Metaphor mapping

• Wangchuan → the distributed system under test
• The shadow city → the simulator
• The keeper of hours → the single event loop and virtual clock
• No sky indoors → no real wall clock, network, disk, or entropy source
• Lots in the sealed book → the seeded PRNG; every nondeterministic choice is drawn from it
• The lamp number → the random seed; the same number is the same reproducible run
• Stopping on the 4,112th beat → pausing at an arbitrary virtual instant to inspect global state
• The missed drumbeat, the two missing measures, the lifted pontoon → injected faults and the real bugs they expose
• Walking into the scene to read the granary and open the clock → frame-by-frame debugging through deterministic replay
• Fixing the vessel, the accounting, the cable → changing the code or configuration under test
• The same lamp number, the second time, and the city stands → the fix verified against the very same scenario, now a regression case
• The troublemaker → the fault injector, including BUGGIFY-style site injection
• Ten thousand lots in one night → swarm testing, thousands of simulated runs
• Thirty years in one night → time dilation; virtual time far outruns wall clock
• The shadow city is not Wangchuan → the fidelity limit: behavior the model omits cannot be tested
• Looking up at the sky tears the shadow city apart → one uncontrolled source of nondeterminism forks the replay
• The numbers on the wall, recited like solar terms → failing seeds settling into a reproducible regression set