In the hollow of Nine Turns there was one water mill.

It sat at the tail of the channel. The water came down off the back hill, wound through nine bends, and arrived at the mill with most of its temper already spent. One stone, sixty sacks a day, no faster and no slower.

Eighty-four households lived in the hollow. After the autumn harvest, each brought one sack, queued, waited for the flour, and carried it home. The longest line anyone remembered reached the great locust tree at the village mouth and waited half a day. Old Mu had turned that stone for forty years. He kept a balance scale and took one scoop of grain per sack as the milling fee.

He used to tell his grandson, Ah Cheng: the mill’s temper is fixed. People’s tempers are not.

On the morning of the fifth of the seventh month, a stone caught in the channel.

It was not large, the size of a palm, a corner that had come down off the back hill. The water could not get through, the stone turned at half speed, and by noon only twenty sacks had been ground.

So the line grew. It turned the bend in the road, and you could not see the end of it.

At the tail of the line stood the Lu family’s wife. She had brought one sack, her grain for the season. She looked up at that line turning the bend, set her sack on the ground, and called back toward her house: bring the two from the west room as well.

Her husband said: those two are for winter.

She said: if we don’t grind now, will there still be a place in line by winter?

So she carried three sacks back.

The Wang family’s grandmother, ahead of her, saw three sacks and sent her son home for more. Old Zhou, ahead of her, sent his grandson.

By noon the line had not gained a single person, and the grain in it had tripled.

Everyone had added two sacks. Everyone believed they were the only one who had thought of it.

— —

That night someone groped in the dark and pulled the stone out.

The next morning the water ran as before, the stone turned as before, sixty sacks a day.

The line was still three li long.

Old Mu stood at the mill door a long while, then went in and opened the ledger. Eighty-four households, not one more. Nobody had moved into the hollow. Nobody had rented new ground. The grain was the same grain. What had changed was how many sacks each household carried.

By autumn it was the only thing he could see clearly: a long line frightens people, frightened people carry more, and more grain makes a longer line. The circle needed no one from outside to push it. It turned by itself.

His first instinct was to add a second millstone.

Two stones ground a hundred and twenty sacks a day. The line emptied in two days. Old Mu breathed out.

By the fifth day the line had grown back around the bend. Because now that people saw the mill running fast, they dared to add two more sacks. The cake had grown, and so had the appetite.

His second attempt was to grind day and night.

Three days and three nights. On the fourth day he ground a sack into half flour — too coarse for the sieve, carried back to be ground again. Sacks waiting to be reground piled up inside the mill, and those sacks stood in the line too, and took a share of the work.

The more he ground, the more there was to grind.

On the seventh day Old Mu sat on the threshold and said to Ah Cheng: the stone is long gone, the water is enough, the millstone is sound. Why won’t this line die?

Ah Cheng had spent those days at the door with charcoal on a board. He drew three lines: how many people came each day, how many sacks each carried, how long the line was.

When the three moved together he saw something strange. The count of people was a flat line, eighty-four, never once moving. But the length of the line and the sacks per household moved like two legs holding each other up — one stepped, the other stepped, and neither could stop.

He told his grandfather: the mill is not sick. The mill is standing in a bad place. That place is steady. Steady enough to stand in for a lifetime.

Old Mu understood half of it. He asked: then how do we push it out?

Ah Cheng said: not with the mill. With the sacks.

— —

The next day Old Mu raised a new board at the mill door, three lines on it.

One sack per household per day. More than that is not accepted.

Anyone who sends a runner to ask his place in the line goes to the back today.

If the yard holds more than one day’s work, the gate closes and opens again tomorrow.

The village cursed him for three days.

On the fourth day the cursing quieted, because the line had retreated from the bend to the great locust tree.

On the seventh day it had retreated to the mill wall.

On the ninth day no one was queuing at all. The same old stone, sixty sacks a day, not one drop more water in the channel.

— —

Later the village asked Ah Cheng what the stone had actually done.

Ah Cheng said: the stone did nothing. The stone only pushed the line, in a single afternoon, past the third bend.

The third bend was the real threshold. Nine Turns queued in ordinary times too. It queued past the first bend, past the second, and people’s hearts stayed level. But once the line passed the third bend, people began to think about winter. Thinking about winter, they added two sacks. Adding two sacks, the line stayed outside the third bend forever.

One road, one bend. Walk this side of it and you are queuing. Walk the other side and there is no way back.

So the thing to guard against was never the stone. There is a stone every year; something always comes down into the channel. The thing to guard against is whether the line gets a chance to pass the third bend. Once it has, you can pull out the stone, add a millstone, grind three nights through, and it will not come back. Before it has, a stone is only a stone.

Old Mu left the board at the mill door. He never took it down.

Someone asked him: in a good year with good water, why keep to one sack a household?

Old Mu said: rules are not made in hard times. Rules are made in good times. In hard times, you cannot make them stand.

By now you’ve probably recognized it: this is Metastable Failure.

What it is

A metastable failure is a class of failure in which no component of the system is broken, yet a very light trigger pushes the system, through load it generates itself, into a second, equally stable bad state — and it does not return when the trigger disappears.

Three things must hold at once:

• Trigger. A brief, usually small disturbance: one slow query, one leader election, a burst of network jitter, one machine restarting. On its own it is harmless.
• Sustained effect. An effect that strengthens with load. The more load, the slower each request, the more likely to time out, the more likely to fail.
• Positive feedback (amplification). Once the system is slow, clients and upstreams manufacture more work: retries, resends after timeout, duplicate queries, refreshes, batch compensation, cache misses falling through to the origin. That extra work makes the system slower still, and the loop closes.

The essential condition is loop gain greater than one. If each slowdown buys less than one extra unit of work, the loop decays and the system recovers once the trigger passes — that was only a wobble. Once the gain exceeds one, the system has two stable points: healthy, and a state that is “slow just enough to keep itself slow.” Both are stable, and between them lies hysteresis: you can get in, and you cannot get out.

The amplifiers seen in real incidents are few and recurring:

• Retry storms. Clients time out and retry; retries make the service slower; more requests time out. Without a global retry budget, one incident’s retry volume can be several times the original traffic.
• The queue-timeout fixed point. Longer queue → longer wait → client timeout and resend → longer queue. Many systems have a mathematical fixed point here where queueing delay equals the client timeout exactly. At that point, any capacity you add gets swallowed.
• Control-plane amplification. The data plane slows, health checks start failing, routes are removed and re-added, and heartbeats, reconnects, and list refreshes all spike together.
• Cold-cache origin stampede. A crushed cache fails broadly and every request falls to the origin, which was designed to carry one percent of the traffic.
• “Checking” requests. Clients repeatedly asking “where is my thing now?” Those queries consume the same resources and solve nothing.

Its handling runs opposite to an ordinary outage:

• Capacity alone will not fix it. Added capacity is usually eaten by the loop; the system re-enters the same bad state at a larger scale. Unless the increase is large enough and sustained, you are only feeding the loop.
• You must shed load. Remove the amplifiers (retry budgets, request deduplication, coalescing, blocking check-type queries), then deliberately drop some requests (load shedding, admission control) to push the system back into the healthy state’s basin of attraction.
• Limit retries as a ratio, not a count. A retry budget caps total retries at a small fraction of original requests — ten percent, say — enforced globally, rather than each client independently retrying three times.
• Make the rules while things are good. Tightening policy after the system has entered the bad state often accomplishes nothing, because by then every load-shedding action itself provokes more client action.

The distinction from neighbouring ideas matters. Backpressure pushes pressure back upstream; it is a signalling mechanism. A circuit breaker trips on one dependency; it is a local switch. A metastable failure describes the whole system as a dynamical system with a second stable point. The symptom is “every part is fine.” The cause is “the loop is sustaining itself.”

Why it matters

Capacity planning at scale generally assumes that load and user count are the same thing. Metastable failure breaks exactly that assumption: load can be generated by the system’s own degradation.

It explains a class of incident that otherwise makes no sense. There was a small trigger at the start. Every component’s metrics stayed inside their normal range throughout. The trigger vanished long ago. The system took hours or days to return. Adding machines did not help. In the postmortem, people read every component’s logs, because every component was in fact not broken.

It also changes the order of incident response. The conventional reflex is “find the broken component.” In a metastable state there is no broken component, and the search finds nothing. The right first question is “which class of work is amplifying itself right now,” and then cut it off. Run in the wrong order and the incident lengthens, because every extra minute of searching is another minute the amplifier keeps turning.

Finally, it explains why recovery comes from shedding load, not from waiting. The common assumption is that the system heals once the trigger passes. The definition of metastability is that it does not. The system has to be pushed back, and the push goes down, not up.

Metaphor mapping

• The mill at Nine Turns → the system; the stone and the channel water → its ordinary processing capacity
• Eighty-four households, one sack each → the stable baseline demand
• The stone in the channel → a light trigger: one slow query, one failover, one burst of jitter
• The stone turning at half speed → the temporary degradation the trigger causes
• The line passing the third bend → crossing the tipping point into the feedback loop’s basin of attraction
• The Lu wife adding two sacks → client-side work amplification: retries, batch compensation, duplicate requests
• Long line → frightened hearts → more sacks → longer line → a positive feedback loop with gain above one
• The stone pulled out, the line still three li long → trigger gone, system still in the bad state; the defining signature of metastability
• One millstone becoming two → adding capacity; the loop rebuilds at a larger scale
• Grinding grain to half flour, sent back for regrinding → quality collapse under overload manufacturing extra rework, a second-stage amplifier
• No one may ask his place in the line → suppressing amplification from “checking” requests
• One sack per household per day → admission control and a work quota
• The gate closing when the yard holds more than a day → load shedding, dropping work deliberately to fall back into the healthy state
• Not with the mill, with the sacks → recovery comes from removing load, not adding capacity
• Keeping the board up in a good year → policies must be established while healthy; in the bad state they will not hold
• A stone every year → triggers are unavoidable; what must be prevented is crossing the tipping point