There was a dye works on the river called Ten Thousand Blues.

It had three courtyards in a row: the well yard, then the mordant yard, then the dye yard. The walls were low, so from a passing boat you could see the rows of vats inside, each with a skin of indigo floating on top, catching the sun like a row of small deep pools.

Ten Thousand Blues had one rule that startled every stranger who heard it.

When a vat turns, you may not fish it out.

A vat of cloth takes three days. On the third morning you open it, and the colour must be as true as a well shaft. If it has gone wrong — grey, green, cloudy, or worst of all, correct to the eye and correct to the hand, but with a thread of sourness when you lean in close — that vat is ruined.

So what do you do with a ruined vat?

Every new apprentice has the same first instinct: fish the cloth out, rinse it in clean water, put in a fresh measure of indigo, press in half a catty of lime to smother the sourness. Three days, forty bolts — surely most of it can be saved.

The old master’s answer was always the same gesture.

He would walk over, look down into the vat once, and then overturn it.

The water went into the ditch. The cloth was dragged out to dry and cut into rags. The vat was scrubbed, refilled from the well, dosed from the indigo cabinet, and started again from the first step of the first day.

Three days. Forty bolts. Into the ditch.

The apprentice would be in too much pain to speak.

The old master said one thing. Ten Thousand Blues passed it down for seven generations:

A half-spoiled vat costs more than a turned one.

—

It took A-Ling three years to understand that sentence.

Her first year, she followed Third Master and watched vats. Third Master spoke very little and taught even less. He taught one thing: when you watch a vat, you do not touch it.

“Watch,” Third Master said. “If it goes wrong, tell me. Don’t stir it. Don’t add water. Don’t patch it.”

A-Ling asked why.

Third Master said: stir it once and it becomes a different thing. It was going to turn. It turns, I know it turned, I start a new vat. You stir it, it doesn’t turn — but it isn’t the same vat either. It will dye a batch that looks excellent, sell, and after three months of wearing fade into patchwork. And when that happens, it won’t be Third Master who comes looking. It will be the whole city.

That year A-Ling recorded thirty-seven turned vats.

Each time, Third Master carved a line into the board on the wall: the day, the vat number, the hour it turned, why it had gone wrong. Only then did he fetch water.

A-Ling asked: why carve that? It’s already turned.

Third Master said: One turn is an accident. Ten turns is a sentence.

—

The spring of her second year, the well yard had a problem.

The water for the mordant yard’s ash liquor was carried up from the well yard. That well had taken two spring rains, and the water had picked up a little iron. Not much — no one could taste it. But ash liquor touched by iron dyes a dull blue.

Vat Seven turned first.

Third Master overturned it and started again.

The next day, Vat Seven turned again.

Third Master overturned it again. On the third morning, Vat Seven turned for the third time.

This time Third Master did not go to the vat. He untied the red cloth from his waist, walked to the mordant yard gate, and tied it to the door ring.

The red cloth went up, and the mordant yard stopped.

A-Ling was frantic: a whole courtyard of cloth is sitting in the vats. What does a stopped day cost?

Third Master said: When one vat turns three times, it is not about the vat.

He said: the first time I thought it was a hand. The second time I thought it was the indigo. The third time I knew I did not understand it. And a thing I do not understand, I can overturn a hundred times and it will still be a hundred times. Overturning vats is what I know how to do. Finding iron in a well is not. So I hand it up.

The red cloth went to the head steward.

The head steward did not overturn a vat, and he did not curse anyone. He did something else: he had the boards from those three days brought to his table and laid them out in a row.

At the ninth board he looked up. More than Vat Seven had turned in those three days. There were Two, Eleven, Fourteen — all of them dyed with liquor from the mordant yard.

So he went to examine the well.

What he found was a crack at the base of the courtyard wall, with rain seeping through the soil into the shaft. They patched the wall, cleaned the well, and the mordant yard reopened.

That afternoon A-Ling stood at the well yard gate and stared at the red cloth for a long time.

She asked Third Master: and if you had overturned it that third time?

Third Master said: then I would have overturned vats until I died. A man who keeps overturning vats never finds the well.

—

Only later did A-Ling learn that the house rules were finer than what she could see.

Well yard, mordant yard, dye yard — that order was not arbitrary. You light the fires in that order to open, and you put them out in reverse to close. Because each courtyard lives off the one before it: no liquor from the mordant yard, no opening the dye yard; no dye yard, nothing to spread in the drying yard.

When a vat turns, how far up you overturn follows the same order.

Sometimes you overturn only that one vat. That is called changing one.

Sometimes a vat in the mordant yard has gone wrong, and then you cannot change only that one — you overturn every vat from that one onward in the opening order. Because all of those used what that vat produced. That is called changing this one and everything after it.

And sometimes, when the colour-head vat of the dye yard goes wrong, the whole dye yard is overturned together. The colour-head sets the year’s colour for the entire yard; if it is wrong, then every “close enough” colour in the other vats stops counting. That is called changing the yard.

Which of the three to use was written on the board beside the vats. It was not allowed to be decided on the spot.

A-Ling asked the head steward: why not just change the yard every time? It’s the safest.

The head steward laughed: three hundred yard-changes a year — are you paying for the cloth?

—

There was something else A-Ling only noticed later.

Not every turned vat gets restarted.

The vat that belongs to the old well is dead: it never rests, and if it turns it is always restarted.

The seasonal indigo vat is alive: if it turns mid-dyeing, restart it; if it turns after the day’s work is finished, leave it and wait for the next season.

The colour-testing vat is once-only: if it turns, it turns. It is never restarted. It existed to be looked at once, and once was enough.

The head steward said: get those three straight and you save half the indigo in a year.

Then there was closing the fires. A yard master never smashes a vat. He calls down and the vat’s keeper scrubs it out and smothers the fire himself. Only after ten calls with no answer does the master break it.

A-Ling asked: why not just smash it? It’s faster.

The head steward said: smash one vat and the whole yard is in splinters. The shards fall into the other vats, and tomorrow you overturn three more.

—

And then there was the rule A-Ling found least reasonable of all.

If a yard overturns three vats within one double-hour, the master must stop the yard.

Not “let’s watch it a while.” Not “let’s hold on a bit longer.” Stop. Red cloth on the door ring. The master stands in the courtyard and waits for the head steward.

A-Ling asked: what if the fourth vat would have been fine?

The head steward said: then it turned for nothing. I accept that.

A-Ling asked: then why three? Why one double-hour?

The head steward said: because those numbers came out of seven generations of ledgers. The numbers themselves don’t matter. What matters is that there is a number.

He said: a master with no number in his head will spend his whole life overturning vats. He will overturn diligently, and painfully, and look like a capable man. And the well that actually broke will never be looked at.

—

In the winter of her third year, A-Ling became a master for the first time.

She had the drying yard. Last courtyard, lightest courtyard. The work was simple: cloth comes up from the dye yard, gets spread on bamboo racks, and you watch the sky and take it in.

Her whole drying yard was one bamboo rack.

The first month, she overturned twice.

The first time, the wind was strong and the rack went over into the mud with the cloth on it. She rolled the cloth up and went back to the dye yard for more.

The second time, she was ill for three days, and when she came back the cloth had dried past its time and was stiff enough to stand on end. She rolled it up and went back to the dye yard for more.

Both times she did the right thing.

The third month, it happened.

It snowed for half a month without stopping. The drying yard took in forty racks of cloth and stacked them indoors. There was a charcoal brazier in the room, and the fumes sat and did not clear, and a little yellow spotting appeared on the cloth.

A-Ling’s first instinct was to pick out the spotted pieces, hang them alone in the draught, and see whether it would blow off.

She crouched by the pile and picked for two double-hours.

At the thirty-seventh piece, her hands stopped.

She remembered what Third Master had said: stir it once and it becomes a different thing.

She stood up, entered all forty racks into the ledger one by one, and went to knock on the head steward’s door.

She said: I overturned the drying yard.

The head steward was eating congee. He looked up: all of it?

A-Ling said: all of it. The spotting is not something I can pick clean. I picked for two double-hours, and everything I picked out was the part I could see. The part I couldn’t see will grow in the customer’s wardrobe.

The head steward put down the bowl and looked at her for a long time.

Then he said: come to my room tonight. I’ll show you the ledger.

—

That night, the head steward opened a very old book.

In it was drawn a tree. But it didn’t look like a tree — it looked like an upside-down dye works: a bar at the bottom, three forks rising from it, and from each fork a few more lines.

The head steward said: this is the root of Ten Thousand Blues.

The bar at the bottom was him. The three forks above it were the three yard masters. Above each master were the vats of that yard. And if something stood above a vat — say the stove that tended the fire in the dye yard — then the stove had its own little book, drawing the sticks of firewood that stood above it.

A-Ling said: and me?

The head steward pointed to a line at the very edge: you are here. Above you is the drying yard. If the drying yard turns, I know. If the drying yard can’t hold, I know and I have to move. And if I can’t hold—

He paused.

If I can’t hold, the works closes.

A-Ling asked: what does closing look like?

The head steward said: closing is not a collapse. Closing is me putting out the fires one by one, from the top down, and nobody is allowed to put out his own. When the last stove is out, I lock the door.

He said: in this seat, I don’t overturn two vats in a year. The day I start overturning vats is the day the works is closing.

A-Ling asked: why can’t you overturn any?

The head steward said: because I am not supposed to have a vat.

He said: look at what’s in this room. One lamp, one ledger, one well key. I have no vat, so I have nothing that can go wrong. The only work I have is: who turned, who should be overturned, and how far up. That work looks light. It is the heaviest work here — which is why nothing else is allowed on top of it.

—

A-Ling grew old in time.

Someone asked her: why has Ten Thousand Blues stood for a hundred years?

She said: because it is not afraid of turned vats.

The questioner said: which dye works is afraid of turned vats?

A-Ling said: plenty of them are not afraid of turning a vat. What makes Ten Thousand Blues different is — it is not afraid of failing to save itself.

She said: one vat is ruined, I overturn it and start again. One vat turns three times running, the master stops the yard. One yard stops three times, I stop the works. I never know what the next turned vat will cost. All I know is that every turn goes up one level. And the level above has something the level below doesn’t — another well, another ledger, another pair of hands with no vat of their own.

So the problem always gets handed, while it is still small, to someone who can see it.

The last thing A-Ling said was carved on the well yard wall:

The one who saves, saves a bolt. The one who overturns, saves a works.

— —

By now you’ve probably recognized it: this is the Erlang/OTP supervision tree — the architecture behind the doctrine called let it crash.

What it is

Erlang’s unit of concurrency is the process: extremely light, single-threaded, with its own heap and no shared memory. Two processes hold no pointers into each other. They exchange messages. This gives rise to one consequence, and that consequence is the foundation of the whole fault-tolerance model:

One process crashing cannot corrupt another process’s data.

So Erlang’s idiom runs against the mainstream. The mainstream anticipates every failure inside every function, catches it at each layer, repairs the bad state in place. Erlang’s slogan is let it crash — do not write recovery logic into business code; let the process die cleanly and restart from a known-good initial state. The reasoning: a process that is half-broken but still running keeps producing answers that look correct, and those answers cannot be distinguished from correct ones. A dead process is at least honest.

A link ties two processes’ fates together: when one exits abnormally, the other receives an exit signal. A monitor is the one-way version — notification without shared fate.

A supervisor is a process that does three things: start its children in order, watch them exit, restart them according to a strategy. It does no work itself. Each child is described by a child spec: how to start it, its restart type, its shutdown timeout.

There are three restart strategies, and they map to real dependency structure:

• one_for_one: restart only the child that died.
• rest_for_one: restart the dead child and every child started after it — because they depended on it.
• one_for_all: one dies, all are killed and restarted together — because they must share a generation.

Restart types come in three flavours: permanent (always restarted), transient (restarted only on abnormal exit), temporary (never restarted).

Restart intensity is the safety valve: each supervisor carries max_restarts and max_seconds. If restarts exceed the limit inside the window, the supervisor stops restarting and terminates itself, handing the problem to its own supervisor. That supervisor has a wider window, a wider view, and less work in flight. If the root supervisor exceeds its limit, the entire application terminates.

The error kernel is the design principle that follows: the closer a component is to the top of the tree, the less it should do. The top holds only a little very stable logic that should never restart — configuration, the name registry, connection-pool bookkeeping — and pushes everything risky, crashy, and complicated down to the leaves. The cost of a crash is thereby bounded to the lower tree, while the top retains enough context to answer “is this actually a problem with the well?”

Around all of this: ordered startup, reverse-order shutdown; shutdown sends shutdown and waits for a timeout before resorting to brutal_kill; registered names let clients find a service by name after a restart instead of chasing a pid; and every crash emits a crash report with the full exit reason and stack trace.

Why it matters

• It turns fault tolerance from prediction into structure. Conventional code asks you to anticipate failure modes and write a recovery branch for each, and failure modes are unbounded. The supervision tree admits it cannot predict: shrink the fault domain, make crashing cheap, make restarting fast. The “nine nines” (99.9999999%) availability figure in the Erlang world does not come from smarter recovery logic. It comes from the arithmetic of “it crashed, so restart it.”
• Restart intensity bounds how far a failure travels. An auto-restart system without max_restarts spirals: a permanent defect is buried under infinite retries, and the system looks like it is healing while it is idling. When the limit is exceeded, the supervisor terminating itself is not surrender — it is handing the decision to the layer that has more information. A leaf can only overturn vats. The top of the tree is what can examine the well.
• Blast radius is set by the shape of the tree, not by code quality. one_for_one spends the least to protect the most. rest_for_one admits that startup order is the dependency graph. one_for_all admits that some components must share a generation. Pick the wrong strategy and you either waste your restart budget or mix two generations of state together — the hardest class of failure to diagnose.
• The error kernel is an inverse relation between importance and workload. The more critical a component is, the less code it should contain. Putting complex logic at the top of the tree puts the one thing that must not fail in the place most likely to fail.
• It forces state out of the process. A restart zeroes memory, so any supervised component must be designed to be rebuilt from outside: state in a database, in a log, in the peer’s hands. That constraint incidentally makes components movable, parallelizable, and testable.
• It converts failure into readable evidence. Every crash is an exit reason plus a crash report. Do not let the process die and you get no report. A silently degrading system produces no evidence at all, and therefore can never be fixed.
• Kubernetes is its large-scale projection. Probes, restartPolicy, CrashLoopBackOff, the parent-child relation between controllers — the same intuitions, with granularity and cost one or two orders of magnitude larger. An Erlang process restart costs microseconds with no scheduler involvement; a container restart costs seconds, an image pull, and a volume reattach. How cheap your restart is decides how boldly you are allowed to crash.

Metaphor mapping

• One vat, one keeper; nobody else puts a hand in → independent per-process heaps, no shared memory; cross-process communication is by message only
• A turned vat leaves the neighbouring vats untouched → a process crash cannot corrupt another process’s state
• Do not fish out a turned vat; overturn and restart → let it crash: no in-place recovery logic; terminate and restart from initial state
• Rinsing, re-dosing, lime to smother the sourness → half-broken state kept in service, producing output indistinguishable from correct
• Correct to the eye, correct to the hand, one thread of sourness → syntactically valid but corrupted state, hardest for downstream to detect
• The cloth fades into patchwork after three months and only then does the customer return → delayed failure exposure; the fault point and the observation point are far apart in time
• A half-spoiled vat costs more than a turned one → degraded operation costs more than one clean restart
• Carving day, vat number, hour, and reason into the board → crash reports: exit reason, timestamp, stack trace
• “One turn is an accident, ten turns is a sentence” → a single crash is noise; repeated crashes are a signal about a systematic defect
• A cord from each keeper’s waist to the master’s hand; a turned vat jerks it → links and exit signals; failure is reported immediately, with no polling rounds
• The master does not overturn vats; he decides who turned, who gets overturned, how far up → supervisors only start, monitor, and restart; they carry no business logic
• Changing one → one_for_one
• Changing this one and everything after it → rest_for_one
• Changing the yard → one_for_all
• Lighting the fires in well yard → mordant yard → dye yard order → children start in child-spec order; startup order is the dependency graph
• Closing the fires in reverse, from the top, nobody allowed to close his own → ordered graceful shutdown initiated by the supervisor
• Ten calls unanswered before the master breaks the vat → shutdown timeout then brutal_kill; the shards that injure other vats stand for external resources left uncleaned by a hard kill
• The well’s vat, dead, restarted whenever it turns → permanent
• The seasonal vat, restarted mid-dye but not after hours → transient: restarted only on abnormal exit
• The colour-testing vat, never restarted → temporary
• Three vats in one double-hour stops the yard → max_restarts / max_seconds, the restart intensity limit
• The yard stops without argument, without trying a fourth, and the red cloth goes up → on exceeding the limit the supervisor terminates itself and escalates
• “The numbers don’t matter; what matters is that there is a number” → intensity thresholds are tunable but must exist; unbounded restart is concealment
• Three and one double-hour came out of seven generations of ledgers → thresholds come from real failure distributions, not intuition
• A master who keeps overturning vats never finds the well → an unbounded restart loop disguises a permanent defect as self-healing
• The head steward compares three days of boards and finds more than Vat Seven → the supervising layer holds a wider time window and a wider observation surface
• The cause is a crack at the wall base letting rain into the well → the root cause is above the leaf level; every leaf-level repair was symptom handling
• The head steward’s room holds one lamp, one ledger, one well key → the error kernel: top-of-tree components must be tiny, dependency-light, almost free of business logic
• “I have no vat, so I have nothing that can go wrong” → the top layer must not hold fallible business state
• “The day I start overturning vats is the day the works is closing” → the root supervisor exceeding its limit means the whole application terminates
• Closing is not a collapse; it is fires put out one by one, then the door locked → ordered application stop, not a crash
• Below each master are stoves; below each stove, firewood; each with its own book → the supervision tree is recursive; supervisors are themselves supervised children
• A-Ling’s drying yard is one bamboo rack; she goes back to the dye yard for more → restarted leaves re-acquire state from upstream rather than rebuilding it themselves
• Two double-hours of picking, and only the visible spotting comes out → manual filtering can only handle exposed damage; latent damage flows downstream
• A-Ling overturns all forty racks → when you cannot distinguish damaged from undamaged, a full reset is the only correct action
• “The one who saves saves a bolt; the one who overturns saves a works” → local recovery preserves one request; structural restart preserves the system
• Every level up brings another well, another ledger, another pair of hands with no vat → higher supervisors have more information, a wider window, and less load