It rained all night. The kitchen hatch at the Lantern Inn was open just a crack, and the runner boys kept emerging from the downpour one after another, pushing soaked paper slips through the gap.

The woman at the hatch was He, who kept the inn. She had learned one thing early: on rainy nights, slips get lost. A boy runs over from the tavern across the river, slips in the mud, and the paper dissolves into the ditch — so the customer, hearing nothing, writes another slip and sends another boy. Sometimes the first boy actually arrived, and only fell on the way back, so the receipt never made it home. Either way, one order of “three bowls of mutton soup” might reach her window three times in an hour.

In her early years He did the obvious thing: one slip, one order. Which is how a man who wanted three bowls of soup ended up with nine, and a bill for three. Slowly she understood: the boys on the road are unreliable and she cannot change that. The only thing she can change is her side of the window.

So a wooden board went up beside the stove, studded with a row of brass hooks, and the rules became six.

One: the customer writes the number. Every slip must carry a ticket number — not one He invents, but one the ordering tavern writes. However many times the same order is resent, the number stays the same, because only the customer knows that two slips are secretly the same event. He does not. When a boy arrives, the first thing she looks at is not the dish. It is the number.

Two: hanging the ticket is the check. The hooks have a peculiarity: one hook holds exactly one ticket. Hanging the slip and checking whether the hook is free are the same reach of the same hand — if it hangs, nobody has done this; if it won’t, someone is doing it or has done it. Even if two boys lunge at the window in the same instant with identical slips, only one of them hangs. The other is sat down by the fire to wait.

Three: the number needs a fingerprint. Once a ticket is on its hook, He writes the order’s fingerprint on the back: how many bowls, which soup, which table. A tavern once got lazy and reused yesterday’s leftover ticket number on a fresh order — same number, but eight bowls of noodles this time. She sent it back. A ticket number identifies the same event, not the same customer. Same number, different event, is a clerical error, not a resend.

Four: serving and settling must be one stroke of ink. This is the rule that matters most. She used to carry the soup out first and mark the ledger afterward. One night a fire broke out in the kitchen: the soup went out, the ledger line never got written — and the next day, when that ticket came back, she read her empty ledger and cooked the whole thing again. Now the soup going into the bowls, three portions of mutton coming off the storeroom count, and the ticket number settling into the ledger all happen in one line of ink: all of it, or none of it. If the lamp tips over halfway, the whole line is void, the ticket comes off its hook, and it is as if the slip never arrived.

Five: keep a copy of the receipt. He discovered that not cooking it twice wasn’t enough. The second boy is still standing at the window with an identical slip, and he needs something to carry home. So the hook holds not just the ticket but a copy of the receipt she wrote the first time: “three bowls mutton soup, served, east wing table two.” When the resend arrives, she cooks nothing — she copies the receipt and hands it over. The customer’s answer is identical to the first one. He never learns that he asked three times.

Six: the board is only so big. Every three days He clears the old tickets off. Her wager is that a boy lost in the rain stays lost for at most three days. Anyone still waving an ancient ticket number after that isn’t resending. That’s a different story.

— By now you’ve probably recognized her: what He is really guarding is not a board of hooks. It’s idempotency.

What this is

An unreliable network can only offer at-least-once delivery. The request may be lost; the reply may be lost; and the sender cannot tell those two cases apart, so it retries. Retrying means the server may receive the same request many times. True exactly-once delivery is unachievable at the network level. What is achievable is an exactly-once effect: receive it as often as you like, the side effect happens once.

The mechanism is the idempotency key. The client generates a unique key per logical operation and reuses it on every retry. The server claims that key in a deduplication table with a single atomic insert-if-absent, and — this is the step most often gotten wrong — commits the business side effect and the key insert in the same transaction. Do the effect first and record the key after, and a crash in between means it never happened. Record the key first and do the effect after, and a crash in between loses the order forever.

You must also store the first response. A duplicate request should replay the original reply, not return a “duplicate request” error: to the client, a retry has to look exactly like a first call. The key is normally bound to a hash of the request payload (so a reused key on a different request is rejected rather than silently swallowed), and carries a TTL — the dedup table cannot grow forever — where the TTL must outlive the client’s maximum retry window.

It’s everywhere: Stripe’s Idempotency-Key header, ClientToken on AWS APIs, Kafka’s producer id plus sequence number, and every Kubernetes controller whose reconcile reads observed state before deciding whether to act. Controller authors especially should remember: being invoked repeatedly for the same event is the normal case, not the failure case.

The mapping

• The rain, and the boys who fall — an unreliable network: lost requests, lost responses, indistinguishable to the sender
• The same soup ordered three times — duplicate requests under at-least-once delivery
• The ticket number written by the customer — the idempotency key, which must be generated client-side and reused across retries
• One hook, one ticket — an atomic insert-if-absent on the dedup table (unique index or conditional write)
• Two boys lunging at once, only one ticket hanging — the concurrent-duplicate race, solved by atomicity rather than check-then-write
• The fingerprint on the back of the slip — binding the key to a hash of the request payload, so a reused key on a different request is rejected
• Serving and settling in one stroke of ink — side effect and key insert committed in the same transaction
• The tipped lamp voiding the line, ticket off the hook — transaction rollback releases the key so the retry can succeed
• The receipt copy hanging on the hook — caching the first response and replaying it for duplicates instead of erroring
• Clearing the board every three days — the dedup table’s TTL, which must exceed the client’s maximum retry window