Two-Granary Ford lived off one river for three hundred years. The north bank rose, and people called it the East Side; the south bank fell away, and they called it the West. One stone bridge crossed the water, narrow enough that exactly two carts could squeeze past each other. At the bridgehead stood a small hut that collected a toll.

The town’s rule for storing grain was so simple nobody had ever asked why: every household’s grain goes in the granary on its own bank.

One granary east, one west. The sizes matter, so here they are: together the two held three years of the town’s flour, but a single granary held a year and a half. That was not an accident. Whoever built them had worked it out on an abacus — one and a half apiece, three between them.

Everyone knew the cart times by heart. Grain from a household on the same bank: one hour, door to granary door. Across the bridge: two hours round trip, before you counted the queue. The bridge was two carts wide and nothing on it ever moved faster. One hour on your own side, two hours over the water — that schedule was carved into the town’s bones.

There was also a finer rule of the founders, one the young had never heard.

The bins inside a granary were not scattered. They ran in one continuous row, bin against bin, from the door to the far wall. And — this is the part that matters — once a household’s row was begun, that household’s entire year of grain went into that one granary. It could not be split. Which granary held a household’s grain was therefore decided by exactly one event: who pushed that household’s first sack of rice through which door.

First pusher fixes the granary. The old people called it the first-sack rule.

Who pushed the first sack had never really been a choice. Every household pushed its own, east siders through the east door, west siders through the west door, and nobody lost anything. The rule had held for three centuries because the rule had never been broken.

That year, the town changed stewards.

The new steward was named Zhou. Strong, quick, and unable to bear the sight of households queueing with carts. He did the arithmetic: three hundred households pushing their own, an hour each, three hundred hours, and they blocked each other on the road besides. Better that he push it all himself. His cart held three times what anyone else’s did, and he could move three hundred households’ grain in a single morning.

He lived on the east bank. He went in through the east door.

By dawn, all three hundred households’ grain sat in the east granary, not one sack missing. Zhou wiped his forehead, well pleased: one morning’s work, where three hundred hours had gone before.

The autumn passed quietly. The real work was winter — three hundred households drawing grain every day to cook.

The hundred east-side households: one hour each, as always.

The two hundred west-side households had to cross the bridge first. Two hours round trip, plus the queue at the bridgehead.

In the first month, the toll hut collected more than the town’s grain tax for a year. In the second month, two flagstones on the bridge worked loose — not from weight but from traffic. In the third month, an old woman from the west bank queued for a load of grain and got home after dark, sat down in her doorway, and cried. She had no strength to curse Steward Zhou. She only could not understand it: her household’s grain had lived in the west granary for three hundred years, and this year she had to cross a river to fetch it.

Zhou refused to accept it. His first thought was to widen the bridge.

The mason said it could be done — two carts abreast to three. For a few days it helped. But the town’s problem was this: the bridge’s width was the town’s annual ceiling on cart crossings. Widen it by a third and traffic grows by a third, while the west bank’s traffic is twice the east’s. No width catches up to twice.

He stood at the bridgehead a whole afternoon and saw one thing clearly: the crossing carts were not “a bit more.” Every single trip cost double, and the queue jammed all of them into one place. A slow bridge you can endure. A jammed bridge brings everything down together.

At the end of the second month he went to the old granary keeper.

The old man heard him out, then asked one question: “When you pushed, was it one man pushing?”

One man, Zhou said. One morning.

The old keeper sighed. “The first-sack rule was never about which granary. It was about which pushers. Whoever pushes a household’s first sack — that household’s row lands on the bank where the pusher stands. You pushed for three hundred households alone. Three hundred rows landed on the bank where you stood.”

Zhou sat in the granary doorway half the night. He worked out three things, and those three became the town’s new rules.

One: whoever uses it pushes the first sack. Drawing grain is work three hundred households do together — a hundred east, two hundred west, starting at the same hour every day. So pushing the first sack should be work three hundred households do together too: each household pushes its own first sack through its own bank’s door. All Zhou’s strength bought him exactly one thing — his own position, not the town’s. He tripled the speed of pushing and turned half the grain-drawers into river-crossers. He saved the first morning and paid for it all year.

Two: grain with no clear owner gets spread across both. Some bins held no household’s grain but the public store — lamp oil for the toll hut, ink and brushes for the school, firewood for winter. Everyone draws from it, east and west alike; there is no owner to name. Those bins get laid out alternately: one east, one west, one east, one west. West-siders drawing public grain skip the bridge half the time; east-siders cross half the time. Nobody is fast, nobody is slow, and the bridge traffic is flattened.

There was objection at first: why should my household’s public grain cross the river? The old keeper answered with one line: the bridge does not fear volume, it fears concentration. Flattened, the bridge lasts a year. Piled on one side, it collapses in three months.

Three: a pinned bin has to be pinned right. In later years the town added a few things used in only one place: anchor storage at the east ferry, used only by the ferrymen; millstone stock on the west bank, used only by the mill. Those bins the steward marks plainly — “east granary only,” “west granary only.” Marking them is a great gain, because the few people who use them constantly are standing on that bank. Marking them is also a great risk: pin it wrong and that household crosses the river every day, and nobody ever finds out. The granary raises no error. The ledger balances. It is simply slow. Slowness does not raise alarms.

In the fourth winter, something new happened.

Two west-bank households shared a set of bins — not a great estate, just a couple who had divided a household without dividing it cleanly: grain from the same field, drawn by both, one making vinegar and one making sauce, two trips each every day. By rule two, those bins should be spread across both banks. But the old keeper had fallen ill that year, and the young man who took over trusted something new: the county’s bookkeeper.

The bookkeeper prescribed nothing in advance. He only counted: who drew grain from which granary each day, how many trips, how many catties — and then moved bins according to that count. West-siders drawing more, so move bins from east to west. East-siders drawing more, move them back. The town gave his office a name: watch who draws, move toward them.

For most households the method was excellent. That summer Zhou tallied a sixty percent drop in bridge crossings, and the toll hut’s revenue fell so far the hut’s owner came to complain.

But the couple who had divided their household without dividing it drove the bookkeeper mad.

Today the west-bank vinegar maker drew four times, and the bookkeeper moved the bins west. Tomorrow the east-bank sauce maker drew four times, and he moved them back east. The day after, both came in turn, and he moved them six times in two days. Moving a bin means crossing the bridge, two hours a trip, a cart holding only so many bins — one move cost three times what the saved trips were worth.

At year’s end, those two households had crossed the bridge twice as often as in any previous year. The young man brought the ledger to the old keeper, now recovered, who glanced at it and laughed. “Those two are not in the wrong place. They are genuinely used from both ends. Treat them as misplaced and they will be misplaced forever.”

That winter the town wrote the fourth rule: the bookkeeper moves what is truly misplaced, never what is truly shared. If you cannot tell the two apart, do not move — a wrong move costs more than no move.

In the fifth spring, a wooden board went up over the granary door, carved at the old keeper’s order. Three lines, for every steward to come:

One: the first sack, the user pushes. Push for someone else and you push their row under your own feet. Two: public grain spread across both banks; a household’s own grain stays its own. Three: count before you move. Count whether it is misplaced, or truly shared.

Beneath the board lay a notebook the bookkeeper had left behind. Page after page of numbers: how many trips drawn from the east granary, how many from the west, how many crossed the bridge, and how many trips took grain that belonged to this granary but had been carried off by the other.

The old keeper said the notebook was more use than the board. The board gives principles. The notebook gives evidence.

The bridge at Two-Granary Ford still stands, still two carts wide. The toll hut still collects, and the toll has not risen since. The town has a saying: you do not widen the bridge, you make it legible.

By now you have probably recognized it: this is NUMA — Non-Uniform Memory Access — together with first touch, interleave, and auto-balancing, the whole discipline of memory placement.

What it is

Start with the hardware. On a multi-socket server, each CPU (socket) brings its own memory controllers and its own half of the DRAM, and that half is a NUMA node. Read memory on your own node and you go through a local memory controller. Read the other node’s memory and you cross the link between chips — Intel’s UPI, AMD’s Infinity Fabric. The idea that memory access takes uniform time has been false since the day multi-socket machines were born.

Order of magnitude: local access is roughly 80–100 ns; cross-node access is 1.3–2× that, and the cross-node link’s bandwidth is limited — aggregate bandwidth on a two-socket machine is typically 1.4–1.7× a single socket’s, not 2×.

Now the software side. malloc hands out virtual addresses, not physical memory. The physical page is allocated on the first write that faults it in, and it is allocated on the node of the CPU that took that fault. That is first touch.

The consequence is immediate: one thread mallocs 8 GB and initializes it, then spawns 32 threads to process it — all 8 GB sits on that one thread’s node, and half of the other threads’ accesses are remote. The time saved during initialization is paid back for the entire run. That is Steward Zhou’s ledger.

Five placement policies:

• Local (MPOL_DEFAULT): allocate wherever the fault happens. The default, and already optimal for a program where the consumer initializes.
• First touch by the consumer: let the threads that will read and write the memory perform the first touch (parallel init, per-thread arenas). The most effective measure, and the most commonly missed.
• Bind (MPOL_BIND): numactl --membind=0 or mbind(2), nailing a region to one node. Right for data only a few fixed threads touch. Bind it wrong and the entire access path is remote — with no error, just slowness.
• Interleave (MPOL_INTERLEAVE): numactl --interleave=all, striping pages round-robin across nodes. Individual accesses are always half remote, but total bandwidth is flattened and no single node’s controller saturates. The usual choice for databases and large scans (PostgreSQL has no NUMA awareness at all; the documented advice is interleave).
• AutoNUMA (/proc/sys/kernel/numa_balancing): the kernel samples faults and scheduling to migrate pages toward the node that accesses them, and tasks toward the node holding their data. Convenient — but for data genuinely written from several nodes it misreads “shared” as “misplaced” and migrates back and forth, slower every round. That is the couple who divided their household.

Measurement: numastat -p <pid> gives four numbers — numa_hit (allocated on this node), numa_miss (should have been here, landed elsewhere), numa_foreign (should have been elsewhere, landed here), interleave_hit. On the perf side, cross-node load counters and misses. That is the first thing to look at, not the code.

Why it matters

Because this is the category of thing that silently adds twenty percent when configured right and silently removes forty percent when configured wrong, and neither case raises an error. In the cloud it is especially easy to hit:

• The instance size is itself a NUMA boundary. Large VMs span nodes; small ones usually do not. The same code behaves on 4 vCPUs and then, on 64, shows “CPU at sixty percent but throughput will not rise” — which is not a CPU bottleneck but a memory-bandwidth one.
• Initialization paths and run paths are usually different threads. The main thread builds indices, loads snapshots, warms the heap; the worker pool starts afterwards. This practically guarantees fully remote access, and it hides in startup where profilers do not look.
• Transparent huge pages amplify the skew. A fault may hand out 2 MB at once, and a huge page can live on only one node — the granularity of skew goes from 4 KB to 2 MB, and huge pages collapsed by khugepaged in the background are even less local.
• Virtual machines must report the topology honestly. If the guest cannot see its vNUMA topology, the guest scheduler assumes memory is uniform — and every placement decision it makes is based on a wrong model of the world.

Three practices, ordered by payoff:

1. Let the consumer do the first touch. Parallel initialization, per-thread arenas, bind the thread before you allocate. This alone usually captures most of the loss.
2. Interleave shared read-mostly and large-scan data. Trade peak locality for flattened bandwidth — a good trade.
3. Do not turn auto-balancing loose on genuinely shared hot data. Check numastat for the ratio of numa_miss to numa_foreign first, and confirm it is misplaced before you move it.

And the trade-off that gets missed: locality is not throughput. Pin everything to one node and local hit rate reaches 100 percent — and that node’s memory bandwidth ceiling becomes the system’s ceiling. You do not widen the bridge at Two-Granary Ford, you make it legible. What you want is not “nobody crosses the river.” It is “no granary is saturated and no granary is idle.”

Metaphor mapping

• East bank / west bank → the two NUMA nodes (each socket’s local DRAM)
• The river and its stone bridge → the chip-to-chip interconnect (UPI / Infinity Fabric), the cross-node path
• The bridge being two carts wide → the cross-node link’s bandwidth ceiling
• One hour on your own bank, two hours across → local access ~80–100 ns, remote 1.3–2×
• Loose flagstones, the old woman queueing until dark → bandwidth saturation and queueing delay, not a single slow access but a systemic jam
• A granary holding a year and a half, both holding three → each node’s local capacity and bandwidth are a finite half
• Each household’s bins laid in one continuous row → a page is contiguous physical memory living on one node
• “First pusher fixes the granary” → first touch: the physical page is allocated on the node of the CPU that takes the write fault
• Steward Zhou pushing for three hundred households → one thread mallocing and initializing, then many threads consuming
• He lives east, enters by the east door → the initializing thread happens to run on one node
• All three hundred rows land in the east granary → every page lands on one node; half the threads now run remote
• Widening the bridge helps for days → more bandwidth does not fix the fraction of traffic that must cross; interconnect bandwidth is not 2× single-socket
• The keeper’s question, “who pushed the first sack?” → the diagnostic entry point is not the code path but the initialization path
• Rule one: the user pushes the first sack → consumer-side first touch (parallel init, per-thread arenas)
• Rule two: public grain spread across both banks → MPOL_INTERLEAVE, pages striped across nodes to flatten bandwidth
• Rule three: pin plainly, but a wrong pin is slow and silent → MPOL_BIND / numactl --membind, wrong pins are remote with no warning
• The bookkeeper’s “watch who draws, move toward them” → AutoNUMA page and task migration
• It works well for most households → auto-balancing pays off on genuinely misplaced data
• The divided couple drawing in turn every day → data genuinely written from several nodes
• Six moves in two days, slower than before → mistaking shared for misplaced; migration cost eats the gain, then thrash
• “Move what is misplaced, never what is shared” → check numa_miss / numa_foreign before migrating
• The notebook’s counts: own-bank trips, crossings, stolen loads → numastat -p counters numa_hit / numa_miss / numa_foreign / interleave_hit
• Huge pages turning one bin into a whole row → THP coarsening the skew granularity from 4 KB to 2 MB
• The inn’s two rooms on opposite sides of the building, guests unaware → the guest cannot see vNUMA, so its scheduler decides on a wrong world model
• “You do not widen the bridge, you make it legible” → the goal is not zero remote access; it is no node saturated and none idle