Old Zhou ran a noodle stall at the end of the alley. Once business picked up, he hired a young cook, Chen, to man the wok. But Zhou didn’t quite trust the kid’s pace — afraid he’d burn through a whole day’s gas in a rush — so he set a rule: the day was carved into little segments, each the length of one burning incense stick (about five minutes). Within each stick, Chen was allowed to keep the flame roaring for only a quarter of that time. The other three-quarters, the burner had to stay dark — no matter what was still sizzling half-cooked in the pan, no matter how long the line outside grew.

On slow days this was no problem. Orders trickled in, and Chen often finished everything using just a tenth of his allotted stick — the stove looked idle nearly all day. But at the lunch rush, orders poured in all at once. However fast Chen’s hands moved, by the time the stick had burned through a third of its length, his quota for that stick was gone. The burner was snuffed out on the spot — pan still half-raw — and Chen could only stand there and wait for the next stick to be lit before he could resume.

Here’s the maddening part: when Old Zhou checked his ledger, all he saw was “Chen uses the flame only 20% of the time on average” — so he concluded the stove had plenty of room to spare and never thought twice. But the customers queued at lunch didn’t care about the daily average; they cared about the few minutes their dish sat stalled mid-cook, and that stall never showed up as a single line in the ledger. Old Zhou was baffled — the books said the stove was practically idle, so why did people keep grumbling about waiting forever at lunch? It wasn’t until he crouched by the stove with a stopwatch, counting things out stick by stick, that he realized the problem was never about the total burned all day — it was about exactly what fraction of each individual stick got used before the flame was snuffed.

By now you’ve probably recognized it — this is really about cgroup CPU throttling. When a container in Kubernetes/OpenShift has a CPU limit set, the kernel’s CFS (Completely Fair Scheduler) slices time into periods (100ms by default), and how much CPU time the container may use within each period is governed by its quota. Even if the container’s average CPU usage across the whole day sits well below the limit, the moment it bursts and exhausts its quota within a single 100ms period, the kernel suspends it (throttled) until the next period begins — producing sharp P99 latency spikes, while the dashboard’s average CPU usage graph keeps insisting everything is fine. That’s exactly why this failure mode is so easy to misdiagnose: averages hide throttling completely. You have to look at metrics like container_cpu_cfs_throttled_periods_total to catch the real culprit.

Metaphor mapping

• Old Zhou’s “only a quarter of each stick” rule → the CPU quota derived from the limit (how much CPU time is allowed per period)
• The length of one incense stick (five minutes) → the CFS period (100ms by default)
• Chen’s burner being snuffed out, forced to wait for the next stick → the container getting throttled, suspended until the next period
• Old Zhou’s ledger showing “only 20% average usage” → the dashboard’s average CPU usage metric (looks healthy, hides the truth)
• The lunch-rush customers’ few minutes of stalled waiting → the request’s P99 latency spike
• Old Zhou crouching by the stove with a stopwatch → switching your investigation from average metrics to cfs_throttled_periods