The 75-cent rule

2026-05-09

In 1986 an astronomer-turned-sysadmin at Lawrence Berkeley Lab was asked to chase down a 75-cent accounting error in the computer billing system. Two accounting programs disagreed by less than a dollar. Most people would have written it off — rounding, surely — and gone to lunch. He pulled the thread instead, and the thread ran all the way to a hacker selling military secrets to the KGB. He wrote a book about it. If you work anywhere near this field and haven't read The Cuckoo's Egg, fix that before you read anything else I've written.

The rule I take from it: small discrepancies are free diagnostics. Not warnings, not noise — diagnostics. Somebody already paid for them. The system has done you the courtesy of disagreeing with itself, and all you have to do is notice.

Why small is the interesting size

Big failures are easy. The disk dies, the alarm fires, everyone gathers around. Big failures get fixed because they insist on it.

Small discrepancies are where the real information hides, for a simple reason: anything operating quietly in your environment — a slow hardware failure, a misconfiguration, a person who doesn't want to be noticed — is by definition producing only small evidence. The adversary's entire job is to stay under the threshold where you'd care. So the stuff under the threshold is exactly where they live. A 75-cent error is what a careful intruder looks like from the outside. So is a log that's ten lines shorter than yesterday's, a backup that runs four minutes longer than it should, a service that restarted once at 3:11 AM with no entry explaining why, free disk space that went up.

What the rule actually says

It does not say chase everything. Nobody has that kind of time, and a person who treats every wobble as an intrusion burns out by Thursday. The rule has three parts:

A discrepancy between two systems that should agree is never "rounding" until you know the mechanism. "Rounding" is a specific, checkable claim. So is "clock drift," so is "caching." Any of them might be the answer. The failure is accepting one as the answer without checking, because that word — surely — is where investigations go to die.
Pull the thread until you can name the mechanism, then stop. The goal isn't paranoia, it's an explanation with a moving part in it. "The two counters disagree because one counts retries and one doesn't" — done, write it down, go to lunch. Stoll's error stayed interesting precisely because every mundane mechanism he tested failed to explain it.
Write down the boring resolutions. This is the part everyone skips. The value of chasing small discrepancies compounds only if the explanations accumulate. A file of "this 4-minute backup wobble is the monthly full" entries is what lets future-you spend ten seconds on the known wobbles and full attention on the new one.

The bench version

I learned this with a screwdriver before I ever read it in a book. The machine that "sometimes" doesn't boot has a cracked solder joint, and sometimes is the diagnostic: it means thermal, it means mechanical, it means a connection that works in one position and not another. The customer's vague little complaint — "it's just slightly weird, probably nothing" — was the most reliable fault report I ever got. People apologized for reporting the exact symptom that solved the case.

It's the same instinct at any scale. The intermittent is not the enemy of diagnosis; the intermittent is the diagnosis, compressed. Same for the slightly-off. A system that is 100% wrong is broken. A system that is 0.3% wrong is telling you something, and there are only so many mechanisms in the world that produce exactly 0.3%.

Seventy-five cents. The cheapest security tool ever deployed was one person's refusal to say "close enough."

leadline / TG-8191

index · last touched 2026-05-09