Results we took back
A five-run sample told us our new probe made startup cheaper. It doesn't. Three claims we published and then had to withdraw, and what each one taught us about the sampling that produced it.
A result that flatters the change you just made is the one nobody else will check. So it should be the one you check hardest. We have not been reliably good at this, and the three below are the evidence.
1. “The probe pays for itself at startup”
agentfs picks its mount backend by attempting a real overlayfs mount in a
throwaway namespace. That probe costs something on every invocation, and we
wanted to know how much.
The first measurement was one 5-run sample on a one-file project: auto at
15 ms/run against fuse at 19 ms. We reported that the new default was
cheaper to start, attached a plausible story about warm caches, and quoted
the finding in a commit message as though it had a direction.
Three alternating rounds of twenty runs reverse it:
| fuse | kernel | |
|---|---|---|
| round 1 | 7.9 ms | 10.4 ms |
| round 2 | 7.5 ms | 10.3 ms |
| round 3 | 7.4 ms | 9.9 ms |
The probe adds ~2.6 ms to every run. It does not pay for itself at startup. It pays for itself a second later, on the first real filesystem work, which is a fine thing for it to do and is what we should have said.
What went wrong is not subtle: n=5, no interleaving, and a result that happened to agree with the change under test. Note also that the absolute numbers moved by more than half — 15 ms became 7.4 ms — because the un-interleaved sample was mostly measuring cache state, not the probe.
Rule. A benchmark that isn’t interleaved is not a benchmark. It is an anecdote about the order you ran things in.
2. “A stale overlay workdir breaks the next mount”
Every overlay mount leaves workdir/work behind at mode 000. We had
cleanup code for it, and a comment in four places explaining that the next
mount would fail without it.
While removing something adjacent we deleted the cleanup and re-ran
a_second_run_still_mounts. It passed. Testing it directly: overlayfs mounts
perfectly happily over a stale, non-empty workdir, and cleans up its own
work subdirectory as it goes.
The cleanup is still correct and still there — for a completely different
reason. Mode 000 means the user cannot rm -rf their own project directory
afterwards, which is a real bug affecting a real person. We had shipped the
right code with the wrong justification, which meant every future reader would
reason from a false premise about the kernel.
Rule. A comment asserting that something breaks is a testable claim. If no test covers it, it is folklore, and folklore in a codebase compounds.
3. A retraction that was itself wrong
The most uncomfortable one. We had claimed write dominated our FUSE overhead
at 74%, then retracted it after samples appeared to refute it, then
reinstated it once the
cause was understood.
The refuting samples were taken during an experimental spike that had already
enabled a different FUSE protocol version. They were measuring an
already-fixed filesystem. The tell was sitting in the data the whole time:
every other operation count in those runs was exactly half a clean build’s,
while write alone differed by 12×. One changed variable, read as one bad
sample.
Rule. A spike that alters a build flag contaminates every measurement taken while it is on, not only the one under test. We wrote this one up in full rather than quietly correcting the file, because the failure mode — discarding a correct result on the strength of a contaminated sample — is much harder to notice than the ordinary kind.
Why these are on the website
Because the alternative is a benchmarks page where everything worked. We would rather publish the numbers we do not stand behind next to the ones we do, and be the sort of lab whose results you can subtract from.