pid1 Labs

Results we took back

2026-08-09T00:00:00+00:00

A result that flatters the change you just made is the one nobody else will check. So it should be the one you check hardest. We have not been reliably good at this, and the three below are the evidence.

1. “The probe pays for itself at startup”

agentfs picks its mount backend by attempting a real overlayfs mount in a throwaway namespace. That probe costs something on every invocation, and we wanted to know how much.

The first measurement was one 5-run sample on a one-file project: auto at 15 ms/run against fuse at 19 ms. We reported that the new default was cheaper to start, attached a plausible story about warm caches, and quoted the finding in a commit message as though it had a direction.

Three alternating rounds of twenty runs reverse it:

	fuse	kernel
round 1	7.9 ms	10.4 ms
round 2	7.5 ms	10.3 ms
round 3	7.4 ms	9.9 ms

The probe adds ~2.6 ms to every run. It does not pay for itself at startup. It pays for itself a second later, on the first real filesystem work, which is a fine thing for it to do and is what we should have said.

What went wrong is not subtle: n=5, no interleaving, and a result that happened to agree with the change under test. Note also that the absolute numbers moved by more than half — 15 ms became 7.4 ms — because the un-interleaved sample was mostly measuring cache state, not the probe.

Rule. A benchmark that isn’t interleaved is not a benchmark. It is an anecdote about the order you ran things in.

2. “A stale overlay workdir breaks the next mount”

Every overlay mount leaves workdir/work behind at mode 000. We had cleanup code for it, and a comment in four places explaining that the next mount would fail without it.

While removing something adjacent we deleted the cleanup and re-ran a_second_run_still_mounts. It passed. Testing it directly: overlayfs mounts perfectly happily over a stale, non-empty workdir, and cleans up its own work subdirectory as it goes.

The cleanup is still correct and still there — for a completely different reason. Mode 000 means the user cannot rm -rf their own project directory afterwards, which is a real bug affecting a real person. We had shipped the right code with the wrong justification, which meant every future reader would reason from a false premise about the kernel.

Rule. A comment asserting that something breaks is a testable claim. If no test covers it, it is folklore, and folklore in a codebase compounds.

3. A retraction that was itself wrong

The most uncomfortable one. We had claimed write dominated our FUSE overhead at 74%, then retracted it after samples appeared to refute it, then reinstated it once the cause was understood.

The refuting samples were taken during an experimental spike that had already enabled a different FUSE protocol version. They were measuring an already-fixed filesystem. The tell was sitting in the data the whole time: every other operation count in those runs was exactly half a clean build’s, while write alone differed by 12×. One changed variable, read as one bad sample.

Rule. A spike that alters a build flag contaminates every measurement taken while it is on, not only the one under test. We wrote this one up in full rather than quietly correcting the file, because the failure mode — discarding a correct result on the strength of a contaminated sample — is much harder to notice than the ordinary kind.

Why these are on the website

Because the alternative is a benchmarks page where everything worked. We would rather publish the numbers we do not stand behind next to the ones we do, and be the sort of lab whose results you can subtract from.

A hash with a hole in it

2026-08-08T00:00:00+00:00

agentfs snapshots an agent’s work during a run, not only at the end. The property it is trying to provide is narrow and worth stating precisely: the agent cannot prevent a checkpoint.

On our FUSE backend, knowing when to snapshot is free. The server sees every mutating operation, so it bumps an AtomicU64 and the watcher reads it. Exact, no tree walk, no inotify registration, and no false positives from our own snapshot activity.

The kernel-overlayfs backend has no server. Nobody is in the path to bump anything. So the question “did anything change?” has to be answered by looking, and looking on every 500 ms tick has to be cheap — which means stat, not content hashing.

The tempting minimal digest

Walk the upper layer, digest each entry’s inode metadata, compare the digest to last tick’s. The obvious fields are inode number, size, and mtime. That is what most “has this changed?” caches use, and for a build system it is fine.

Here it has a hole. Consider an agent that wants to edit a file without being snapshotted:

Write the new contents in place, at exactly the same length.
utimensat the mtime back to what it was.

Inode number: unchanged. Size: unchanged. Mtime: restored. The digest matches, the watcher concludes nothing happened, and it does not walk.

That last clause is what makes this a real failure rather than a latency bug. A missed bump on the FUSE counter delays a checkpoint; the next mutation bumps it again and the walk records everything. Here the walk is gated on the probe, so a digest that fails to notice a change produces a suppressed checkpoint, not a late one. The property at the top of this note quietly stops holding, and nothing reports an error.

`st_ctime` closes it

Add ctime to the digest and the trick stops working. Ctime moves on any inode change — including the utimensat intended to cover the tracks — and userspace cannot set it. Forging it requires CAP_SYS_TIME and a clock change, which is a different and much louder kind of attack.

The general shape is worth keeping: when a digest is the input to a decision about whether to look, every field you leave out is an action someone can take invisibly. Mtime is advisory metadata that the owner of a file is allowed to rewrite. Ctime is bookkeeping the kernel keeps for itself. Only one of those belongs in a security-relevant fingerprint.

What it costs

About 1% of an agent-shaped workload — twelve rounds of edit, build, git add, commit, run inside the sandbox:

	watcher off	watcher on
mean	10.177 s	10.293 s
spread	0.312 s	0.444 s

+1.1%, with the ranges overlapping — the off arm peaked above the on arm’s minimum, so at n=4 this is not separable from noise. We trust it because it matches the cost model rather than because the delta is convincing: a 10.2 s run at a 500 ms interval is ~20 probes, the walk measures 6.3 ms on a corpus with a full .git, so predicted 126 ms against 116 ms measured.

The scaling rule is tax ≈ walk_cost / 500 ms, which means what governs it is the number of non-ignored files, not the workload. The same walk is 0.5 ms on source alone, 6.3 ms with .git, and 80 ms — 16%, extremely visible — if target/ were not excluded.

So the ignore list is doing load-bearing performance work, and it is the first thing to check if this tax ever appears. It also has to be exactly the same predicate the snapshot uses. Watch a path the snapshot skips and every build fires a checkpoint that records nothing at all.

Container-ness is the wrong question

2026-08-07T00:00:00+00:00

agentfs gives an agent a writable view of your project without letting it touch the original. Until recently that was always a FUSE server we wrote: your files as a read-only lower layer, an upper layer catching every write, snapshots taken from the upper.

It works, and on the inner loop it costs a lot. An incremental cargo build through the sandbox ran 2.21× slower than the same build outside it. The kernel has had its own overlay filesystem for a decade. The question was when we could use it instead.

The obvious probe is wrong twice

The intuition is that overlayfs needs privilege, so it works on a host and not in a container — check /.dockerenv or /proc/1/cgroup and branch. Both halves of that are false.

Default Docker cannot mount overlayfs at all. No CAP_SYS_ADMIN, no mount. So container-ness is not sufficient to rule it out — it correctly predicts this case and nothing else.
A plain host can, with no container anywhere in sight. An unprivileged user namespace holds CAP_SYS_ADMIN over itself, and the kernel has allowed overlayfs inside one since 5.11. We already create a user namespace for every rootless run, so we already had the privilege we thought we lacked. This is a user-namespace feature, not a container feature.
--cap-add SYS_ADMIN is still not enough when upperdir is itself on overlayfs — which is exactly where a container’s ~/.agentfs lands by default. EINVAL. You may not stack an upper on an overlay.

That third case is the one that kills the heuristic outright. The capability is present, the container is privileged, and the mount still fails — because the constraint was never about privilege. It was about which filesystem the upper directory happens to sit on.

As a bonus, the detection itself has rotted: on cgroup v2, /proc/1/cgroup says 0::/ and nothing more.

So don’t ask. Try it.

The probe performs the real mount, in a throwaway namespace, against scratch directories inside the project directory — not /tmp. That last detail is the whole point: the constraint is a property of the filesystem the upper lives on, so probing somewhere else answers a different question convincingly.

If the mount succeeds, we tear it down and use the kernel backend. If it fails for any reason, we fall back to FUSE and say why.

What it bought

Measured end to end through a full agentfs run, so the numbers include namespace setup, the bind set and the mount:

task	native	fuse	kernel
cargo build (incremental)	0.824 s	1.819 s	0.839 s
grep -r	11.6 ms	77.1 ms	13.4 ms
git status (dirty)	8.3 ms	26.4 ms	9.6 ms
git add + commit	10.2 ms	50.1 ms	27.7 ms

The inner loop goes 2.21× → 1.02×: on the kernel backend, building inside the sandbox costs what building outside it costs. grep -r collapses the same way, 6.67× → 1.16×.

What is left is git add + commit at 2.72×, and it is structural rather than incidental. Git renames every object into its final place, and a rename across overlay layers is a copy-up. That is the one primitive where layering itself has a price.

Two things we had to get right

The mount happens inside the sandbox’s own mount namespace, not on the host. That is what makes it work without root, and it has a second effect worth more than the first: the workspace is invisible from outside. There is no mountpoint for a stray editor to write through, and nothing left to unmount if the sandbox dies.

-o userxattr, always — in both privilege modes. Left to itself the kernel uses trusted.overlay.* when privileged, and a later host-side restore running as an ordinary user cannot write those. One dialect everywhere beats matching the kernel’s default.

The cost of asking

The probe runs on every invocation and adds ~2.6 ms. It does not pay for itself at startup — we published the opposite claim first and had to take it back — it pays a second later, on the first real filesystem work.

One implementation detail that cost an afternoon: the probe forks from a process that is already multi-threaded, so everything the child touches has to be built before the fork. A malloc in the child of a threaded fork deadlocks if another thread happened to hold the allocator lock at fork time. The bug does not reproduce under a debugger, naturally.

36% from a version number

2026-08-06T00:00:00+00:00

We had spent two commits looking for the source of our FUSE overhead in the wrong place. Adding byte counters alongside the operation counters settled it immediately: one incremental build moved 295 MB in 56,634 writes — about 5.2 KB per call, against a transport that can carry far more per round trip.

The operation count was not high because the build writes a lot. It was high because every write was small.

fuser gates its FUSE protocol version behind cargo features and defaults to abi-7-9. Raising it to 7.23 opts into no new capabilities and changes nothing else, but the negotiated maximum write goes from ~5 KB to ~65 KB:

	time	writes	bytes
default (`abi-7-9`)	2.837 / 2.811 s	56,694 / 56,591	294.8 MB
`abi-7-23`	1.792 / 1.804 s	4,527 / 4,499	295.0 MB

Identical bytes, 12.5× fewer operations, 36% faster. End to end on the real workload the inner-loop overhead falls from +1.81 s to +1.10 s — 39% of it removed by a one-line dependency change.

We measured abi-7-31 too: 4,047 writes, no further change in time. Taking the lowest version that delivers the win means fewer protocol differences to reason about and a lower kernel floor — 7.23 is Linux 4.5.

The part worth keeping: we had been counting operations for two commits and the answer was in the bytes. An operation count tells you how often you crossed the boundary. It cannot tell you whether crossing it that often was necessary.

This also reinstated a claim we had previously retracted — the retraction was the mistake, and it is written up with the others.

The delete that wasn't there

2026-08-05T00:00:00+00:00

An overlay filesystem has to record deletions somehow. The lower layer is read-only, so removing a file that exists down there cannot actually remove anything — it has to leave a marker in the upper layer saying “stop looking.”

There are two conventions for that marker, and they could not be less alike:

	AUFS (what we write)	kernel overlayfs
delete	a regular file named `.wh.<victim>`	a character device, major 0 minor 0, named `<victim>`
opaque dir	`.wh..wh..opq` inside it	an `overlay.opaque` xattr on it

We write the AUFS dialect deliberately: it keeps mknod out of the FUSE server and it does not block a macOS port. But once we added a kernel-overlay backend, the snapshot store had to read both, because an upper layer could now arrive from either.

The bug

Our tree builder had a perfectly sensible arm that skipped device nodes. Snapshots record files, directories and symlinks; a snapshot has no business serialising a /dev entry that wandered into a project directory.

That arm ran before marker classification.

A kernel whiteout is a device node. So every delete recorded by a kernel-overlayfs upper hit the skip arm and vanished. No error, no warning. The resulting tree held exactly the files that had survived, which is indistinguishable from a correct snapshot of a directory that never contained the deleted ones — and the next restore duly resurrected everything the agent had removed.

The fix is one line of ordering: classify first, dispatch on file type second. The cost of getting it wrong is silent data resurrection, which is about the worst failure mode a snapshot system has.

Why the tests would not have caught it

The unit tests build whiteout markers by hand. They encode our model of the kernel’s behaviour, so they agree with that model whether or not it is right. They would have passed.

The only thing that catches this is mounting a real overlayfs and asking the kernel to produce the markers itself. Those tests need /dev/fuse and unprivileged user namespaces, so they are #[ignore]d — and they run in CI behind a guard that asserts a non-zero number of them actually executed, because cargo test -- --ignored exits 0 when it selects nothing. A renamed file would otherwise turn the whole step into a green no-op.

Three things this pinned down

Tree objects stay dialect-free. Whiteout and opaque-directory entries are symbolic, and an opaque xattr is normalised into the same synthetic entry the file-based marker produces. That is precisely what lets a snapshot taken over one overlay restore onto the other. The moment a tree records which dialect wrote it, snapshots stop being portable between backends.

The two dialects cannot be confused, which is why reading both needs no mode flag. One is a regular file with a reserved prefix; the other is a device node. Classification accepts both unambiguously, and only the writer has to pick.

metacopy is refused rather than tolerated. A metacopy upper holds a file’s metadata while its data is still down in lower. Snapshotting one records an empty file — silently truncating the user’s work. We never pass that mount option, and we check for an upper that arrived with it anyway.

An aside on privilege

Character device 0:0 is the one device number mknod(2) will create without CAP_MKNOD; the kernel special-cases it as WHITEOUT_DEV. So restoring a kernel-dialect upper needs no privilege at all.

Do not generalise that into “we can create device nodes.” Try 1:3 or 8:0 and you get EPERM. It is exactly one number, special-cased for exactly this purpose.

You cannot authenticate a process that shares your uid

2026-08-04T00:00:00+00:00

agentfs takes automatic snapshots while an agent works, so that a run which goes badly can be walked back. The obvious way to place them well is to let the harness say when: Claude Code fires a hook at the end of each turn, which is a natural boundary.

Which raises the natural next question. How do we know a checkpoint request came from the harness and not from the agent?

You don’t, and you can’t

They share a uid. They share a process tree. Whatever secret the harness could present — a token in a file, an environment variable, a nonce on a socket, a value in its own memory — the agent can read, because the kernel has no basis on which to distinguish them. Same uid means same access to /proc, same access to the filesystem, same everything. Any proof of identity is replayable by precisely the party you are trying to exclude.

This is the same dead end as trying to make a game client trustworthy, and it fails for the same reason: the thing being authenticated has full control over the machinery doing the authenticating.

We spent a while looking for a clever way around it. There isn’t one. Not a hard one — an impossible one.

Change the property instead

The move is to stop trying to build the check and pick a property that does not need it:

The trigger lives outside the sandbox and runs unconditionally. A watcher on the host samples for change, fires on quiet, and has a ceiling that fires regardless.
The in-sandbox channel is a hint that can only add a checkpoint, never suppress one.
Forging a hint is therefore a non-event. You get an extra snapshot.

The property we provide is “the agent cannot prevent a checkpoint” — deliberately not “only the harness can request one.” The second is unachievable. The first is what we actually needed, and it survives an agent that is actively hostile and knows exactly how the mechanism works.

This only works because snapshots are additive. Nothing destructive can ever go behind a forgeable signal. restore is not on that channel and must not be put there — an agent that can trigger a restore can destroy work, and no amount of care in the hint parser changes that.

The second-order problem

Checkpoints self-prune to a budget, which quietly reintroduces the attack from the other side. If eviction were naive, an agent could spam hints until every genuine idle checkpoint had been crowded out of the budget — defeating the mechanism through exactly the channel we left forgeable on purpose.

So eviction order is timed → hint → idle: agent-influenced entries go first. The forgeable channel can add noise; it cannot displace signal.

Two smaller consequences of the same reasoning. Hint labels are untrusted input that ends up in a commit message, so they are stripped of control characters and truncated. And the label never claims a hint came from the harness, because that is not a thing we can know — it says a hint arrived, which is all that is true.

The general shape

When two parties are indistinguishable to the kernel, stop looking for a credential and start looking for an asymmetry that already exists. Ours was position: the watcher is outside the boundary the agent is confined by, and that is a difference no forged message can erase.