The delete that wasn't there
Two filesystems spell "this file is deleted" differently. We read one of those spellings straight into a branch that discarded it, and nothing errored — the next restore resurrected every file the agent had removed.
An overlay filesystem has to record deletions somehow. The lower layer is read-only, so removing a file that exists down there cannot actually remove anything — it has to leave a marker in the upper layer saying “stop looking.”
There are two conventions for that marker, and they could not be less alike:
| AUFS (what we write) | kernel overlayfs | |
|---|---|---|
| delete | a regular file named .wh.<victim> | a character device, major 0 minor 0, named <victim> |
| opaque dir | .wh..wh..opq inside it | an overlay.opaque xattr on it |
We write the AUFS dialect deliberately: it keeps mknod out of the FUSE
server and it does not block a macOS port. But once we added a kernel-overlay
backend, the snapshot store had to read both, because an upper layer could
now arrive from either.
The bug
Our tree builder had a perfectly sensible arm that skipped device nodes.
Snapshots record files, directories and symlinks; a snapshot has no business
serialising a /dev entry that wandered into a project directory.
That arm ran before marker classification.
A kernel whiteout is a device node. So every delete recorded by a kernel-overlayfs upper hit the skip arm and vanished. No error, no warning. The resulting tree held exactly the files that had survived, which is indistinguishable from a correct snapshot of a directory that never contained the deleted ones — and the next restore duly resurrected everything the agent had removed.
The fix is one line of ordering: classify first, dispatch on file type second. The cost of getting it wrong is silent data resurrection, which is about the worst failure mode a snapshot system has.
Why the tests would not have caught it
The unit tests build whiteout markers by hand. They encode our model of the kernel’s behaviour, so they agree with that model whether or not it is right. They would have passed.
The only thing that catches this is mounting a real overlayfs and asking the
kernel to produce the markers itself. Those tests need /dev/fuse and
unprivileged user namespaces, so they are #[ignore]d — and they run in CI
behind a guard that asserts a non-zero number of them actually executed,
because cargo test -- --ignored exits 0 when it selects nothing. A
renamed file would otherwise turn the whole step into a green no-op.
Three things this pinned down
Tree objects stay dialect-free. Whiteout and opaque-directory entries are symbolic, and an opaque xattr is normalised into the same synthetic entry the file-based marker produces. That is precisely what lets a snapshot taken over one overlay restore onto the other. The moment a tree records which dialect wrote it, snapshots stop being portable between backends.
The two dialects cannot be confused, which is why reading both needs no mode flag. One is a regular file with a reserved prefix; the other is a device node. Classification accepts both unambiguously, and only the writer has to pick.
metacopy is refused rather than tolerated. A metacopy upper holds a
file’s metadata while its data is still down in lower. Snapshotting one
records an empty file — silently truncating the user’s work. We never pass
that mount option, and we check for an upper that arrived with it anyway.
An aside on privilege
Character device 0:0 is the one device number mknod(2) will create without
CAP_MKNOD; the kernel special-cases it as WHITEOUT_DEV. So restoring a
kernel-dialect upper needs no privilege at all.
Do not generalise that into “we can create device nodes.” Try 1:3 or 8:0 and
you get EPERM. It is exactly one number, special-cased for exactly this
purpose.