We ran grsecurity and PaX on everything we cared about: trusted path execution, a real chroot jail, address-space randomisation before mainline had it, non-executable memory enforced properly, role-based access a compromised service couldn’t talk its way around. Then the public patches stopped, and while mainline slowly took in a lot of the ideas, it didn’t take all of them and didn’t take them with the same posture. This is the field note on rebuilding the ones we missed — clean-room against the current LSM framework, not lifted from anyone’s patches — and, more importantly, on what it is honestly for. The latest testing round was carried out with Claude (Mythos 5.1) under the Cyber Defender program.
“Un-exploitable” is a fairy tale
Let’s get the honest part out of the way first, because the goal is easy to oversell. You cannot make a general-purpose kernel un-exploitable; anyone who tells you otherwise is selling something. A real exploit is a chain: leak something to defeat randomisation, get code or a data-only primitive to run, escalate, then persist. Every link is a place the attacker can be made to spend more — more bugs, more reliability, more noise. That’s the whole game. Not a wall; a toll booth at every corner, each one raising the price and the chance of getting caught.
The question isn’t “can this be exploited.” It’s “how expensive is each stage, and what does the attacker have to give up to pass it.”
Walls at each stage
Finding their footing. The classic first move is leaking a kernel pointer to beat KASLR. The cheapest defence is to not have the surface at all: the symbol table, loadable modules, and the raw-memory interfaces are compiled out, and pointers that do reach userspace are hashed. What’s left of the restricted interfaces gets a tripwire — an unprivileged process that keeps probing the files a kernel exploit reads for layout is treated as reconnaissance, counted, and after a threshold dropped into containment. Removing a surface beats faking it.
Getting something to run. This is where the old grsec muscle memory lives. Trusted Path Execution refuses to exec anything out of a directory an unprivileged user could have written. On top of it, a trusted-origin lock only allows execution — execve, and mapping or re-marking a file executable — from an explicit allowlist of trusted directories; a root-owned binary dropped somewhere unexpected still will not run, and that applies to root too. W^X refuses writable-and-executable memory. And underneath it all, the mainline blockers are on: kernel control-flow integrity (kCFI), structure-layout randomisation, stack protection, hardened usercopy, and the CPU-level mitigations available on the test platform. Each one turns a category of “and now I run my shellcode” into “and now I need a different, harder technique.”
Abusing the kernel’s own helpers. One of the prettiest local-root tricks is to point a kernel-spawned helper (the core-dump handler, the module loader) at your own program and trigger it — the kernel runs it for you, as root. A Helper Gate vets every user-mode helper against an allowlist keyed by a boot-random secret; in the focused attack-range proof, redirecting one of those paths to an attacker binary was denied. There’s more in the same spirit — chroot hardening, System-V IPC and symlink restrictions, a brute-force throttle. None is novel on its own. The point is the stack: a chain now has to survive all of them.
Not the end of the world
Here’s the part we care about most, and the reason for the title. Assume the attacker wins anyway and lands root in userspace — a good enough bug, a bad enough day. In most setups that’s the end of the story: root turns the defences off and settles in. We wanted the story to keep going. One honest caveat up front, because it matters: everything in this section is about a userspace root compromise. An attacker who holds an arbitrary kernel-write primitive is a different and harder problem — these walls raise its cost and make it leave traces, but we do not claim to stop it.
Three things change what userspace root can do:
- Protections can’t be silently switched off during the current boot. Once the maintenance lock is armed, turning a protection down is gated behind a signed maintenance window. The kernel will only relax a runtime control if it’s shown a signature, over a fresh single-use challenge it just issued, that verifies against an offline key. A root attacker can tighten things all they like; to loosen a locked runtime control they need a key they do not have. The signed challenge cannot be replayed; the resulting short window permits maintenance writes and then re-locks itself. The explicit boot-only recovery switch is a separate boundary: a deployment must protect its bootloader, command line and early userspace if it wants the same guarantee across reboot.
- The attacker can be contained and expunged. A compromised or suspicious process can be dropped into a cage — a containment mode that strips its ability to open sockets, exec, trace other processes, signal out, or use privilege, and the cage is inherited by everything it forks, so it can’t spawn its way free. At the top rung the task is frozen outright. The cage can be tripped by hand or by one of the tripwires firing.
- Triage is faster because the log is tamper-evident inside the stated threat model. Cage and tripwire events are written to a small ring whose records are chained with a keyed hash, so a root attacker in userspace cannot quietly rewrite or snip out an interior record during the current boot without breaking the chain. The key lives in kernel memory, so forging that chain undetected takes arbitrary kernel read-and-write or equivalent control over its output, not just userspace root. The ring can still be flooded or wrapped, and a reboot can destroy unexported history; useful incident evidence therefore has to be exported and anchored somewhere the compromised machine cannot rewrite.
And the newest piece ties the escalation stage off: credential integrity-sealing. The live credentials — the uid/gid set, the capability sets — are sealed with a boot-keyed hash that is re-sealed at every legitimate change (exec, setuid, capset, fork). A direct overwrite of the credential struct, the final step of a whole family of local-root exploits, goes through none of those paths, so when the forged root is used, the seal no longer matches and the forgery is caught — even though the in-memory uid now reads zero. On our deliberately-vulnerable test kernel, the same overwrite that reads CRED_PWNED_ROOT with sealing off reads CRED_BLOCKED with it on. (The honest limit, again: an attacker who can both overwrite credentials and read the in-kernel seal key could forge it. It raises cost; it is not a wall.)
None of that un-rings the bell of a root compromise. It changes the tested aftermath from game over to contain, preserve evidence, expunge — and it buys the defender the one thing incident response never has enough of, which is time and a record userspace root cannot quietly rewrite in place.
The tiny harness was too polite
The first suite was built to make each feature answer one question at a time. That worked, and it was also too polite. A separate harness hit the same controls through different paths and found six holes after the feature suite was green: W^X remembered only a mapping’s current permissions, not whether it had been writable earlier, and five maintenance setters could still loosen policy without a signed window. The green suite was not wrong. It was incomplete.
Those six holes now have regression tests. On the current secure-default build, the stack passes 19 of 19 headless suites: 771 assertions, zero failures. An independent harness passes 72 of 72. The signed-maintenance path alone carries 615 assertions because every writable control has to distinguish tightening, which stays free, from relaxation, which needs the offline key. In a live root-level sweep with the lock armed, 44 weakening writes were refused at the write itself, three were unchanged no-ops, and zero policy relaxations landed.
Then we gave it a real userspace
The initramfs tests still omitted the thing most kernels spend their lives under: a normal distribution. We booted the hardened stack under systemd and did boring administrator work — log in, use sudo, restart services, make an HTTPS request, install a package. That found problems the small guests did not: RBAC and procfs rules that were too broad for service accounts, credential sealing mistaking a legitimate securebits transition for tampering, a setuid-root helper judged by the wrong uid, a procfs path match that caught ordinary network sysctls, and a chroot check that could sleep from a hook where sleeping is forbidden.
Each one became a patch and a regression test. Service access is now granted with immutable boot-time rules for one identity, one operation and one path instead of exempting an entire account. The hard policies now start hard: W^X, trusted-origin enforcement, credential sealing, the leak tripwire, cage mode and the Helper Gate are on at boot, with an explicit boot-only relaxation path for provisioning and recovery.
Then we made the compatibility test meaner. A package with no maintainer script installed cleanly; packages that execute post-install scripts did not. Neither did scheduled jobs or legacy init scripts, because they live outside the original trusted-origin set. That was the policy doing exactly what we asked and still making the system impractical. We measured the minimum extra origins and per-service reads needed for package management, scheduled work and ordinary administration, applied only those, and reran the sweep. Logins, sudo, service restarts, HTTPS, package installs with maintainer scripts, scheduled jobs and log rotation all worked. A new binary in a home directory remained denied by design.
Widening a trusted-origin list creates a new promise: those directories now matter more. We paired the measured allowlist with an integrity database and tested it by adding and changing package, init and scheduled-job files. The check reported three additions and two modifications across those tests. That is after-the-fact detection, not prevention, and an attacker with userspace root can tamper with an on-host database. The useful deployment keeps the trust anchor and known-good copy somewhere the compromised machine cannot rewrite.
Testing the tests
The debug build adds KASAN, UBSan, lockdep, KCOV and atomic-sleep checks. It found the chroot bug during an ordinary boot, before the fuzzer had anything clever to say. The first parser-fuzz harness then taught us a second lesson: it could cage itself, after which every later write was denied and a “clean” result meant nothing. Those results were thrown out. The corrected harness reports accepted writes per control, detects containment, and runs irreversible controls last.
The corrected control-file passes drove 20,000 structured mutations across the writable policy surface and 1,615,903 concurrent writes through setter and hot paths, with no kernel finding. Inputs included boundary lengths, malformed lists, split and positional writes, scatter writes and races. The point was not merely that the kernel stayed up: every control had to accept some writes, readbacks had to remain sane, and the harness had to prove it had not caged itself.
The signed-maintenance lane now mutates real certificates and signed unlock messages instead of spraying random bytes at a parser that rejects them immediately. An independent OpenSSL check separates parser leniency from an actual authentication failure. Across two final campaigns, 3,000 structured mutations produced four parser-lenient cases in header bytes that are not covered by the signature and zero unauthorized maintenance windows. We did not find a changed or untrusted signature that opened one. That is “no demonstrated bypass in this campaign,” not a proof that none exists.
For W^X and trusted-origin, crash detection is the wrong oracle. A kernel can remain perfectly alive while allowing a forbidden executable mapping. The invariant fuzzer checks the process map after every randomized memory operation: no writable-and-executable mapping, no executable mapping that diverged from its backing file during the sequence, and no executable content from an untrusted origin. Across 900 corrected randomized sequences — 54,000 memory operations — in user and root contexts, with origin enforcement both on and relaxed, it found no oracle violation. Some execute-only anonymous mappings could not be read back, so they remain inconclusive rather than quietly counted as passes. W^X tracks mapping history, not a file’s entire content history: if origin enforcement is relaxed, writing a temporary file first and mapping it executable later is outside the W^X guarantee.
The origin harness separately tried 30 execution routes: ordinary paths, symlinks, file-descriptor paths, anonymous files, shebangs, the dynamic loader, preloads and loaded libraries. It reproduced the known limit: a trusted interpreter can still consume an untrusted script as data. Several descriptor routes also failed before policy with ordinary file errors, so “30 cases” is coverage of attempted routes, not 30 direct enforcement proofs.
Finally, two completed Syzkaller campaigns executed more than 2.25 million generated programs against the instrumented kernel without a reported crash. Syzkaller runs with W^X softened and trusted-origin relaxed so its own executor can live; it is a crash, race and concurrency campaign, not an enforcement proof for those two policies. A more deliberately seeded campaign started next; because it was unfinished at this update, it is not counted here.
That is progress, not closure. The longer seeded Syzkaller campaign is unfinished. Origin/path races, resource-abuse tests, stronger execute-only memory observability, reproducible-build work and a written threat-model pass remain open. The post stays marked OPEN for the same reason the first green suite needed a second harness: a green number is only as strong as the question and oracle that produced it.
What’s proven, and what isn’t
Honesty, because the whole point is to be trustworthy about limits. Every shipped feature above has a focused headless-qemu proof, built out-of-tree from a clean kernel plus a numbered patch series, with a baseline-versus-feature comparison — the attack path works with the feature off and is blocked, contained, or logged with it on. The current non-range suite runs green. These are functional proofs in VMs, not 80 percent code coverage, not a formal guarantee and not a field pentest. We are not claiming the result is un-exploitable: each wall raises cost, a determined attacker with the right primitive may still clear one, and the design bet is that they must clear many and leave evidence doing it. An arbitrary kernel read-and-write primitive can recover or rewrite secrets these defenses keep in kernel memory. A trusted interpreter can still read an untrusted script. And it’s a clean-room reimplementation — the ideas are a love letter to grsecurity and PaX; the code is written fresh against the current LSM framework.
We missed these features for twenty years. It turns out the good ones port forward cleanly, and the most interesting one isn’t any single wall — it’s that a compromise stops being the end of the world.
← Back to lab notes