Every entry in this programme has ended at the same wall. The binary ladder put it plainly — a frozen small model reads reliably and authors unreliably, so keep it on reads — and then Phase 2 found the sharper version: on a twenty-rung chain through noisy, decoy-filled output, the single hardest thing the model does is read the right value out of plausible wrong ones. We fixed that once already, by asking the right question and validating the answer at the moment it’s born instead of two rungs later. But that fix was about where the guard sits. This experiment asks a different question: does the shape of the read prompt itself change how well a small model reads?
The idea: a sealed evidence packet, not a terse prompt
The current harness read is about the most minimal thing you could send: a one-line question, a /no_think tag, and a raw slice of tool output pasted underneath. It works, mostly, because the harness catches what it misses. The alternative — taken from AaronCTI’s OSINTClaw Part 2, whose controlled-pivot stage hands every model the same sealed evidence with bounded budgets and provenance — is to stop sending a loose question over a transcript and start sending a bounded evidence packet: the question, a scoped span of just the relevant evidence, the validated facts already on the blackboard, and an explicit allowed-answer pattern, all framed as one self-contained problem with one required output.
The bet is narrow and worth stating precisely, because it is easy to oversell. Bounded evidence cannot make a small model author an exploit, crack a hash, write a pivot, or plan a campaign — those stay harness-owned, as every prior entry here argued. The only claim on trial is that the read itself — pulling one correct value out of noisy evidence — gets more reliable when the question arrives sealed instead of loose.
Doing it honestly: preregister, then measure
The trap with a test like this is grading your own homework after the fact, so the whole thing was preregistered before a single run. Model, lab, seeds, decoding, retry budget, and task chain all held constant; only the prompt shape varies, across five conditions — current (today’s terse prompt), scoped_prose, structured_full, scoped_structured, and strict — which cleanly separate scoping the evidence from structuring it.
Three commitments made before measuring, each one there to stop a flattering misread:
- Attribute the ceiling first. Before the A/B, bucket every current failure by cause — empty read, decoy read, transform error, or a boundary the harness owns (authoring, lab plumbing). Only the read-attributable subset is fixable by a prompt packet, so that subset is the maximum lift on offer. If the failures were mostly authoring or lab noise, a flat result would be the honest answer, not a broken experiment.
- Count the prompt length. A bounded packet that wins by being three times longer hasn’t won — the point is to beat the terse prompt at equal or smaller size. Median read-prompt characters is a first-class metric, not a footnote.
- Score strict rejections against an oracle, without logging secrets. The strict mode refuses answers that don’t match the evidence; a “false reject” only counts if the rejected candidate actually matched the ground-truth oracle — recorded as a boolean, never by writing the secret value into the results.
The result
Twelve seeds, five modes, sixty valid runs, qwen3-4b, thinking off, on the same noisy chained lab from Phase 2.
mode objective reads parsed fallbacks median prompt capture decoys current 12 / 12 16 / 36 20 437 chars — scoped_prose 12 / 12 36 / 36 0 656 chars — structured_full 12 / 12 36 / 36 0 930 chars — scoped_structured 10 / 12 36 / 36 0 984 chars 2 strict 10 / 12 36 / 36 0 984 chars 2 (both correctly rejected)
Read the two number columns against each other and the whole story is there. On the objective — did the campaign reach root and get independently verified — current already scores a perfect 12 of 12. There is no room above it to win. But look at how it got there: of 36 model read calls, only 16 parsed cleanly. The other 20 fell through to a deterministic fallback, and the validation loop re-read until the chain survived. The model was misreading on more than half its calls, and the harness was quietly absorbing every one.
Every bounded mode erases that. 36 of 36 reads parse, zero fallbacks — a +55.6 percentage-point jump in read success. The sealed packet does exactly what it claimed: the model reads the right value directly, instead of the harness rescuing a bad read after the fact.
Bounded evidence fixed the read and left the score untouched — not because it did nothing, but because the harness’s own safety net was already hiding the exact failures the packet prevents. The win is real; it just lands on the read metric, not the scoreboard.
The cost is length. Every bounded packet is bigger than the terse prompt — scoped_prose at 656 characters is the smallest of them, still 219 over current’s 437. Structure on top of scoping (the structured_full and scoped_structured modes) only added more characters without buying any read accuracy the prose packet didn’t already have. And the two heaviest modes actually regressed the objective to 10 of 12: both losses were capture-stage decoys, where the model produced a flag-shaped answer that didn’t match the oracle. strict caught and rejected both of those correctly, with zero false rejects — it behaved exactly as a diagnostic should, it just didn’t turn a rejection into a completion.
What it actually means
The decision, under the preregistered rule, is not to promote bounded packets into the engine as the default from this matrix — and the reason is the interesting part. This CTF has a deterministic oracle and a fallback path, so a misread never costs a root; the harness always has a second way to get the value. That makes it the wrong place to prove a read fix pays off, because the payoff is invisible: the score was already saturated. The read-reliability win is genuine and large, but where it would actually change an outcome is a setting with no fallback oracle — a real target, where a misread is just wrong and stays wrong.
So scoped_prose comes out of this as the prototype, not the promotion: it keeps the full read win at the smallest bounded size. The follow-up running now is a compact_scoped_prose pass whose whole job is to hold 36/36 reads while pushing the median prompt back down to current’s 437 characters — win on reliability and size, or it doesn’t ship. After that, the real test: a controlled pivot fixture where root-rate fallback can’t hide a model-read failure, so the read metric and the objective finally point the same way.
The follow-up: smaller, and still every read
The main run left one question hanging. scoped_prose fixed every read, but at 656 characters — longer than the 437 it was beating. Was the reliability paying for length? So the follow-up asked for the same result at less: a compact_scoped_prose packet, tuned only on its answer line. The first attempt over-trimmed — a cryptic A: exact <pattern> instruction — and lost the win back to 24/36. Spelling the instruction out plainly (“return only the exact value matching the pattern”) brought it back. Same qwen3-4b, same twelve noisy seeds:
mode objective reads parsed fallbacks median prompt current 12 / 12 16 / 36 20 437 chars scoped_prose 12 / 12 36 / 36 0 656 chars compact_scoped_prose 12 / 12 36 / 36 0 180 chars
Every read still lands, zero fallbacks — now at 180 characters, comfortably under half the terse prompt it beats (max 349 vs 640). The read win was never paying for length; it was paying for a clearer question. Which is the same lesson from the very first Phase 2 entry, one level down: the model was fine, the prompt was the variable.
Fix the read for a fifth the fallbacks, at less than half the words. The bounded packet isn’t a bigger prompt that works better — once it’s scoped and the answer line is plain, it’s a smaller one that works better.
So the renderer graduated from experiment to a shared harness component. It lives in agent_runtime/evidence.py, carries the mode table, and plugs into the playbook read path one call at a time; a job opts in per run, and the default stays the terse prompt byte-for-byte. It is deliberately not the default yet, for the same reason the main run couldn’t justify promotion: a CTF whose score is saturated by fallback can’t see a read fix. The last gate is a controlled-pivot fixture with no fallback oracle — where a misread finally costs something — and that’s running now.
Two models, two lanes, one board
This experiment was run by two different frontier models working in parallel, which is worth documenting because the setup is part of the point. Codex (GPT-5.5) owned the Phase 2 research lane — the evidence-packet builder, the A/B runner, the preregistration, and exclusive use of the lab and the on-device model while the matrix ran. Claude (Opus 4.8) owned the harness lane — a reusable playbook engine that drives small models with direct per-step commands and one pointed read per step, plus a renderer seam built so that a winning packet mode can drop into the general playbooks with a single line, no logic duplicated. The two never edited the same file; coordination ran over a plaintext board on the lab network, one claim per lane, hand-offs and blockers posted as they happened.
The collaboration caught things a single agent would have shipped. Codex found its own false-reject accounting was too broad mid-run and stopped to fix the metric before trusting the strict numbers. Claude traced the packet pipeline and found the “one-line promotion” it had been assuming was missing an input — the allowed-answer pattern the strict and scoped modes depend on — and widened the seam before anything was wired, so the eventual hand-off stays clean. Neither catch was in either agent’s own lane by accident; each came from one model reading the other’s work. And the hand-off held: when the compact renderer landed in agent_runtime/, it dropped into the pre-widened read seam and the playbook-dispatch opt-in with no rework — two lanes meeting exactly where they had agreed to, on a plaintext board, without either editing the other’s files.
Preregister the rule, attribute the ceiling before you measure, and let the other model check your pipeline. Two of the three sharpest corrections this round came not from a result but from that discipline — and one of them stopped a flat A/B from being read as a failure.
Credit
The bounded evidence-context method tested here is AaronCTI’s, from OSINTClaw Part 2 — specifically its controlled-pivot stage, where every model is handed the same sealed evidence under bounded budgets with provenance and scoring that separates a bad answer from an invalid run. We transferred that framing onto our own blackboard-and-validation harness and measured it directly. The reliability finding is his idea holding up under a preregistered test; the “it fixed the read and the safety net hid it” twist is what our particular harness added to the picture.
Part of the same programme — the harness this builds on, and the “the read is the wall” finding it starts from, are in the previous entry: Phase 2: The Chained CTF.
← Back to lab notes