Ask It Twice: The One Step a Frozen Model Cannot Take
The previous entry closed Phase I with a recommendation: stop trying to make learned updates reason on their own, and start treating them as detachable memory objects selected outside the reasoner. Thirty-one experiments later, that recommendation turned out to be right for a reason nobody had measured yet.
The result, in three numbers
The whole finding is one trace
The evaluation world is a synthetic catalogue: each body contains an element, and each element emits a radiation. Training states each link separately and never states the pair. Asking for the radiation of a body therefore requires joining two stored facts.
One pass. “What radiation should PLYURX-91 emit?” → NXAIRN-15 ✗
Two calls. Step one → NXAIRN-15. Then “What radiation does NXAIRN-15 emit?” → MVEIVN-49 ✓
Same adapter, same item, and the identical intermediate token in both cases. The only difference is whether that token is handed back as a query. The model computes the first step correctly and emits it — and cannot use its own output as the key for the next lookup inside a single forward pass.
That is not a knowledge failure, a capacity failure, or a retrieval failure. Recall over a held-out set spanning both links is 100%. The model knows the second fact perfectly well. It just cannot get from the first to the second without leaving the forward pass.
Every configuration we measured
Seven ways of asking the same questions. Chance is 0.1667.
The best and the worst rows in that table are the same adapter answering the same questions. Everything between them is how the question was issued.
Memory and context fail at opposite ends
Putting the facts in the prompt instead of the weights does not fail the same way. It fails at the first step: asked which element sits on a given body, with that exact sentence visible in its context, the model echoes the body back 36 times out of 48. Given a correct first step, its second step is 0.9167.
The adapter is the mirror image. Its first step is perfect and its second never happens. So the two storage methods are complementary failures, and the composition score that the whole field treats as a reasoning measurement turns out to be, in this setting, retrieval accuracy multiplied through two lookups. When we predicted composition from the two hops independently, the residual was −0.0209.
What transfers, and what does not
The most practically useful result was not about memory at all. We rebuilt everything in an unrelated subject area — certified operators and laboratory instruments instead of astronomical bodies and elements, no shared vocabulary, no shared sentence templates — and retrained from scratch.
Architecture transferred exactly
Recall, detachment, interference, the composition ceiling, and the two-call repair all reproduced with a difference of 0.0000 across all seven measured quantities. Even the failure mode replicated: the new adapter answers with the intermediate and stops, exactly as the first one did.
Prompt engineering transferred not at all
A prompt change worth +0.3958 in the first world was worth +0.0000 in the second. It had been repairing a quirk specific to that world, not improving reasoning. Two separate mechanisms, each worth about +0.15 on their own, both collapsed to nothing.
This also retired an assumption the project had been carrying since the beginning. What we had been calling a replication world was the same task with the names changed — its sentence templates were identical to the original's. It agreed with the original about everything, including the artifacts. Only a genuine change of subject matter tests anything.
Four times, the instrument was the thing at fault
Every threshold in this programme is committed to version control before the measurement that judges it, and failures are reported as failures. That discipline paid for itself in an unglamorous way: on four separate occasions, the thing that failed was our own test harness.
- An answer-format instruction — “reply with a single digit and nothing else” — was enough on its own to make the model answer 0 to “what is 2 plus 3?”. It had been hiding a capability completely, and the failure looked principled: it landed exactly on the majority-class baseline.
- A chaining harness was feeding the second question an entire sentence rather than the entity in it, which silently re-injected the subject.
- A scoring rule counted a refusal as correct because the expected name appeared inside it.
- A metric counted a restatement as an echo, so it read as maximal on the best-performing condition.
All four were caught by small pre-run smoke tests rather than by inspection of the results, which is now standing practice: any guard that reports a failure is first pointed at a question the model is known to answer. A guard that fails its own sanity check is measuring the instrument.
The experiments
| ID | Question | Outcome | Key figure |
|---|---|---|---|
| P / Q | Can a rule be installed the way a fact can? | Both stopped at their guards | 0 / 24 |
| R | Was the answer format hiding ability all along? | Programme cleared | +0.0209 |
| S | Is composition reasoning, or retrieval? | Retrieval, multiplied through | −0.0209 |
| T | Why does a reasoning scaffold help? | Two separable mechanisms | +0.1458 ea. |
| T-rep | Do those mechanisms survive a new subject? | No — neither works alone | +0.0000 |
| U | Memory versus prompt, compared fairly | Opposite failure points | 0.0000 |
| V | Does memory survive a new subject? | Exact, all seven quantities | Δ 0.0000 |
| W | Can the chaining be done outside the model? | Complete recovery | 1.0000 |
| W-rep | Does that survive a new subject too? | Exact | 1.0000 |
| X | Can the model plan the two steps itself? | Running | 0 / 48 so far |
| Y | Does it hold at three hops instead of two? | Running | — |
What we are not claiming
The two-call harness is told how to split the question. We supply the second step's wording, so the result shows that if a two-hop question is decomposed, memory answers each part perfectly — not that the system decomposes questions on its own. The composition is performed by the harness, not by the model. Asked directly to name its own next lookup, the model has so far scored zero out of forty-eight: it answers the sub-question instead of stating it.
Two calls is also twice the inference, the model is a small one, and everything above is a two-hop chain in a synthetic world of twelve chains. Depth is being measured now, and the risk we registered in advance is a real one: the third hop passes through a link that several chains share, where the earlier hops were always unique. If chaining only works while the intermediate is unique, that is a genuine boundary on the architecture, and it would be the headline rather than a footnote.
Why this is the interesting outcome
The goal was permanent, pluggable knowledge without retraining: something you can attach to a frozen model, use, and detach. Storage was solved early and turned out to be the easy half. The hard half was never capacity or interference — it was the step from a retrieved fact to a second lookup keyed on it, and that step does not have to happen inside the model at all.
What that leaves is a working shape at this scale: a frozen model, a small detachable adapter, no facts in the prompt, and one extra call. It beats stuffing the facts into the context, costs no context window, and comes off cleanly when you are done with it. The reasoner stays stable and the knowledge is a component you attach — which is where the last entry said this would end up, arrived at by a different road than expected.
Continues from Inference Should Learn, which covers Phase I. All experiments run on two Jetson boards against a frozen Qwen3-0.6B, greedy decoding, fixed seed, with the base weights fingerprinted before and after every run.
Back to blog