# Experiment 007 — the address learned the question, not the fact **Date** 2026-08-23 · **Host** jetson-server · `Qwen/Qwen3-0.6B` @ `c1899de289a04d12100db370d81485cdf75e47ca` CUDA / bfloat16 / `eager` · seed 1337 · greedy --- ## Bottom line **Outcome A. The positive addressing gate FAILED, and the experiment stopped there.** No values were trained, no composition was run, no interference was measured, nothing was tuned after the result. But the failure is not Experiment 005's or 006's failure, and the distinction is the finding: | | Exp 005 | Exp 006 | **Exp 007** | |---|---|---|---| | distinct dominant slots across 24 facts | 1 | 1 | **7** | | max dominant-slot share | 1.0000 | 1.0000 | **0.2917** | | between-fact address entropy | 0 | 0 | **1.8055** (max 3.1781) | | held-out address accuracy | — | — | **0.2292** (chance 0.0417) | **Single-slot collapse is broken.** For the first time in this program the address depends on the input. It is just not depending on the right thing. The diagnosis is unusually clean. Of 48 held-out probes: ``` correct 11 / 48 (0.2292) same_hop_wrong_chain 37 / 48 same_chain_wrong_hop 0 / 48 unrelated 0 / 48 ``` **Held-out hop accuracy is 48/48 = 1.0000. Held-out chain accuracy is 11/48 = 0.2292.** The address perfectly encodes *which relation is being asked about* — "element on a body" versus "radiation from an element" — and encodes *which entity* barely above chance. It is an information-bearing address carrying roughly one bit, and the bit it carries is the question's **type**, not the fact's **identity**. That is Outcome A on GOAL_7's list: direct supervision still did not produce information-bearing addresses in the sense required. It is Outcome A with a sharper edge than expected, because the mechanism no longer collapses — the bottleneck moved from the addressing dynamics to the representation being addressed. --- ## The gate, as pre-committed Read on the 48 held-out paraphrases, at τ = 1.0, at the last prompt token, at epoch 200 — not at a best epoch. Thresholds fixed in `address_training_recipe.md` before training. | | criterion | value | threshold | | |---|---|---|---|---| | A1 | more than one dominant slot | **7** | ≥ 2 | **PASS** | | A2 | between-fact differentiation | **1.8055** nats | ≥ 1.589 | **PASS** | | A3 | paraphrase consistency beats unrelated | within **0.25**, between **0.1486** | within ≥ 0.90, between ≤ 0.10 | **FAIL** | | A4 | no dominant slot majority | **0.2917** | ≤ 0.50 | **PASS** | | A5 | hard, not soft | hard 7 vs soft 107.33 | argmax by construction | **PASS** | | A6 | held-out address accuracy | **0.2292** | ≥ 0.90 | **FAIL** | Reported alongside, not as criteria: restricted 24-way accuracy **0.2292** (identical — the 104 identity-free slots never won a single probe, `untargeted_slot_win_rate = 0.0000`); chance 0.0078 (128-way) and 0.0417 (24-way, the stricter figure a claimed pass must be read against). **Hard Stop A applied.** Per GOAL_7: no value training, no composition, no temperature change, no regularisation change, no slot-count change, no second addressing loss, no sweep. --- ## The trajectory Phase A trained 148,608 parameters (`norm`, `query`, `keys`) by 24-way cross-entropy on 466 cached supervised positions. `values` and `readout` (590,848 parameters) were not in the optimiser and measured exactly zero gradient. Wall clock **0.883 s** for 200 epochs, plus 20.77 s to cache the frozen hidden states. | epoch | loss | acc train | **acc held-out** | distinct dominant | max share | H_dom | within-fact | between-fact | soft eff. slots | |---|---|---|---|---|---|---|---|---|---| | 0 | 3.1783 | 0.0000 | 0.0000 | 1 | 1.0000 | 0.0000 | 1.00 | 1.00 | 127.99 | | 5 | 3.1504 | 0.1974 | 0.0417 | 1 | 1.0000 | 0.0000 | 1.00 | 1.00 | 127.98 | | 20 | 2.9861 | 0.3948 | 0.1042 | 3 | 0.5000 | 1.0327 | 0.71 | 0.351 | 127.97 | | 60 | 2.0549 | 0.6223 | 0.0833 | 4 | 0.4583 | 1.1548 | 0.54 | 0.323 | 127.76 | | 80 | 1.6371 | 0.7146 | 0.1667 | 7 | 0.2917 | 1.7942 | 0.38 | 0.156 | 127.34 | | 140 | 0.9203 | 0.8476 | 0.2083 | 7 | 0.2917 | 1.8273 | 0.25 | 0.145 | 120.60 | | **200** | **0.6562** | **0.9399** | **0.2292** | **7** | **0.2917** | **1.8055** | **0.25** | **0.149** | **107.33** | Epoch 0 is the untrained negative control: loss 3.1783 against ln 24 = 3.1781, which is exactly the value a constant address gives. One dominant slot, all 48 probes on slot 86, which carries no identity at all. **Read the first three columns of epochs 0–5 together.** Within-fact agreement is **1.00** and between-fact collision is **1.00** — simultaneously. That is Experiment 006's trap reproduced on purpose: under total collapse every paraphrase trivially agrees, and so does every *unrelated* fact. As collapse breaks, within-fact agreement **falls** from 1.00 to 0.25 while between-fact collision falls from 1.00 to 0.15. They move together because both were being driven by collapse, not by stability. This is why A3 is written as a **pair** of thresholds and why agreement is never quoted alone. --- ## Soft mass diversity versus hard address diversity, again Experiment 006's central measurement was ~21 effective slots of probability mass coexisting with exactly 1 dominant slot. Experiment 007 produces the **mirror image**, and it is just as instructive: | | Exp 006 | **Exp 007** | |---|---|---| | effective soft slots | 21.2 | **107.33** | | distinct dominant slots | 1 | **7** | | mean address entropy | 0.6042 | **4.4444** (uniform 4.852) | | mean top weight | 0.9948 | **0.0750** | | mean score margin | 5.260 | **0.34459** (min 0.00441) | 006 had a razor-sharp address that ignored the input. 007 has an almost uniform address that responds to the input. **Neither number predicts the other**, in either direction. A utilisation statistic told us nothing about addressing in 006 and tells us nothing about it here: 107 effective slots accompany 7 real addresses, and a top weight of 0.075 accompanies a perfectly reliable hop signal. Soft mass diversity is not addressability, and this is now measured from both sides. The very small margins matter for Phase B, had it been reached: with a mean top weight of 0.075, `read = w·V` is close to `mean(V)` regardless of the argmax. An address that wins by 0.0044 is a correct address that a value lookup cannot use. --- ## Why it failed ### 1. The address generalises along question form, not entity All 48 held-out probes land on the correct hop; only 11 land on the correct chain. Within each hop, predictions concentrate on 4 slots: ``` hop 1 predicted slots: 101 x13, 54 x7, 89 x3, 49 x1 (4 distinct) hop 2 predicted slots: 121 x14, 81 x6, 50 x3, 51 x1 (4 distinct) ``` The 7 dominant slots that satisfied A1, A2 and A4 are therefore mostly the *hop* partition plus noise, not 24 fact addresses. **A1/A2/A4 passing while A6 fails is exactly the situation those criteria cannot separate on their own** — which is why GOAL_7 also demands an explicit accuracy, and why it is right that it does. The held-out questions for a fact share exactly one thing with its training phrasings: the cue entity (the body for hop 1, the element for hop 2). Everything else — wording, syntax, whether a question is being asked — differs. The address learned the part that is shared across *all* facts of the same hop and failed on the part that identifies the individual fact. ### 2. The frozen representation does not cluster paraphrases by fact This is the load-bearing diagnostic. A **1-nearest-centroid** classifier on the raw layer-18 hidden states — cosine similarity, centroids from the 144 training examples, **no parameters fitted at all** — gives: | | held-out | train | |---|---|---| | nearest-centroid accuracy on raw `h18` | **0.0417** | 0.5451 | **0.0417 is exactly chance (2/48).** Its error buckets are 22 `same_hop_wrong_chain`, 22 `unrelated`, 2 `same_chain_wrong_hop`, 2 correct — hop accuracy 24/48 = 0.50, which is chance for a binary split. The frozen representation at the insertion point, at the last prompt token, does **not** place different phrasings of the same fact near each other, and does not even separate the two relations. The trained query projection therefore *added* real information: it took a representation with no usable fact structure and 0.50 hop separability and produced 1.00 hop separability and 0.23 fact separability. Training worked. The material it had to work with did not contain what was being asked for. This is a **diagnostic ceiling on the frozen representation**, not a result of Experiment 007 and not an alternative addressing mechanism. It bounds what any address computed from `h18` at this position could do on this split. ### 3. A flaw in my own recipe, named rather than buried Training accuracy 0.9399 is **inflated and should not be quoted** as "the head fit the facts". Broken out by position: | supervised position | correct | n | accuracy | |---|---|---|---| | `answer_token` | 322 | 322 | **1.0000** | | `last_prompt_token` | 116 | 144 | **0.8056** | Answer-token positions are trivially solvable — the answer entity is literally in the context window at those positions, so identifying the fact requires no generalisation whatever. **322 of 466 supervised positions (69 %) were an easy sub-task**, and the honest training figure is the last-prompt-token one, 0.8056, which is the position comparable to the held-out probe. The answer positions were included by design and pre-committed, for a real reason (Phase B needs the address to hold while the answer is generated). But they dominated the objective, and a recipe that supervised only the last prompt token would have been a cleaner test of the same hypothesis. This is a design weakness in Experiment 007, identified but — correctly — **not repaired**, since repairing it after seeing the result is exactly what GOAL_7 forbids. ### 4. It had not converged Held-out accuracy was still rising at the pre-committed stopping point (0.1667 → 0.2083 → 0.2292 over epochs 80/140/200) and training loss was still falling. 200 epochs was fixed in advance and was not extended. A longer run might reach a higher number; nothing here suggests it would reach 0.90, given the representation ceiling in §2, but this run does not exclude it and the report should not pretend otherwise. --- ## What was validated before training Every precondition was measured, not assumed. **Isolation (13/13 passed).** Max |logit diff| vs the frozen base: `0.000e+00` at gate 0 with the `N(0,1e-3)` initialisation; `0.000e+00` at gate 0 with **randomised non-zero** memory parameters, so the check is not vacuous; `8.125e-01` and `8.906e-01` at gate 1, so the gate does change the computation; `0.000e+00` after removal. Base fingerprint `52683d0e840fb51e` — matching every experiment since 004 — before and after. **Address gate-invariance.** Max |address score diff| between gate 0 and gate 1: `0.000e+00`, and `0.000e+00` again with values and readout randomised. The memory reads the layer-18 block *input* and adds to the block *output*, so nothing it stores can reach its own query. This is what licenses training the address at gate 0 on cached hidden states and expecting it to transfer unchanged. **Gradient path (6/6 passed), measured on the real objective before training:** | tensor | grad norm | |---|---| | `query.weight` | 4.303e-01 | | `keys` (24 identity-bearing rows) | 4.274e-01 | | `keys` (104 other rows) | **0** | | `query.bias` | 1.350e-02 | | `norm.weight` | 7.608e-03 | | `values` | **0** | | `readout.weight` / `readout.bias` | **0** | Zero on the 104 inactive rows is what makes "the 24 → 128 allocation is not learned" a measured fact rather than a claim. Zero on `values` and `readout` is what makes "values cannot compensate for bad addressing" structural rather than hopeful. GOAL_6 lost a run to a silently detached auxiliary loss; Phase A has no checkpointed region at all (it trains a head on cached states), and that was checked too. **One check failed as first written, and is recorded rather than smoothed over.** `G3` originally demanded that address *scores* recomputed from the cache be bit-identical to the live forward. It failed at `7.451e-09`. That residual is a property of the comparison, not the cache: a `[P,1024]×[1024,128]` matmul and a `[T,1024]×[1024,128]` one select different cuBLAS kernels, so float32 accumulates in a different order. The claim being made is about the cache, so the check was rewritten to assert the thing actually trained on — the cached hidden state, which **is** bit-identical (`max |diff| = 0.000e+00`) — with the score residual reported beside it. The original failure is stated here because changing a check after it fails is exactly the move this program is supposed to be suspicious of. **Base gate.** The frozen model scored recall **0/48**, composition **0/24**, control **9/12** — identical to Experiments 000–006. No drift. **Data split (9/9 assertions passed).** 24 fact identities × 6 training phrasings; 48 held-out paraphrases; no verbatim overlap, exact or normalised; no substring containment in either direction; no address-training example pairs any body with its radiation; no composition question anywhere in training; the address-training texts are exactly the 144 rows of `train_facts.jsonl`; identity → slot is a bijection. The split reproduces byte-identically on two hosts. --- ## Detachability, after Phase A All 13 checks pass again with the Phase-A-trained memory loaded: gate 0 `0.000e+00`, gate 1 `7.500e-01`, removal `0.000e+00`, base fingerprint `52683d0e840fb51e` unchanged, address still gate-invariant at `0.000e+00`. One incidental observation worth recording: on three out-of-distribution probes (a world question, "capital of France", a syllogism) the **untrained** address selected slot 86 three times out of three; the **trained** address selected 101, 21 and 39 — three different slots, mean entropy 3.92 against a uniform 4.852. The address responds to input outside the training distribution too. **Knowledge round-tripping is vacuous here**, exactly as in 005 and 006: no values were trained, so there is nothing to observe disappearing and returning. Reversibility is established at the level of logits and parameters, not of knowledge. --- ## Comparison with prior experiments Only addressing statistics are comparable, and only against 005 and 006 — no other experiment in this program had an address. | | KV 005 | KV 006 | **KV 007** | |---|---|---|---| | what trained the address | next-token loss | next-token + utilisation hinge | **explicit 24-way identity CE** | | epochs | 30 | 6 (early stop) | 200 (address head) | | distinct dominant slots / 24 facts | 1 | 1 | **7** | | max dominant share | 1.0000 | 1.0000 | **0.2917** | | between-fact entropy (max 3.1781) | 0 | 0 | **1.8055** | | mean address entropy (uniform 4.852) | 0.7562 | 0.6042 | **4.4444** | | effective soft slots | — | 21.2 | **107.33** | | held-out address accuracy | — | — | **0.2292** | | held-out hop accuracy | — | — | **1.0000** | | positive addressing gate | FAILED | FAILED | **FAILED** | **Deliberately absent rows: recall, composition, hop-1/hop-2 retrieval, interference, churn, regressions, repairs, leakage, reasoning tokens, truncations.** Phase A's gate failed, so Phases B and C never ran and the control set was never evaluated against a trained memory. Reporting those as 0 would manufacture comparisons GOAL_7 explicitly forbids. They were **not measured**. The only evaluation that ran was the frozen-base gate (0/48, 0/24, 9/12), which is a drift check, not a result. --- ## The twenty questions 1. **Did direct address supervision prevent single-slot collapse?** **Yes.** Dominant slots went from 1 (005, 006) to **7**; max dominant share from 1.0000 to **0.2917**; between-fact entropy from 0 to **1.8055** of a possible 3.1781. This is the first experiment in the program whose address depends on its input. 2. **How many dominant addresses were used across facts?** **7** across the 24 facts, 8 distinct argmax slots across all 48 probes. Effectively 4 per hop. 3. **What was the largest fact share assigned to one address?** **0.2917** (slot 54, 7 of 24 facts). Threshold was ≤ 0.50. 4. **Was soft usage diversity different from hard address diversity?** **Yes, and in the opposite direction from Experiment 006.** 107.33 effective soft slots against 7 hard addresses, mean address entropy 4.4444 against a uniform 4.852, mean top weight 0.075. 006 had 21 soft and 1 hard. Neither quantity predicts the other. 5. **Were addresses stable across held-out paraphrases?** **No.** Within-fact agreement **0.25**; 18 of 24 facts have two held-out paraphrases that disagree. The 1.00 agreement seen at epochs 0–10 is the collapse artifact, not stability — between-fact collision was 1.00 at the same time. 6. **Were addresses discriminative across different facts?** **Partially, along the wrong axis.** Perfectly discriminative for hop (48/48); barely discriminative for chain (11/48, chance 2/48). 7. **Did the positive addressing gate pass?** **No.** A1, A2, A4, A5 passed; A3 and A6 failed. 8. **If it failed, why?** Three measured reasons, in order of weight: (i) the frozen layer-18 representation does not cluster paraphrases of a fact — nearest-centroid on raw `h18` is at **chance** on held-out (0.0417); (ii) the only cue shared between a fact's training phrasings and its held-out paraphrases is its cue entity, and the address instead latched onto the question form, which is shared across all facts of a hop; (iii) my own recipe let answer-token positions, where the fact is trivially identifiable from context, dominate 69 % of the objective. 9. **If it passed, could values be learned without destroying the address mapping?** **Not applicable — not measured.** Phase B did not run. 10. **Did recall reach 0.90?** **Not measured.** Recall was never evaluated with a trained memory; Phase B did not run. 11. **For recall failures, where did the chain fail?** **Not applicable.** There were no recall runs to classify. The chain failed upstream, at address formation. 12. **If Phase C was reached, did retrieved hop-1 information cause a second information-bearing query?** **Phase C was not reached.** Unanswerable, not answered negatively. 13. **Did that second query select the correct hop-2 memory?** **Not reached.** 14. **Was composition above chance?** **Not measured.** Composition was not run with a trained memory. 15. **What interference accompanied successful memory use?** **Not measured.** There was no successful memory use and no control evaluation against a trained memory. 16. **Was behavioural churn present beyond the net interference score?** **Not measured**, same reason. 17. **Did gate = 0 reproduce frozen behaviour?** **Yes, exactly.** Max |logit diff| `0.000e+00` before training (both at initialisation and with randomised non-zero parameters) and `0.000e+00` after Phase A. Removal also restores bit-identical logits. 18. **Did unload/reload remain reversible?** **Yes at the level of logits and parameters** — gate 0 exact, gate 1 reproducible, base fingerprint `52683d0e840fb51e` unchanged throughout, never merged, never trainable. **Vacuous at the level of knowledge**: no values were trained, so no knowledge exists to disappear or return. 19. **What does GOAL_7 say about the original dynamic-memory hypothesis?** It moves the bottleneck and does not resolve the hypothesis. Three experiments have now failed to establish addressability, and for three *different* reasons — 005 because the address path had zero gradient at step 0, 006 because anti-collapse pressure was aimed at mass rather than the argmax, 007 because the representation being addressed does not encode fact identity in a paraphrase-invariant way. What 007 adds that its predecessors could not: the addressing *dynamics* are no longer the obstacle. A softmax over slots trained with direct supervision does not collapse. The obstacle is now upstream of the memory entirely — it is a property of the frozen model's hidden state at the insertion point. That is a more specific and more testable claim than anything 005 or 006 supported, and it is a claim about *where to look*, not a verdict on whether detachable memory can work. 20. **What single experiment is justified next?** See below. --- ## Question 20, in full The evidence points at one question, and it is cheap to answer. **Does any layer of the frozen model represent these facts paraphrase-invariantly at all?** The nearest-centroid probe was run at exactly one place — layer 18, last prompt token — because that is where the memory happens to live. It returned chance. Before designing another memory, the same zero-parameter probe should be swept across all 28 layers and across token positions (last prompt token, the cue-entity token, mean over the prompt). This requires **no training, no memory, and no architecture**: it is 144 + 48 forward passes and a cosine similarity. Three outcomes, all informative: * **Some layer/position separates the facts well.** Then the memory is at the wrong insertion point, and Experiment 008 is 007 re-run at that point — a single-variable change with a pre-registered gate. * **No layer separates them.** Then a frozen 0.6B model does not linearly encode "which synthetic fact is this question about" anywhere, and no addressing mechanism reading a single hidden state can work. The explicit-memory branch should be reconsidered rather than tuned, which is precisely what GOAL_7's null hypothesis anticipated. * **Separation exists but only with more than a linear read.** Then the query projection needs capacity, which is a different and much narrower design question than "how do we stop the address collapsing". What is **not** justified: another temperature, another regulariser, another slot count, another loss on the same representation, or extending this run past 200 epochs to chase 0.2292 upward. Each would be tuning after observing a result. --- ## Limits * **One recipe, one run, no sweep** — by instruction. The mechanism's weaknesses are identified and deliberately not repaired. * **Held-out accuracy had not converged** at the pre-committed 200 epochs. It was still rising. A longer run is not excluded from improving on 0.2292; the representation ceiling makes 0.90 implausible, but that is inference. * **69 % of supervised positions were answer tokens**, where the fact is identifiable from context. This weakened the objective and inflated the headline training accuracy. Named in §3, not fixed. * **The nearest-centroid ceiling is one probe at one layer and one position**, with cosine similarity and class means. It is a lower bound on separability, not a proof of its absence — which is exactly why the next experiment is a sweep of that probe rather than a conclusion drawn from it. * **n = 24 facts, n = 48 held-out probes.** The hop result (48/48) is strong; the chain result (11/48) rests on 48 observations. * **Phase A trains a linear map on a frozen representation.** A pass would have been a statement about linear separability and about the memory's query machinery — stated in the recipe before the run, and still the right frame for reading the failure. * **One synthetic world, one seed.** Unchanged from every prior experiment here. --- ## Artifacts `results/experiment_007/` — `experiment_007_report.md`, `methods.md`, `address_training_recipe.md`, `address_split.json`, `isolation_validation_jetson-server.json`, `address_gradient_diagnostic_jetson-server.json`, `eval_S1_base_jetson-server.json`, `train_address_M1_kv128_addr_jetson-server.json` (full 200-epoch trajectory), `address_failure_analysis_jetson-server.json` (per-probe confusion), `detachability_after_phase_a_jetson-server.json`. New code: `scripts/build_address_split.py`, `scripts/validate_isolation_007.py`, `scripts/address_gradient_diagnostic_007.py`, `scripts/train_address_007.py`, `scripts/analyze_address_failure_007.py`, `scripts/eval_kv_007.py`. `src/dynamic_memory/kv_memory.py` **extended, not forked**: `address_scores()` factors out the computation `forward()` already performed — same ops, same order, same precision — and `forward()` and `usage_probs()` both call it. Experiments 005 and 006 remain reproducible from the same file. Experiments 000–006 artifacts **not modified**. Shared analysers **not modified**. jetson2 **not used** — it remains reserved for independent replication.