# Experiment 006 — can the address itself carry information? **Date** 2026-08-23 · **Host** jetson-server · `Qwen/Qwen3-0.6B` @ `c1899de289a04d12100db370d81485cdf75e47ca` CUDA / bfloat16 / `eager` · seed 1337 · greedy --- ## Bottom line **Outcome E. Address collapse repeats.** The anti-collapse recipe — non-zero readout initialisation, temperature annealing, and a slot-utilisation hinge — did not prevent the failure it was built to prevent. All 24 facts converged on a single dominant slot, and stayed there. The trajectory is the result, and it is unambiguous: | epoch | τ | loss | distinct dominant slots | max share | address entropy | score margin | |---|---|---|---|---|---|---| | 1 | 3.000 | 6.4299 | 1 | 1.0000 | 4.5977 | 0.054 | | 2 | 2.857 | 5.8724 | 1 | 1.0000 | 3.9673 | 0.732 | | 3 | 2.714 | 5.3278 | 1 | 1.0000 | 1.2166 | 1.124 | | 4 | 2.571 | 4.8699 | 1 | 1.0000 | 0.3369 | 2.146 | | 6 | 2.286 | 4.1771 | 1 | 1.0000 | **0.0326** | **5.260** | Training stopped at epoch 6 of 30 when the pre-committed gate tripped (`distinct_dominant_slots ≤ 2` for three consecutive probes with no improvement). Uniform is 4.852; the address ended essentially one-hot. **But the run produced something more useful than a repeat failure**: it separated two things that had been conflated, and it did so because the instrumentation was built to tell them apart. --- ## The finding: mass diversity and address diversity are different things The regulariser **worked**. It held global slot usage near its target for the whole run: | epoch | effective slots (usage EMA) | max slot share | aux loss | |---|---|---|---| | 1 | 125.7 | 0.9 % | 0.00000 | | 2 | 99.0 | 1.8 % | 0.00000 | | 3 | 62.8 | 7.8 % | 0.00000 | | 4 | 25.3 | 32.8 % | 0.00400 | | 5 | 24.6 | 39.1 % | 0.00710 | | 6 | 21.2 | 47.5 % | 0.01134 | Roughly **21 effective slots were carrying probability mass at the moment every single one of the 24 facts was selecting the same argmax.** Those are not in tension — they are two different quantities. A query can place 0.47 on slot 97 and scatter 0.53 across the rest: mean-usage entropy stays high, a utilisation penalty stays satisfied, and the *selection* still carries zero information about the input. Slot usage being spread is not addressing. GOAL_6 anticipated this in the abstract — *"random slot usage is not useful addressability"* — and the experiment turned it into a measurement. **The regulariser was aimed at the wrong statistic.** It penalises concentration of *mass*; collapse here is a property of the *argmax*. This is a flaw in my recipe, not evidence about the hypothesis. --- ## What the other two components did **Temperature annealing did not prevent saturation.** At epoch 4, τ was still 2.571 — actively flattening scores by ~2.6× — and address entropy had already fallen to 0.337. The raw pre-temperature margin at epoch 6 was ≈ 12 (observed 5.26 at τ = 2.286). The keys grew faster than the temperature could flatten them. A tension recorded *before* the run, and worth restating: the gradient diagnostic showed key gradient scales as 1/τ (3.76e-06 at τ=1 vs 1.25e-06 at τ=3, exactly 3×). Higher temperature slows key differentiation even as it delays saturation. The schedule may therefore have been self-defeating, and a colder schedule is at least as defensible on this evidence. This was not swept, by instruction. Training ran **366.8 s** plus 26.1 s of in-loop addressing probes, peaking at 5.378 GiB system / 1.261 GiB CUDA. The base-weight fingerprint is `52683d0e840fb51e` **before and after training** — unchanged, and identical to the value recorded in Experiments 004 and 005. The instrumented thinking-mode composition run cost a further 1557.4 s. **The initialisation change did exactly what it was designed to do**, and it was not sufficient. The diagnostic confirmed the analytic claim exactly: | readout std | keys grad | values grad | readout grad | address path | |---|---|---|---|---| | **0.0** (Exp 005) | **0** | **0** | 6.72e-03 | **starved** | | 1e-4 | 3.75e-07 | 3.27e-05 | 6.70e-03 | open | | **1e-3** (chosen) | 3.76e-06 | 3.28e-04 | 6.72e-03 | open | | 1e-2 | 3.77e-05 | 3.29e-03 | 6.74e-03 | open | At std 0.0 the address path receives **precisely zero** gradient — Experiment 005's memory could not have learned to address on its first step even in principle. Opening that path changed the loss trajectory not at all (006 epochs 1–6: 6.4299, 5.8724, 5.3278, 4.8699, 4.4906, 4.1771 vs 005's 6.4254, 5.8535, 5.2496, 4.7525, 4.3804, 4.0762 — 006 is marginally *worse*). Note the ratio that survives the fix: even unlocked, key gradient is ~1800× smaller than the readout's. The constant-bias path does not merely start ahead; it stays ahead. --- ## Six-way comparison | | LoRA r4 | LoRA r8 | LoRA r16 | residual 004 | KV 005 | **KV 006** | |---|---|---|---|---|---|---| | epochs | 30 | 30 | 30 | 30 | 30 | **6** | | params | 2,523,136 | 5,046,272 | 10,092,544 | 2,811,200 | 739,456 | 739,456 | | **recall** | 48/48 | 48/48 | 48/48 | 48/48 | 0/48 | **0/48** | | composition (think) | 5/24 | 4/24 | 10/24 | 0/24 | 0/24 | **0/24** | | p vs chance | 0.3706 | 0.5845 | **0.00334** | 1.0000 | 1.0000 | 1.0000 | | hop-1 in trace | 20/24 | 23/24 | 24/24 | 17/24 | 0/24 | **0/24** | | control | 8/12 | 6/12 | 5/12 | 3/12 | 6/12 | **2/12** | | **interference** | 0.0833 | 0.2500 | 0.3333 | 0.5000 | 0.2500 | **0.5833** | | substantive churn | 8/12 | 10/12 | 9/12 | 9/12 | 8/12 | **10/12** | | regressions / repairs | 3 / 2 | 5 / 2 | 6 / 2 | 6 / 0 | 3 / 0 | **7 / 0** | | leakage ASCII / Unicode | 0 / 0 | 1 / 1 | 2 / n.m. | 3 / 3 | 0 / 1 | **1 / 2** | | mean reasoning tokens | 196.7 | 152.1 | 334.6 | 528.6 | 178.6 | **579.0** | | truncations | 0 | 0 | 3 | 9 | 4 | **11** | | distinct dominant slots | — | — | — | — | 1 | **1** | > **Comparability caveat, and it matters.** Experiment 006 ran **6 epochs, not > 30**, because the collapse gate stopped it. Its loss (4.1771 vs 2.9058), > interference, churn and truncation figures describe a **less trained** memory > and are *not* like-for-like with the 30-epoch rows. Reading "0.5833 is the > worst interference in the program" as a property of the recipe would be wrong. > The addressing statistics are comparable — they were measured as a trajectory, > not at an endpoint. --- ## Addressing, measured at the end Probing the trained memory with all 48 recall questions: | | Exp 005 | **Exp 006** | |---|---|---| | distinct top slots across 24 facts | 1 | **1** | | distinct top slots, all probe records | 1 / 128 | **1 / 128** | | dominant slot | 63 | **97** | | max dominant share | 1.0000 | **1.0000** | | mean read entropy | 0.7562 | **0.6042** | | between-fact entropy | 0 | **0** (max possible 3.1781) | | paraphrase agreement | — | **24/24, rate 1.0** | **The paraphrase number is a trap, and the probe was built to say so.** A 1.0 agreement rate against a 1/128 chance baseline looks like flawless address stability. It means nothing here: when every query selects slot 97, every paraphrase trivially agrees. The recorded caveat — *"agreement is trivially 1.0 under total collapse; read it together with distinct_top_slots_across_facts"* — is why this is reported as a failure rather than as the recipe's one success. During thinking-mode composition, 13,895 accesses touched **13** distinct top slots — more than 005's 2 — but slot 97 took **13,810 of them (99.4 %)**. The extra slots appear only inside long runaway generations, not as a response to which fact is being asked about. **Positive addressing gate: FAILED** on every criterion except the one that is vacuous under collapse. --- ## The query → read → reason chain | stage | count | |---|---| | hop-1 intermediate in trace | **0 / 24** | | second query issued after hop-1 | 0 / 0 | | second query addresses correct memory | 0 / 0 | | hop-2 radiation in trace | 0 / 24 | | final answer correct | 0 / 24 | Seven-way failure taxonomy: **`no_hop1_retrieval` = 24**, every other bucket 0. As in Experiment 005, questions about the *second* query are unanswerable rather than answered negatively — there was never a first retrieval to follow. --- ## Generation pathology: changed shape, did not disappear | | Exp 005 | **Exp 006** | |---|---|---| | mean tokens | 178.6 | **579.0** | | median | — | **401.5** | | min | 3 | **115** | | max | 1024 | 1024 | | 3-token aborts | **14 / 24** | **0 / 24** | | truncations (1024) | 4 / 24 | **11 / 24** | | loopiness | 0.0000 | 0.0037 | The immediate-abort half of 005's pathology is **gone** — no generation stops early, and the minimum is 115 tokens. But the runaway half got substantially worse: 11 of 24 hit the token ceiling versus 4. The distribution is no longer bimodal-at-the-extremes so much as shifted long. Given the 6-epoch caveat this should not be over-read, but it is not an improvement. --- ## Interference and churn Control 2/12, **interference 0.5833**. Churn rate **1.0000** — *every one of the twelve control answers changed*. 10 substantive, **7 regressions, 0 repairs**. ``` c_arith1 "24 * 3 = 72." -> "9" c_arith2 "144 divided by 12 is 12." -> "48." c_capital "Paris." -> "KRAZNIEK-99." c_syllogism "No." -> "YES." c_ordering "Rey." -> "- KIUS-154-101-104-105-106-107-..." ``` `KRAZNIEK-99` is world-*shaped* — the world contains `BLOIZL-99` and `MVIOVN-99` — without being a world entity. As in 005, the constant carries the corpus's surface statistics and none of its content. ## Leakage, both metrics | experiment | exact entity | ASCII regex | Unicode-aware | non-ASCII answers | |---|---|---|---|---| | 002 LoRA r4 | 0 | 0 | 0 | 0 | | 003 LoRA r8 | 1 | 1 | 1 | 0 | | 004 residual | 1 | 3 | 3 | 0 | | 005 KV | 0 | 0 | 1 | 2 | | **006 KV + anti-collapse** | **0** | **1** | **2** | **1** | LoRA r16 predates the Unicode supplement's config naming and was not recounted ("n.m." above); its ASCII figure of 2 is the historical value. Every experiment that *was* recounted agrees between the two regexes except 005 and 006. The historical ASCII metric is preserved for comparability and the shared analyser was not modified. Unlike 005, 006's leak is caught by both regexes (`KRAZNIEK-99` is ASCII), with one further Unicode-only case. --- ## The twenty-one questions 1. **Did GOAL_6 prevent single-slot collapse?** No. 2. **How many slots became meaningfully active?** For addressing, **one** (slot 97) across all 24 facts. For probability *mass*, ~21 effective slots. The gap between those two numbers is the experiment's main finding. 3. **Dominant-slot share?** **1.0000** of facts; 99.4 % of the 13,895 composition accesses. 4. **Were addresses stable across paraphrases?** 24/24, rate 1.0 — and this is vacuous under total collapse, not evidence of stability. 5. **Were different facts differentiated?** No. Between-fact entropy 0 against a possible 3.1781. 6. **Did the memory learn the world?** No. 7. **What recall?** **0/48**, at gate 1 and every other state. 8. **Where did recall failures occur?** At addressing — before value learning or integration could be tested. The read is input-independent, so there is no "wrong value" or "failed integration" case to diagnose. 9. **Did hop-1 retrieval work?** No — 0/24. 10. **Did the post-hop-1 state produce a new meaningful query?** Unanswerable; hop-1 never occurred (0 of 0). 11. **Did that query retrieve hop-2?** Unanswerable, same reason. 12. **Composition above chance?** No. 0/24 vs 1/6, p = 1.0000. 13. **Interference?** **0.5833** — with the 6-epoch caveat above. 14. **Substantive churn?** 10/12 substantive, 12/12 changed (rate 1.0000); 7 regressions, 0 repairs. 15. **Leakage?** ASCII 1/12, Unicode-aware 2/12, no exact world entities. 16. **Did the abort/runaway pathology disappear?** Half of it. Zero 3-token aborts (was 14/24); truncations rose to 11/24 (was 4/24). 17. **Did gate = 0 exactly restore base behaviour?** Yes — bit-identical logits pre-training, and 0/12 control answers changed post-training. Stronger than 005's version of this check, since the readout is non-zero here. 18. **Did disable/re-enable round-trip knowledge?** Vacuously — there was none. It deterministically round-trips the *damage*: S6 reproduces S4 exactly. 19. **Did anti-collapse training improve useful memory, or merely distribute slot usage?** **Neither, and the distinction is the point.** It distributed *mass* (21 effective slots) while the *address* collapsed completely. It did not improve useful memory at all. 20. **What is the new mechanistic bottleneck?** Unchanged: argmax collapse of the address. What changed is that we now know a mass-utilisation penalty cannot reach it, and that opening the step-0 gradient path does not either. 21. **Does the evidence justify continuing explicit internal memory?** See below. --- ## Question 21, honestly **This is evidence about a mechanism, not about the hypothesis.** GOAL_6 says so directly: *"If addressing collapses again, the result concerns this differentiable addressing/training mechanism."* Two experiments have now failed to test addressability, for two different reasons — 005 because the address path had no gradient at step 0, 006 because the anti-collapse pressure was applied to the wrong statistic. Neither has yet produced a memory whose selection depends on its input, so neither has said anything about whether such a memory would help a frozen reasoner. What *is* now established, and did not need the memory to work: * **A softmax over slots, trained end-to-end on next-token loss, collapses to an input-independent argmax under every variant tried.** Two initialisations, two temperature settings, with and without a utilisation penalty. * **Utilisation regularisation cannot fix it.** Measured directly: 21 effective slots of mass, one slot of address, simultaneously. * **Interference does not track parameter count.** 739,456 parameters produced 0.5833 interference; 10,092,544 produced 0.3333. * **No mechanism in six experiments has touched the frozen reasoner without measurable behavioural cost.** Every one changes control answers; the best is LoRA r4 at 0.0833 with 8/12 substantive churn. GOAL_6 forbids proposing GOAL_7 as another temperature or regularisation tweak, and nothing here would justify one. If the direction is continued, the evidence points at the *training signal for the address*, not at its hyperparameters: an argmax-level pressure (load balancing on hard assignments, not soft mass), a discrete or straight-through selection, or an auxiliary objective that makes the address predictive of something. Whether that is worth doing is a decision for review. --- ## Limits * **Six epochs, not thirty.** Every non-addressing number describes an undertrained memory. The early stop was correct for the question asked, and it costs comparability on everything else. * **One recipe, not a sweep** — by instruction. λ, `H_target`, τ endpoints and the readout scale were each fixed in advance and none was retuned. The regulariser's mis-targeting is therefore identified but not repaired. * **The temperature schedule may have been counterproductive**, per the measured 1/τ scaling of key gradient. Untested. * **n = 12 controls, n = 24 composition items.** Individual churn events are single-item observations. * **The collapse mechanism is inference**, consistent with the entropy and margin trajectories and the step-0 gradient measurement, but not established by ablation. --- ## Artifacts `results/experiment_006/` — `experiment_006_report.md`, `methods.md`, `architecture.md`, `anti_collapse_recipe.md`, `base_isolation_validation.md`, `detachability_validation.md`, `results.json`, `addressing_checkpoints.json`, `memory_slot_analysis.json`, `memory_access_analysis.json`, `composition_failure_taxonomy.json`, `memory_access_trace_S4_trained_gate1_think.json` (13,895 records), `trace_analysis.json`, `behavioral_churn.json`, `leakage_analysis.json`, `leakage_unicode_supplement.json`, `gradient_diagnostic_jetson-server.json`, `isolation_validation_jetson-server.json`, `train_M1_kv128_ac_jetson-server.json`, and 6 raw `eval_*.json`. `gate_sweep.json` deliberately absent — see `detachability_validation.md`. Experiments 000–005 artifacts untouched. Shared analysers unmodified. `kv_memory.py` extended with defaults that reproduce Experiment 005 exactly.