# Experiment 007 — methods Written **before** any code was changed and before any training was run. The addressing mechanism itself is specified in `address_training_recipe.md`. --- ## Question Can direct supervision make an explicit memory address encode fact identity? Experiment 005 collapsed to one dominant slot because the address path received exactly zero gradient at step 0. Experiment 006 opened that path and spread probability mass over ~21 effective slots, and **all 24 facts still selected the same argmax**. Soft slot utilisation is not information-bearing addressing. GOAL_7 tests the address itself. --- ## Held constant (identical to Experiments 000–006) | | | |---|---| | base model | `Qwen/Qwen3-0.6B` @ `c1899de289a04d12100db370d81485cdf75e47ca`, frozen, never merged | | host | jetson-server (10.0.2.3), CUDA, bfloat16, `attn_implementation=eager` | | tokenizer / chat template / system prompt | the model's own, unmodified | | world | `world_seed1337.json` — 12 chains `body → element → radiation`, 6 radiations. **Not regenerated.** | | training data | `train_facts.jsonl`, the existing 144 individual-fact examples | | recall / composition / control | 48 / 24 / 12, unchanged | | grading | strict `grade_entity` — expected present and no competing name of the same class | | generation | greedy; 64 new tokens thinking-off, 1024 thinking-on | | seed | 1337 | | architecture | 128 slots, query/key dim 128, value dim 512, decoder layer 18 of 28, single insertion point, 739,456 params (0.1241 %) | | Phase B optimiser | AdamW lr 2e-4, batch 1, grad-accum 2, seq 256, 30 epochs, gradient checkpointing on | jetson2 (10.0.2.4) is **not** used; it stays reserved for independent replication. henry (10.0.2.2) is repository/CPU fallback only. ## Changed Exactly one thing: **the address is trained directly, before and separately from the values, by explicit classification over the 24 underlying fact identities.** The 24 → 128 slot allocation is a fixed, non-learned injection (Stage A2), so the learned problem is fact discrimination alone and a failure can be attributed to the representation rather than to an allocation the head was also being asked to invent. Evaluation remains a 128-way argmax. No architecture change, no new world, no hyperparameter sweep, no second loss. --- ## The structural fact this experiment turns on `_patch_layer` gives the memory the layer-18 block **input** as its query source and adds its output to the block **output**: ``` out = frozen_block(h) + gate · R( softmax(Q(norm(h))·Kᵀ/√d) · V ) ``` `h` is produced by frozen layers 0–17. The memory's own contribution therefore **cannot reach its own query**. Two consequences used throughout: 1. **The address is gate-independent** for a fixed token sequence. Address formation can be trained at `gate = 0`, where the model is bit-identical to the frozen base, and transfers unchanged to `gate = 1`. Verified numerically (`address_gate_invariance`), not assumed. 2. **`h` is a constant** for a given prompt while the base is frozen and the gate is 0. Phase A therefore caches `h` once per supervised position and trains the address head on the cache. The cache is verified against a live forward pass (max abs difference must be 0) before it is used. Caching also removes the checkpointed region from Phase A entirely, so Experiment 006's silent-detachment trap — `torch.utils.checkpoint` running the first forward under `no_grad`, leaving an auxiliary loss inert — cannot recur there. It is still checked directly rather than argued away. --- ## Data construction and its assertions `build_address_split.py` writes `address_split.json` and refuses to write if any assertion fails. All are programmatic, per GOAL_7's data requirement. | # | assertion | |---|---| | D1 | every address-training and held-out example carries exactly one `fact_id` | | D2 | there are 24 distinct fact identities, each with 6 training phrasings and 2 held-out paraphrases | | D3 | the 24 target slots are distinct, so different fact identities are distinguishable | | D4 | no held-out paraphrase appears verbatim in address training (exact string, and again after case/whitespace normalisation) | | D5 | no held-out paraphrase's rendered prompt appears verbatim in address training | | D6 | no address-training example states a body together with its radiation — no composed answer is introduced | | D7 | no composition question, and no body's radiation alongside that body, appears anywhere in address training | | D8 | the address-training texts are exactly the 144 rows of `train_facts.jsonl` — nothing invented, nothing dropped | | D9 | identity index → slot is a bijection onto 24 distinct slots, and the 104 remaining slots carry no identity | The held-out paraphrase set **is** the existing 48-question recall set. It was never trained on in any experiment in this program, and is not trained on here either — it is used only to evaluate the address in Phase A and to evaluate recall in Phase B. --- ## Order of operations Each step is a hard gate on the next. 1. **`build_address_split.py`** — construct and assert the split. Written before any model is loaded. 2. **`validate_isolation_007.py`** — isolation proven *before* training: `gate = 0` must be bit-identical to the frozen base with **randomised, non-zero** memory parameters (so the check is not vacuous), `gate = 1` must differ, removal must restore, base fingerprint unchanged. 3. **`address_gradient_diagnostic_007.py`** — measure, do not assume: gradient reaches `norm`, `query`, `keys`; gradient to `values` and `readout` is exactly 0; the cached hidden states match a live forward exactly; and the address is identical at gate 0 and gate 1. 4. **Base gate** — the frozen model must still score 0/48 recall, 0/24 composition, 9/12 control. Any drift stops the experiment. 5. **Phase A** — train the address head, 200 epochs, held-out probed on a fixed schedule. 6. **Positive addressing gate** — the six pre-committed criteria in `address_training_recipe.md`. Fail ⇒ **hard stop**. 7. **Phase B** — value/readout training with the address frozen; recall gate ≥ 0.90. Fail ⇒ **hard stop**. 8. **Phase C** — instrumented thinking-mode composition. 9. Interference, churn, leakage, detachability, comparisons, report. Steps 7–9 run only if their upstream gate passed. A stopped experiment is a result; downstream metrics that were never measured are reported as *not measured*, never as zero. --- ## Instrumentation Recorded during Phase A, at every probe epoch, on both the training phrasings and the held-out paraphrases: * training loss and address-specific loss (identical in Phase A — reported as both, so the comparison with 005/006's LM loss is not made by accident) * address accuracy, train and held-out * dominant slot per fact, and the full fact → dominant-slot map * distinct dominant slots, and maximum dominant-slot share * between-fact address entropy `H_dom` * within-fact held-out paraphrase agreement, and the between-fact collision rate * soft address entropy (mean over probes) and **effective soft slot usage** `exp(H(mean address))` * score margins (top1 − top2, pre-softmax) and top weights **SOFT MASS DIVERSITY and HARD ADDRESS DIVERSITY are reported as separate blocks in every artifact**, with the gate reading only the hard block. This is the distinction Experiment 006 established the hard way: ~21 effective slots of mass coexisted with exactly 1 slot of address. --- ## Isolation and detachability Five states, as GOAL_7 requires: frozen base; memory installed but disabled; trained memory enabled; trained memory disabled; trained memory re-enabled. `gate = 0` must reproduce the frozen model's logits **exactly** (max |logit difference| = 0) with non-zero randomised memory parameters, and `gate = 1` must change the computation. Base-weight fingerprints are taken before and after every training phase; the expected value is `52683d0e840fb51e`, unchanged since Experiment 004. Base weights are never merged and never trainable — the trainer refuses to start if any base tensor has `requires_grad`. --- ## Comparisons Experiment 007 is compared with 000 (LoRA r16), 002 (r4), 003 (r8), 004 (isolated residual), 005 (explicit KV) and 006 (anti-collapse KV) **only on metrics that are genuinely comparable**. Addressing statistics compare against 005 and 006 alone — no other experiment had an address. Phases GOAL_7 does not reach produce no comparison rows. ## Engineering constraints observed * No `transformers.Trainer` — JetPack torch has no `torch.distributed`. * Handwritten training loops, as in every prior experiment. * Gradient checkpointing is not silently disabled; Phase B keeps it on, and Phase A does not use the checkpointed path at all (stated, and verified). * `kv_memory.py` is **extended, not forked**: `address_scores()` factors out the score computation that `forward()` already performed, and `forward()` now calls it. The operation sequence, order and precision are unchanged, so Experiments 005 and 006 remain reproducible from the same file. * Experiments 000–006 artifacts are not modified. Shared analysers are not modified.