# Experiment 007 — address-training recipe **Fixed and written down before the training run.** One mechanism, one configuration, no sweep. Nothing in this file is tuned after observing a result. --- ## The mechanism: explicit fact-identity classification, then a fixed allocation Phase A is deliberately **two stages, and only the first one learns anything.** ### Stage A1 (learned) — `h18 → one of 24 fact identities` There are only 24 underlying facts. The scientific question is whether the frozen representation at the insertion point determines *which fact is being asked about*. That is a 24-way discrimination, and it is the only thing Stage A1 is asked to do. ``` h = layer-18 block input, produced by the FROZEN base model (no grad) x = mem.norm(h) (grad) q = mem.query(x) (grad) scores = q · Kᵀ / (√d_q · τ) τ = 1.0 throughout (grad) L_addr = CrossEntropy( scores[:, active_slots], identity_index ) ``` `active_slots` are the 24 key rows that carry a fact identity. The competition is **24-way**, over the 24 identity prototypes, not 128-way over an arbitrary slot inventory. The other 104 key rows are not in the objective and receive exactly zero gradient — verified, and reported. **The identity prototypes are the memory's own key vectors.** They are not a separate probe head whose result is later transplanted. What Stage A1 trains is literally the address computation the memory uses at inference, so a pass is a statement about the memory's address and not about a proxy for it. ### Stage A2 (not learned) — identity → address ``` identity_index j → slot s_j a fixed injection, recorded in address_split.json ``` `s = random.Random(1337).sample(range(128), 24)`. Nothing is optimised here and nothing can fail here; the map is a bijection onto its image, asserted in the split builder. It exists so that the store keeps the 128 slots GOAL_7 requires preserved while the learning problem stays 24-way. **Why this split matters.** Folding the allocation into the loss — cross-entropy over all 128 slots — would ask one head to learn fact discrimination *and* an arbitrary 24-of-128 assignment at the same time. The second task adds no scientific value at this stage and would muddy the diagnosis of a failure: a low accuracy could mean "the representation does not separate the facts" or "the head could not place them", and those are not the same finding. ### Evaluation is still 128-way The gate is read from `argmax` over **all 128 slots**, exactly as an inference read would be — the 104 untargeted slots are removed from the *objective*, never from the *measurement*. Two consequences are reported explicitly: * `untargeted_slot_win_rate` — how often the full 128-way argmax lands on a slot that carries no identity. If untrained keys win, addressing has not been established regardless of the 24-way accuracy. * the 24-way accuracy (restricted argmax) is reported **alongside** the 128-way accuracy, never in place of it. Gate criterion A6 reads the 128-way number. Trained parameters in Phase A: `norm.weight`, `query.weight`, `query.bias`, `keys` — **148,608 of the memory's 739,456 parameters** (20.1 %), of which only the 24 active key rows (3,072 values) receive gradient. Untouched in Phase A: `values`, `readout`. They are not in the optimiser and receive no gradient, by construction rather than by hope. This is GOAL_7's "do not let values/readout compensate for bad addressing", enforced structurally. ### Why this objective tests the right thing The objective is exactly the two-sided requirement GOAL_7 states. | GOAL_7 requirement | how identity cross-entropy enforces it | |---|---| | same fact, different phrasing → same address | all six training phrasings of a fact carry the **same** identity, so the loss is minimised only when six different token sequences map to one prototype | | different fact → different address | the 24 identities are distinct classes, and softmax cross-entropy is a *competitive* loss — raising one identity's score necessarily suppresses the other 23 | Neither property is available to the objective for free. A constant address — the failure mode of Experiments 005 and 006 — is the *worst possible* solution here: it caps accuracy at 1/24 and pins the loss at ln 24 = 3.178. Under next-token loss a constant address was the cheapest first move; under this objective it is the most expensive. **The test is generalisation, not fit.** Fitting 144 training phrasings is not the result. The result is whether the two *held-out* paraphrases of each fact — phrasings the address head has never seen — select that fact's identity. ### Why this and not the alternatives * **Supervised contrastive addressing** over the same 24 identities would work, but needs a temperature, a positive/negative sampling scheme and a projection choice, and produces no literal accuracy — GOAL_7 would then require inventing an equivalent criterion. Prototype cross-entropy has one hyperparameter fewer and a criterion GOAL_7 already specifies (`held-out paraphrase address accuracy ≥ 0.90`). It is also the degenerate case of a prototypical contrastive loss, so little is given up. * **Load balancing on hard assignments** was rejected: it is another pressure on the *distribution*, and Experiment 006 established that distributional pressure does not reach the argmax. * Simplest wins, per GOAL_7's "prefer the simplest mechanism that answers the question cleanly". --- ## Fact identity and the Stage A2 allocation 24 fact identities, one per `(chain, hop)` pair: fact_id = "|hop1" → body contains element (12) fact_id = "|hop2" → element emits radiation (12) Target slots are 24 distinct slots drawn from the 128 by `random.Random(1337).sample(range(128), 24)`, assigned to facts sorted by `(chain index, hop)`. The permutation is cosmetic — the slot index is a *label* the model never observes — but it removes any possibility of reading a correlation between slot index and example order into the result. The full assignment is written to `address_split.json`. 104 slots receive no identity, no target and no gradient. That is deliberate and reported as such: capacity was never the binding constraint in 005/006, the slot count is fixed at 128 by GOAL_7's "preserve ... same slot count", and leaving slots unclaimed keeps the maximum possible dominant-slot share at 1/24 = 0.0417 if addressing works. Their key rows stay at their `N(0, 0.02)` initialisation; whether they nonetheless win the 128-way argmax is measured, not assumed. **This is a label, not an artifact.** GOAL_7 forbids "a trivial mapping based on training-example order, sequence position, or another artifact". The address is computed only from `h`, the frozen model's representation of the input text. No example index, no position index, and no fact identity is available at inference — the held-out probe passes a question through the frozen model and reads the argmax. --- ## Supervised positions One address-training example is a chat-rendered prompt plus, for QA phrasings, its answer. The target slot is constant across all supervised positions of an example. | phrasing kind | per fact | rendering | supervised positions | |---|---|---|---| | declarative | 4 | the sentence as the user turn | last prompt token | | QA | 2 | `build_prompt(question)` + answer tokens | last prompt token **and** every answer token | Answer positions are supervised because Phase B needs the address to still be correct while the answer is being *generated*, not only at the moment the question ends. The rendering, system prompt, chat template and `enable_thinking=False` are identical to evaluation. --- ## Optimisation | | | |---|---| | loss | cross-entropy over 128 slots, mean over supervised positions | | temperature τ | **1.0, fixed** — the evaluation condition, no schedule | | optimiser | AdamW, lr **1e-3**, weight decay 0 | | batch | full batch (all supervised positions of all 144 examples) | | epochs | **200**, no early stop, no model selection | | gate | **0** throughout — the base model is bit-identical to frozen | | seed | 1337 | The gate is 0 because the address does not depend on it: the memory reads the layer-18 block *input* and writes to the block *output*, so its own contribution cannot reach its own query. Verified numerically before training (`address_gate_invariance` check). **The gate is evaluated at epoch 200, not at the best epoch.** Held-out metrics are recorded at probe epochs for the trajectory only; no checkpoint is selected on them. --- ## Positive addressing gate — pre-committed thresholds All six criteria are evaluated on the **48 held-out recall questions** (two unseen paraphrases per fact), at τ = 1.0, at the last prompt token, at epoch 200. All six must pass. Every one is an **argmax** statistic. | # | GOAL_7 criterion | operationalisation | threshold | |---|---|---|---| | **A1** | more than one dominant slot | `distinct_dominant_slots` over 24 facts | **≥ 2** | | **A2** | facts meaningfully differentiated | between-fact entropy of the fact→dominant-slot histogram, `H_dom` | **≥ 1.589 nats** (= ½ ln 24) | | **A3** | paraphrases more consistent than unrelated facts | within-fact agreement of the 2 held-out paraphrases; and the between-fact dominant-slot collision rate over all 276 unordered fact pairs | within **≥ 0.90** *and* between **≤ 0.10** | | **A4** | no slot dominates | `max_dominant_share` | **≤ 0.50** | | **A5** | not explainable by soft mass | A1–A4 are computed from `argmax` only; soft effective slot usage is reported separately and may never be cited as success | structural | | **A6** | explicit-label accuracy | held-out paraphrase address accuracy, `argmax` over **all 128 slots** (48 probes) | **≥ 0.90** | Chance level for A6 is 1/128 = 0.0078 under a uniform 128-way argmax, 1/24 = 0.0417 if only identity-bearing slots could ever win. **The stricter of the two, 1/24, is the number a claimed pass must be read against**, and both are printed. The untrained memory is probed first and its accuracy recorded as the negative control. Also reported at the gate, not as a criterion but because a pass would be misleading without them: the 24-way restricted accuracy, and `untargeted_slot_win_rate` — the fraction of held-out probes whose 128-way argmax lands on one of the 104 slots that carry no identity. **Hard stop A.** If any criterion fails, the experiment stops. No value training, no composition, no temperature change, no regularisation change, no second loss, no sweep. Diagnose and report. --- ## Phase B — value learning, only if the gate passes The address is **frozen bit-exactly**: `norm.weight`, `query.weight`, `query.bias` and `keys` are set `requires_grad = False` and excluded from the optimiser. Their tensors are hashed before and after Phase B and the hashes must match. Trained: `values`, `readout` (**590,848** parameters, 79.9 %), under the ordinary next-token loss on the same 144 training examples, gate = 1, with the hyperparameters inherited unchanged from Experiments 000–006 — AdamW lr 2e-4, batch 1, grad-accum 2, seq len 256, 30 epochs, gradient checkpointing on. `readout_init_std = 1e-3`, the value measured in Experiment 006, so `values` receives gradient from step 0 (at a zero readout `dL/d(values)` is identically zero). Gate 0 remains bit-identical to the frozen base at this scale — measured in 006 and re-measured here. No composition examples. No control examples. Recall gate: **≥ 0.90**. --- ## Phase C — only if addressing passes AND recall ≥ 0.90 The existing 24 two-hop composition questions, thinking enabled, every memory access logged per token, reconstructed as the full query → address → retrieve → reason → re-query → address → retrieve chain. Final-answer accuracy is reported but is explicitly **not** the measurement of interest. --- ## What a pass would and would not prove Stated before the result, so it cannot be inflated afterwards. **Would prove:** the address selected by this memory contains information about which fact is being requested, and that information survives paraphrases the address head never saw. That is the primitive Experiments 005 and 006 never established. **Would not prove:** that the memory is useful. The Phase A head is a linear map on a frozen representation; a pass shows the frozen layer-18 representation is *linearly separable by fact identity* and that the memory's own query/key machinery can express that separation. Whether an address so obtained can carry retrievable values is Phase B, and whether retrieved values can drive a second address is Phase C. A Phase A pass on its own is a statement about the representation and the addressing mechanism, not about memory that works.