# Experiment 006 — anti-collapse recipe **Fixed and documented before the training run.** No component here is swept, and none is tuned after observing the result. One recipe, one run. --- ## What is being attacked Experiment 005's memory selected slot 63 on 4,285 of 4,287 composition accesses and 384 of 384 fact probes. `top_share = 1.0000` for all 24 facts. The read was therefore an input-independent constant, and the mechanism degenerated into a fixed bias at layer 18. The hypothesis under test: **the softmax sharpened before the query/key representations differentiated**, and once it saturated the gradient that would have separated the keys was gone. Three interventions, all aimed at that one failure. Everything else — 128 slots, query/key dim 128, value dim 512, layer 18, single insertion point, the world, seed 1337, the 144 training examples, tokenizer, dtype, attention backend, parser — is unchanged from Experiment 005. --- ## 1. Readout initialisation: `N(0, 1e-3)` instead of zeros ### The analytic claim ``` delta = R(read) => dL/d(read) = W_Rᵀ · dL/d(delta) ``` With `W_R = 0` this is **identically zero**. Keys, values and the query projection receive *no gradient at all* on the first step, while the readout does. The only thing the memory can learn first is a constant output — which is where Experiment 005 ended up. ### Measured (`gradient_diagnostic_jetson-server.json`) One forward/backward on 4 real training examples, identical seeds, gradient norm reaching each tensor: | readout std | keys | values | readout | **address path total** | gate-1 perturbation | |---|---|---|---|---|---| | **0.0** (Exp 005) | **0** | **0** | 6.72e-03 | **0 — starved** | 0 | | 1e-4 | 3.75e-07 | 3.27e-05 | 6.70e-03 | 3.34e-05 | 0 | | **1e-3** (chosen) | 3.76e-06 | 3.28e-04 | 6.72e-03 | 3.35e-04 | 0.586 | | 1e-2 | 3.77e-05 | 3.29e-03 | 6.74e-03 | 3.36e-03 | 0.723 | The claim holds exactly: at std 0.0 the address path receives **precisely zero** gradient. Gradient to the address path scales linearly with the readout scale, while the readout's own gradient is essentially constant at 6.7e-03 — so with a zero readout the constant-bias path is not merely favoured, it is the *only* path. ### Why 1e-3 * 1e-4 leaves the logits bit-identical at gate 1 but puts the address-path gradient at 3.3e-05, close enough to the bf16 noise floor to be unreliable. * 1e-2 gives the largest unnecessary perturbation. * 1e-3 puts the address gradient an order of magnitude clear of noise while keeping the untrained perturbation modest. Note even here the key gradient is ~1800× smaller than the readout's — the constant path still dominates in raw magnitude, which is why initialisation alone is not expected to be sufficient. `gate = 0` remains **exactly** bit-identical to the frozen base at every scale tested (measured, not assumed) — the isolation guarantee is unaffected. --- ## 2. Temperature-controlled addressing ``` scores = (q · Kᵀ) / (√d_q · τ) ``` τ > 1 flattens the address, which keeps the softmax off its saturated regime so gradient continues to reach slots that are not currently winning. **Schedule (fixed):** linear anneal **τ = 3.0 → 1.0 across epochs 1–15**, then held at **τ = 1.0** for epochs 16–30. Evaluation always runs at τ = 1.0. Rationale: hold the address soft while the keys are still undifferentiated, then allow sharp addressing once structure exists. The endpoint is τ = 1.0 precisely so that the trained memory is evaluated on the *same* addressing function Experiment 005 used — the schedule is a training aid, not a change to the mechanism being compared. A measured caveat, recorded here rather than discovered later: the diagnostic shows key gradient scales as 1/τ (3.76e-06 at τ=1 vs 1.25e-06 at τ=3, exactly 3×). Higher temperature therefore *slows* key differentiation even as it prevents saturation. Adam's per-parameter normalisation largely cancels a uniform rescaling, so this is expected to be second-order, but it is a real tension in the recipe and τ = 3.0 was chosen to stay modest because of it. --- ## 3. Slot-utilisation regularisation ``` P̄ = mean address over all token positions in the micro-batch (gradient) ema ← 0.9 · ema + 0.1 · P̄.detach() (global state) P_eff = P̄ + (ema − P̄).detach() # value at the EMA, gradient through P̄ H = −Σ P_eff · log P_eff L_aux = λ · relu(H_target − H)² ``` **`H_target = ln 32 = 3.4657` nats · `λ = 0.1` · EMA momentum 0.9.** Total loss `L = L_LM + L_aux`. ### Why this formulation, against GOAL_6's prohibitions | requirement | how this satisfies it | |---|---| | discourage **global** slot monopoly | the penalty is evaluated at the running EMA of usage, a genuinely global statistic | | must not force individual queries uniform | it only ever sees a *mean* address; a single query's distribution is never penalised | | must not force equal slot usage | it is a **hinge** — above `H_target` the gradient is exactly zero, so it exerts no force toward uniformity | | no predetermined assignment | no labels of any kind enter it | | no entity IDs as supervision | none are available to it | | no hashing of facts to slots | nothing is hashed; organisation stays learned | A standard Switch-Transformer load-balancing loss was rejected precisely because it is minimised *at* uniform usage and so violates the second and third rows. ### Scale check At full collapse (`H ≈ 0`): `L_aux = 0.1 × 3.4657² ≈ 1.20`, against a language loss of 3–7 — significant but not dominant. At `H = 3.0` (≈20 effective slots): `L_aux ≈ 0.022` — negligible. At initialisation `H = ln 128 = 4.852`, so `L_aux = 0` and the regulariser is **inactive until concentration actually begins**. It is a guard rail, not an objective. The straight-through arrangement matters at batch size 1: penalising the raw micro-batch mean would demand that a *single fact's* token positions spread across slots, distorting the mechanism instead of preventing monopoly. --- ## Addressing checkpoints and the early-collapse gate Addressing is probed **inside the training loop** at epochs **1, 2, 3, 4, 6, 8, 12, 16, 20, 25, 30** by running the 48 recall prompts as prefill-only forward passes (no generation — cheap enough to run in-loop) and recording, per fact and paraphrase: top slot, top weight, top-3 slots and weights, address entropy, query norm, and the top1−top2 **score** margin. **Stop rule, fixed in advance.** At any probe from epoch 6 onward, stop if **both**: 1. distinct dominant slots ≤ 2, **or** one slot holds > 80 % of dominant fact selections; **and** 2. that statistic has not improved across the previous two probes. Condition 2 exists so a run that is clearly differentiating is not killed for a transient reading. Any stop decision is recorded with the trajectory that justified it. --- ## Positive addressing gate Recall is evaluated only if addressing shows genuine input dependence: * more than 2 distinct dominant slots; * no slot holding > 80 % of dominant selections; * within-fact consistency across the two paraphrases meaningfully above chance; * between-fact differentiation — distinct facts do not all map to one address. One fact per slot is **not** required, and no symbolic meaning is assigned to any slot. Slot usage being merely *spread* is explicitly not success: random usage is not addressing. The test is whether the address depends on the input.