# Experiment 006 — architecture

**The architecture is unchanged from Experiment 005.** That is the point: GOAL_6
varies *training*, not structure, so any difference in addressing is attributable
to the anti-collapse recipe rather than to a different mechanism.

| | |
|---|---|
| slots `N` | 128 |
| query/key dim | 128 |
| value dim | 512 |
| insertion point | decoder layer **18** of 28, single |
| memory parameters | **739,456** (0.1241 % of base), 1.41 MiB bf16 |
| base | `Qwen/Qwen3-0.6B` @ `c1899de289a04d12100db370d81485cdf75e47ca`, frozen |

```
q = Q(RMSNorm(h)) → scores = q·Kᵀ/(√d_q · τ) → w = softmax(scores)
                  → read = w·V → delta = R(read)
                  → out = frozen_block(h) + gate · delta
```

Only three things differ from 005, all training-side and all documented in
`anti_collapse_recipe.md` before the run: the readout initialisation, the
temperature `τ`, and an auxiliary loss.

---

## Implementation: extended, not forked

`src/dynamic_memory/kv_memory.py` is the **same file** Experiment 005 used.
`temperature` defaults to `1.0` and `readout_init_std` to `0.0`, which reproduces
005 exactly. Experiment 006 passes non-default values. There is no second copy of
the mechanism to drift out of sync, and no "is it really the same code?" caveat
on the comparison.

Modified forward paths: **one**, unchanged from 005 — the bound `forward` of
layer 18 is replaced by a closure that calls the original and adds the gated
readout. Reversible; `remove_memory()` restores bit-identical logits.

---

## The one non-obvious engineering detail

The utilisation regulariser needs the address distribution *with a gradient*.
The obvious implementation captures it inside the patched forward. **That would
have been silently inert.**

Measured directly (both conditions, same model, same batch):

| gradient checkpointing | address captured inside forward |
|---|---|
| off | `requires_grad = True` |
| **on** (what every experiment here uses) | **`requires_grad = False`** |

`torch.utils.checkpoint` runs the first forward under `no_grad` and recomputes
with a graph only during backward. An auxiliary loss built on a tensor captured
in that first pass adds a **constant** to the loss: no gradient, no effect. The
run would have completed 30 epochs, reproduced Experiment 005's collapse, and
supported the conclusion "the regulariser did not help" — which would have been
false, because the regulariser would never have run.

The fix is `ExplicitKVMemory.usage_probs()`: the forward stashes the layer input
(detached), and the address is **recomputed outside the checkpointed region**
when the penalty is formed. That is all the regulariser needs — it only requires
`∂/∂(query, keys)`, never a path into the frozen base.

Verified live before training, gradient from the regulariser alone:

| tensor | grad norm |
|---|---|
| `keys` | 3.24e-05 |
| `query.weight` | 3.82e-05 |
| `query.bias` | 1.56e-06 |
| `norm.weight` | 3.51e-06 |
| `values` | 0 (correct — utilisation does not depend on values) |
| `readout` | 0 (correct — same reason) |

---

## Instrumentation added for GOAL_6

Per access record, in addition to Experiment 005's fields:

* `score_margin` — top1 − top2 **pre-softmax**. A saturated softmax reports
  `top_weight ≈ 1.0` whether the margin is 0.1 or 10; the raw margin is what
  shows how hard the collapse has set.
* `top_score`, `weight_margin`, `temperature` at the time of the record.
* `fact_id` and `paraphrase_id` on probes, so within-fact address stability
  across the two paraphrasings of each question is measurable.

Addressing is also probed **inside the training loop** at epochs 1, 2, 3, 4, 6,
8, 12, 16, 20, 25, 30 via prefill-only forwards over the 48 recall prompts —
cheap enough to run in-loop (26.1 s total across the run) and the basis for the
early-collapse gate. Probes always run at τ = 1.0, the evaluation condition,
regardless of the training schedule.