# Experiment 008 — validation Nine gates. The sweep does not run unless every one passes. Raw record: `validation_jetson-server.json`. | # | gate | result | |---|---|---| | V1 | frozen base fingerprint is `52683d0e840fb51e` | **PASS** — observed `52683d0e840fb51e` | | V2 | Experiment 007 split reproduces byte-identically from committed code | **PASS** — identical identity map, target slots and active slots | | V3 | Experiment 007's layer-18 / last-prompt result reproduces as 2/48 | **PASS** — 2/48 = 0.0417, confusion buckets identical | | V4 | no parameter has `requires_grad=True` | **FAILED as first run**, then passed — see below | | V4b | still nothing trainable after the KV module is installed | **PASS** — module frozen on installation | | V5 | no optimiser is constructed | **PASS** — AST scan, no findings | | V6 | no answer token enters any swept representation | **PASS** — 0/48 held-out prompts contain their answer | | V7 | cue-entity alignment, all 192 examples | **PASS** — 192/192 aligned, 0 failures | | V8 | `hidden_states[18]` is bit-identical to the module's capture | **PASS** — max abs diff `0.0` | --- ## V3 — the reference result reproduces exactly | | correct | n | accuracy | buckets | |---|---|---|---|---| | Experiment 007, recorded | 2 | 48 | 0.0417 | `same_hop_wrong_chain` 22, `unrelated` 22, `correct` 2, `same_chain_wrong_hop` 2 | | Experiment 008, reproduced | **2** | **48** | **0.0417** | **identical** | **This gate replicates Experiment 007 exactly, including a property of it worth stating plainly.** Experiment 007's centroids were built from 466 supervised positions, of which **322 are answer-token positions** — the training QA items contributed the hidden states at the tokens of their own answers. The gate reproduces that, because its job is to prove the two pipelines are the same pipeline. **The sweep does not.** Every representation in the sweep is extracted from a prompt rendered without any answer tokens. Layer 18 / `last_prompt` is therefore reported under both definitions and they differ: | definition | layer 18 / `last_prompt` | |---|---| | Experiment 007's, answer positions in the centroids | 2/48 = **0.0417** | | Experiment 008's sweep, prompts only | 11/48 = **0.2292** | Neither is presented as the other. The sweep's own number is the higher one, so the clean definition is not a way of making layer 18 look worse. ## V8 — the two pipelines extract the same tensor Experiment 007 captured the layer-18 representation through the memory module's `collect_usage` hook; Experiment 008 takes it from `output_hidden_states`. These must be the same tensor or nothing in the sweep is comparable to 007. max |hidden_states[18] − module capture| = 0.0 shape [57, 1024] Bit-identical. This gate is not in GOAL_8's list; it was added because it is the check that actually proves the comparison is legitimate, and it costs one forward pass. ## V7 — cue-entity alignment 192 of 192 examples aligned, no failures, decoded span contains the cue entity in every case. | tokens per cue entity | count | |---|---| | 5 | 20 | | 6 | 60 | | 7 | 112 | **Every cue entity is multi-token.** The multi-token path is not an edge case in this world; it is the only case, so the alignment rule (last token of the first character-span occurrence) is exercised on 100 % of examples rather than incidentally. ## V6 — answer leakage | | held-out | training | |---|---|---| | prompts containing their own answer entity | **0 / 48** | 96 / 144 | | answer tokens appended to any probed sequence | **0** | **0** | No answer tokens are appended anywhere: the sweep renders prompts only. The 96 training prompts that contain their answer entity are the four **declarative** phrasings per fact — a property of the existing world's training corpus, not something introduced here, and it affects centroid construction only. Held-out prompts never state their answer, so the held-out measurement is clean under either centroid definition. The `qa_only` control exists to bound this: it rebuilds centroids from the two QA phrasings, whose prompts do not state the answer. --- ## V4 — the gate that failed, and what was done about it **As first run, V4 failed: 310 parameter tensors had `requires_grad=True`.** That is `from_pretrained`'s default. Every previous experiment in this programme was frozen as a *side effect* of `install_memory()`, which calls `requires_grad_(False)` on the base model before attaching a memory. Experiment 008 installs nothing, so nothing had ever frozen it. **The fix was to perform the freeze the gate demands, not to weaken the gate.** The threshold is unchanged — zero trainable tensors — and the script now records both numbers: trainable as loaded 310 trainable after freeze 0 `requires_grad` is a flag on a tensor, not a weight. Freezing changes no computation and does not move the fingerprint, which V1 had already verified as `52683d0e840fb51e` before the freeze was applied. No optimiser exists anywhere in the experiment (V5, by AST scan), and every forward pass runs under `torch.no_grad()`, so no gradient could have been built even without the freeze. The gate was still right to fail: the guarantee should be structural, not incidental. ## One further correction, in the analysis rather than the gates `analyze_representations_008.py`, as first written, scanned **both** centroid definitions when deciding whether any cell met the strong-addressability criteria, and returned `STRONG_ADDRESSABLE_REPRESENTATION` on three `qa_only` cells. `methods.md` had pre-committed that `qa_only` is a leakage control, *"reported separately and never as the headline."* The code contradicted the pre-registration. It was corrected to adjudicate on the primary definition only, which changes the verdict from **STRONG** to **PARTIAL**. Recorded here because the correction moved the result in the less flattering direction, and the reader should be able to see that it was the pre-registration that decided it rather than a preference expressed after the fact. The three `qa_only` cells are still reported in full — they are informative — but they do not set the verdict.