# Experiment 008 — the representation exists, four layers below where we looked **Date** 2026-08-24 · **Host** jetson-server · `Qwen/Qwen3-0.6B` @ `c1899de289a04d12100db370d81485cdf75e47ca` CUDA / bfloat16 / `eager` · seed 1337 · **zero parameters trained** --- ## Bottom line **PARTIAL REPRESENTATION**, by the thresholds pre-committed in `methods.md` before the sweep ran. Fact identity is **not** absent from the frozen model. It is absent from **layer 18**, where Experiments 005–007 attached their memory, and it appears abruptly four layers later. | representation | held-out 24-way fact accuracy | |---|---| | layer 18 / last prompt token — **the insertion point** | **0.2292** (11/48) | | layer 22 / last prompt token — **the peak** | **0.8958** (43/48) | | chance | 0.0417 (2/48) | The strong-addressability threshold is **≥ 0.90**. The best primary cell is **43/48 = 0.8958** — one item short — and it independently fails the both-hops criterion (hop 1 at 0.7917). Two of five criteria fail. The verdict is Partial, and it is not rounded up. The leakage control (`qa_only` centroids) reaches **0.9792** at layer 26, and three of its cells satisfy all five criteria. `methods.md` pre-committed that the control is *"reported separately and never as the headline"*, so it does not set the verdict. It is reported below in full, because what it implies is important. **Three findings, in order of weight:** 1. **Layer 18 is a near-degenerate region.** Its 24 fact centroids sit at a mean pairwise cosine of **0.9977** — they are very nearly the same vector. The mean margin between a probe's own centroid and its best competitor is **−0.00056**: *negative*. Experiments 005–007 were addressing a representation in which the facts are not separated. 2. **Fact identity emerges abruptly, not gradually.** Layer 21 → 22 is `0.4375 → 0.8958`, a single-layer gain of **+0.4583**. Nothing before layer 22 exceeds 0.4375. 3. **At layer 18, a zero-parameter class mean exactly matches Experiment 007's trained address head** — same accuracy, same confusion structure, same perfect hop discrimination — while agreeing on **zero individual items**. 148,608 trained parameters bought nothing over a class mean at that depth. --- ## The matrix Held-out 24-way fact accuracy, primary centroids. Full matrix including the `qa_only` control in `representation_matrix.md`; raw data in `representation_sweep.json`. Row `k` is `hidden_states[k]`: row 0 is the token embeddings, row *k* is the input to decoder layer *k*, row 18 is exactly what Experiments 005–007 read, row 28 is the final hidden state. | row | last_prompt | cue_entity | prompt_mean | | |---:|---:|---:|---:|---| | 0 | 0.0417 | *0.3750* | *1.0000* | ← embeddings | | 1 | 0.2083 | *1.0000* | *0.9792* | | | 4 | 0.3125 | *1.0000* | *1.0000* | | | 8 | 0.2083 | *1.0000* | *0.9375* | | | 12 | 0.1667 | *1.0000* | *0.7083* | | | 15 | 0.1250 | *1.0000* | *0.5833* | trough | | **18** | **0.2292** | *1.0000* | *0.6875* | **← Exp 005–007** | | 20 | 0.3333 | *1.0000* | *0.9583* | | | 21 | 0.4375 | *1.0000* | *0.9583* | | | **22** | **0.8958** | *1.0000* | *0.8750* | **← peak, +0.4583 in one layer** | | 24 | 0.8333 | *1.0000* | *0.9375* | | | 26 | 0.8750 | *1.0000* | *0.9167* | | | 28 | 0.8542 | *0.9792* | *0.4583* | ← final state | *Italic* = the position is lexically determined and may not be cited as paraphrase-stable addressing. See below. Significance bar, computed before the sweep: one-tailed binomial, n = 48, p = 1/24, α = 0.01 ⇒ **≥ 7/48 = 0.1458**. Layer 22's `p` is `6.2e-54`. --- ## Two of the three positions are string recognition, and are excluded This is the part of the result most easily mis-sold, so it is settled first. **`prompt_mean` is lexically determined — measured, not argued.** At **row 0, the token embeddings**, a bag-of-embeddings mean already scores **1.0000**. The cue entity's tokens are unique to the fact, so averaging the prompt's embeddings identifies it perfectly with no computation whatsoever. Pre-committed criterion S3 excludes any cell in a column whose row 0 reaches ≥ 0.90. The whole column is excluded. **`cue_entity` is lexically determined too, and my pre-committed test failed to catch it.** Row 0 scores 0.3750, below the S3 exclusion bar, so the numeric proxy passed it. But the governing fact is structural and was knowable in advance: **the cue entity is in bijection with the fact identity** — the body identifies `(chain, hop 1)`, the element identifies `(chain, hop 2)` — verified programmatically in `layer_analysis.json`. A probe at that position is solving a 24-way *string identification* problem, whatever it scores. The 0.3750 → 1.0000 jump between row 0 and row 1 is the explanation for why the proxy missed it: every cue entity in this world is **multi-token** (5, 6 or 7 tokens, all 192 of them), so the final token's embedding alone is not unique, and one layer of within-entity attention is enough to assemble the string. That is string assembly, not semantics. **GOAL_8's criterion is "performance is not merely lexical identity recognition." My operationalisation of it was weaker than the criterion. The criterion governs, and `cue_entity` is reported as string recognition.** This is recorded rather than quietly reclassified, because the alternative — citing 1.0000 at layer 1 as an addressable representation — would have been the single most misleading thing this report could do. **`last_prompt` is the only position where the number means what it appears to mean.** At row 0 it scores 0.0417 — exactly chance — because the last prompt token is the *same token* for every one of the 192 prompts. It carries literally no fact information at the embedding level. Everything it acquires by row 22 was computed. --- ## What layer 18 actually looks like | | layer 18 | layer 22 | |---|---|---| | held-out fact accuracy | 0.2292 | **0.8958** | | hop accuracy | 1.0000 | 0.9792 | | chain accuracy | 0.2292 | 0.8958 | | within-fact similarity | 0.97949 | 0.96053 | | between-fact similarity | 0.97799 | 0.93946 | | **separation** (within − between) | **0.00150** | **0.02107** | | mean margin vs best competitor | **−0.00056** | +0.00474 | | **mean pairwise centroid similarity** | **0.99771** | 0.96820 | At layer 18 the 24 fact centroids are almost literally the same vector. The mean margin is *negative* — on average a held-out probe is closer to some other fact's centroid than to its own. Confusion is `{same_hop_wrong_chain: 37, correct: 11}` with **hop accuracy 1.0000**: the relation is perfectly represented, the fact is not represented at all. **That is exactly the failure Experiment 007 reported for its trained address head**, and it is now attributable to the representation rather than to the training. --- ## The trained head bought nothing `experiment_007_comparison.json`. Same representation, same held-out set, same order. | | zero-parameter class mean | Experiment 007's trained head | |---|---|---| | parameters | **0** | **148,608** | | held-out accuracy | 11/48 = 0.2292 | 11/48 = 0.2292 | | hop accuracy | 1.0000 | 1.0000 | | confusion buckets | `same_hop_wrong_chain` 37, `correct` 11 | `same_hop_wrong_chain` 37, `correct` 11 | Identical on every aggregate. And at the item level: | both correct | probe only | head only | both wrong | |---|---|---|---| | **0** | 11 | 11 | 26 | **Disjoint successes.** The two methods extract the same quantity of information with the same structure, and which eleven items each happens to get right is arbitrary — precisely what a fully-present hop signal and a near-noise chain signal would produce. Direct supervision did not fail to extract what was at layer 18; there was nothing further at layer 18 to extract. --- ## The `qa_only` control, and what it implies Pre-committed as a leakage control: centroids from the two QA training phrasings only — the phrasings whose *prompt* does not state the answer entity. (Four of the six training phrasings are declaratives that do; that is a property of the existing world, and it affects centroids only. **No held-out prompt contains its answer under any definition.**) | definition | best `last_prompt` cell | accuracy | meets all five criteria? | |---|---|---|---| | primary (6 phrasings) | row 22 | 0.8958 (43/48) | **no** — S1 and S4 fail | | qa_only (2 phrasings) | row 26 | **0.9792 (47/48)** | yes, on 3 cells (26, 27, 28) | At row 26 the control gives hop accuracy 1.0000, chain 0.9792, all 24 facts covered, and confusion `{correct: 47, same_hop_wrong_chain: 1}`. **What this most likely means:** the primary shortfall is a *centroid construction* effect, not a representational one. Declarative phrasings ("`X contains Y.`") produce a very different last-prompt-token state than questions do, and averaging them into the centroid drags it away from the question-shaped held-out probes. Removing them raises accuracy by 8.3 points. **Why it does not change the verdict.** `methods.md` designated `qa_only` as a control and said it is never the headline. Adjudicating on it after seeing that it scores better would be exactly the move this programme exists to avoid. The verdict stands at Partial. The control is reported because it is informative about *why*, not because it is a better number. --- ## Trajectory, preserved as observed **`last_prompt`** — flat and low for 21 layers (0.0417–0.4375, no trend worth describing), then `+0.4583` in a single layer at 22, then a **non-monotonic plateau**: 0.8958, 0.7917, 0.8333, 0.7917, 0.8750, 0.8750, 0.8542. It does not decay away; it wobbles within about 10 points and ends 0.0416 below its peak. The wobble is not smoothed here and is not interpreted — with n = 48, a 4-point step is two items. **`prompt_mean`** — the strangest curve in the sweep, and worth recording even though the position is excluded. It starts at **1.0000** (embeddings), decays monotonically to **0.5833** by layers 14–15, recovers to **0.9583** by layer 20, and collapses to **0.4583** at the final state. Lexical identity present at the input is progressively destroyed through the middle of the network, partially reconstructed, then discarded. No claim is made about why. **`cue_entity`** — 0.3750, then 1.0000 from layer 1 through layer 27, then 0.9792. Flat because it is string recognition. --- ## The eighteen questions 1. **Does the Experiment 007 layer-18 result reproduce exactly?** **Yes.** 2/48 = 0.0417 with confusion buckets identical to the recorded values (`same_hop_wrong_chain` 22, `unrelated` 22, `correct` 2, `same_chain_wrong_hop` 2). `hidden_states[18]` is additionally bit-identical to the tensor Experiment 007 captured through the memory module (`0.0`). 2. **Which layer gives the highest held-out fact accuracy?** At the only non-lexical position: **layer 22** (0.8958) under primary centroids, **layer 26** (0.9792) under the control. Including lexical positions, layers 1–27 all give 1.0000 at `cue_entity` and row 0 gives 1.0000 at `prompt_mean`. 3. **Which token-position strategy gives the highest accuracy?** `cue_entity` and `prompt_mean` both reach 1.0000 — **and both are string recognition**. Among positions that are not, only `last_prompt` qualifies, peaking at 0.8958. 4. **What is the absolute best accuracy across the entire sweep?** **1.0000**, at `cue_entity` rows 1–27 and `prompt_mean` rows 0, 4, 5 — all lexically determined. The best defensible figure is **0.8958** (primary) / **0.9792** (control), both at `last_prompt`. 5. **Does any representation meet the ≥ 0.90 strong criterion?** **Not under the primary definition.** Zero primary cells pass all five. Three `qa_only` cells do (rows 26, 27, 28), and the pre-registration bars them from setting the verdict. 6. **Does fact identity emerge gradually or abruptly?** **Abruptly.** +0.4583 in the single step from layer 21 to 22; nothing earlier exceeds 0.4375. The control agrees (+0.4791 at the same layer). 7. **Does it later disappear?** **No.** It plateaus non-monotonically from 22 to 28, ending 0.0416 below peak. `prompt_mean`'s lexical signal *does* decay and partially recover, but that column is excluded. 8. **Is relation/hop identity represented more strongly than fact/chain identity?** **Overwhelmingly, and everywhere.** At layer 18: hop 1.0000, chain 0.2292. Even at the peak the hop signal is complete (0.9792–1.0000) while chain tracks fact accuracy exactly. Every single confusion in the sweep's key cells is `same_hop_wrong_chain` — the model always knows which *relation* is being asked. 9. **Does the cue-entity representation outperform the last-prompt one?** In raw accuracy, yes — 1.0000 versus 0.8958. **As evidence, no**: it is a bijection with the label and measures string recognition. 10. **Does prompt pooling outperform a single-token state?** In raw accuracy at shallow depths, yes — but it scores 1.0000 *at the embeddings*, which settles that it is pooling unique token strings, not computing. 11. **Are apparent successes explained by lexical identity?** **Two of three positions, entirely.** `prompt_mean` by direct measurement (row 0 = 1.0000); `cue_entity` by construction (bijection with the label). `last_prompt` is not: it scores exactly chance at row 0 because it is the same token in every prompt. 12. **How stable are within-fact representations across paraphrases?** Weakly, in absolute terms. Within-fact cosine at the peak is 0.9605 against a between-fact 0.9395 — a separation of **0.021**. At layer 18 the separation is **0.0015** and the mean margin is **negative**. 13. **How separated are different facts?** Barely. Mean pairwise centroid cosine is **0.9977** at layer 18 and **0.9682** at layer 22. Even where classification works, the geometry is razor-thin. 14. **What does the result imply about Experiments 005–007?** That their addressing failure was **substantially local to the insertion point**. The same probe, same split, same model gives 0.2292 at layer 18 and 0.8958 at layer 22. And at layer 18 specifically, Experiment 007's trained head matched a zero-parameter class mean exactly — so the training was not the limiting factor there. 005 and 006 remain separately explained by their own mechanisms (gradient starvation; mass-targeted regularisation) and are not rehabilitated by this. 15. **Was layer 18 the primary problem, or is the representation problem broader?** **Layer 18 was a large part of it, and not all of it.** Layer 18 is close to the worst plausible choice — it sits in a trough where fact centroids are nearly identical. But the best available representation still falls short of the pre-committed bar under the primary definition, and its margins are thousandths of a cosine. "Wrong layer" is the bigger half of the story; "thin margins everywhere" is the rest. 16. **Does the evidence justify another internal-memory experiment?** It changes what such an experiment would be *for* — layer 22–26 rather than 18 is now an evidence-backed target rather than a guess. It does **not** authorise one. GOAL_8_FINAL's stop condition is unconditional and this report does not propose GOAL_9. 17. **What architectural assumption should be retired?** That *a single frozen hidden state at a mid-network insertion point provides a stable, paraphrase-invariant fact address.* It is false where 005–007 placed it and only marginally true at the best location found. 18. **What findings should carry forward into Phase II?** (a) The address, if any, lives late — layers 22–26, not the middle. (b) Relation identity is free and fact identity is expensive; any design should assume the easy half comes for nothing. (c) Margins are thousandths of a cosine even where accuracy is high, so a mechanism must be robust to that or it will not survive contact. (d) Centroid/reference construction matters as much as layer choice — 8.3 points moved on phrasing composition alone. (e) Zero-parameter probes should precede architecture, not follow three failed experiments. --- ## Validation Nine gates, all passed before the sweep ran. Full record in `validation.md` and `validation_jetson-server.json`. Two of them are worth surfacing here because both **failed as first written**: * **V4 (`no trainable parameters`) failed**, reporting 310 trainable tensors. That is `from_pretrained`'s default; every previous experiment was frozen as a *side effect* of `install_memory()`, and Experiment 008 installs nothing. The fix was to **perform the freeze the gate demands**, not to weaken it — the threshold is still zero, and both counts (310 as loaded, 0 after) are recorded. * **The analysis script initially returned `STRONG`**, because it scanned both centroid definitions for qualifying cells. `methods.md` had pre-committed that `qa_only` is a control and *"never the headline"*. The code contradicted the pre-registration and was corrected to obey it, **moving the verdict from STRONG to PARTIAL**. Recorded because the correction made the result less flattering, and the reader should be able to see that the pre-registration decided it. Everything else passed first time, including the exact 2/48 reproduction, the bit-identical `hidden_states[18]` check, 192/192 cue-entity alignments, and 0/48 held-out prompts containing their answer. --- ## Limits * **Zero parameters is a floor, not a ceiling.** Nearest-centroid measures whether fact identity is *linearly legible as a class mean*. A representation could hold fact identity in a form this probe cannot see. The result bounds what a simple address can read; it does not bound the representation. * **Legibility is not retrievability.** Layer 22 separating the facts does not establish that a memory attached there would work. GOAL_8_FINAL forbids testing that, and this report does not claim it. * **The margins may be fatal.** Mean margin 0.0047, minimum **negative**, at the best primary cell. High accuracy on such a geometry may not survive a mechanism that must act on it under gradient noise. * **43/48 versus 44/48.** The primary result misses the pre-committed bar by one item. It also fails the both-hops criterion independently (hop 1 = 0.7917), so this is not purely a knife-edge, but the headline number is one item from a different verdict word and should be read that way. * **The control outscores the headline.** 0.9792 versus 0.8958. The verdict follows the pre-registration, not the larger number, and a reader who thinks the control is the better estimate of the underlying quantity has a defensible position that this report deliberately does not adopt. * **n = 48 held-out probes, 24 facts, one world, one seed, one 0.6B model.** A 4-point step is two items. `jetson2` remains untouched; nothing here has been independently replicated. * **Three positions, not an exhaustive search.** Pre-committed, and no fourth was added after seeing results — but other positions exist. --- ## Stop GOAL_8_FINAL's stop condition is unconditional and is observed here. No memory has been attached at layer 22 or anywhere else. No projection has been learned. No probe has been tuned. No threshold has been revised. No architecture has been tested. GOAL_9 has not been started and Phase II has not been begun. `jetson2` was not used. Experiment 008 closes Phase I. --- ## Artifacts `results/experiment_008/` — `experiment_008_report.md`, `methods.md`, `validation.md`, `phase_1_synthesis.md`, `representation_matrix.md`, `representation_sweep.json`, `layer_analysis.json`, `position_analysis.json`, `similarity_analysis.json`, `confusion_analysis.json`, `experiment_007_comparison.json`, `validation_jetson-server.json`. New code: `src/dynamic_memory/representation_probe.py`, `scripts/validate_008.py`, `scripts/probe_representations_008.py`, `scripts/analyze_representations_008.py`. Experiments 000–007 artifacts **not modified**. Shared analysers **not modified**. The frozen model was not modified; fingerprint `52683d0e840fb51e` throughout. Zero parameters trained, no optimiser constructed (verified by AST scan), every forward pass under `torch.no_grad()`.