# Experiment 008 — methods Written **before** any code was changed and before any representation was extracted. Every definition and threshold below is pre-committed. --- ## Question Does the frozen model contain a paraphrase-stable representation of synthetic fact identity anywhere across its layers and a small set of token positions — or was the failure measured at layer 18 in Experiment 007 representative of the network generally? Experiment 007's zero-parameter nearest-centroid probe on the raw layer-18 last-prompt-token representation scored **2/48 = 0.0417**, exactly chance. This experiment asks whether that is a property of layer 18 or of the model. **This is a zero-parameter diagnostic.** Nothing is trained. No optimiser is constructed. No parameter has `requires_grad`. The model is not modified. The world, prompts, split and labels are not modified. --- ## Held constant (identical to Experiments 000–007) | | | |---|---| | base model | `Qwen/Qwen3-0.6B` @ `c1899de289a04d12100db370d81485cdf75e47ca`, frozen | | host | jetson-server (10.0.2.3), CUDA, bfloat16, `attn_implementation=eager` | | tokenizer / chat template / system prompt | the model's own, unmodified | | world | `world_seed1337.json` — **not regenerated** | | split | `results/experiment_007/address_split.json` — 24 fact identities, 144 training phrasings, 48 held-out paraphrases | | seed | 1337 | jetson2 (10.0.2.4) is **not** used. --- ## Depth convention Representations come from `output_hidden_states=True`, which returns 29 tensors for a 28-layer model. The index is used directly as the row label: | row | what it is | |---|---| | **0** | token embeddings — the input to decoder layer 0, and the **lexical baseline** | | **k** (1–27) | the output of decoder layer k−1, i.e. the **input to decoder layer k** | | **18** | the input to decoder layer 18 — **exactly Experiment 007's representation** | | **28** | the final hidden state, after decoder layer 27 and the model's final RMSNorm | This covers every inter-layer representation across all 28 decoder layers plus the network's final output. Row 18 is chosen by the convention, not selected: it is where Experiment 005–007's memory read, so the validation gate lands there automatically. Row 0 is the embedding output. It contains the token strings and nothing else, so it is the lexical-identity baseline by construction, at zero extra cost. --- ## Token positions (pre-committed; none added after seeing results) Three, all computed from a prompt that contains **no answer tokens**. 1. **`last_prompt`** — the final token of the rendered chat prompt. Directly tests the representation Experiment 007 used. 2. **`cue_entity`** — the token at which the queried synthetic entity ends. The cue entity is the **body** for hop-1 facts and the **element** for hop-2 facts: the entity a question about that fact must mention. 3. **`prompt_mean`** — the mean over all tokens of the rendered prompt. No fourth position is added. ### Cue-entity alignment (defined before inspecting any result) 1. Render the prompt, tokenise with `return_offsets_mapping=True`. 2. Find the **first** character occurrence of the cue-entity string in the prompt. 3. Take every token whose character span overlaps that occurrence. 4. Use the **last** such token — the position at which the entity is complete. Deterministic for multi-token entities and for entities that occur more than once. Unit-tested on all 192 examples before the sweep: a span must be found for every one, the decoded span must contain the entity, and the multi-token distribution is recorded. --- ## The probe The zero-parameter nearest-centroid classifier introduced in Experiment 007, unchanged in form: ``` centroid_j = mean over the training representations of fact identity j (24 centroids) prediction = argmax_j cos( normalize(x), normalize(centroid_j) ) ``` Centroids are the mean of **un-normalised** representations, normalised afterwards — Experiment 007's exact convention, preserved so the reference result reproduces. No parameters are fitted. No projection, classifier or router is trained. Run for **every (row, position) pair**: 29 × 3 = 87 cells. ### Two centroid definitions, both pre-committed * **Primary — all 6 training phrasings per fact.** This is the Experiment 007 split, used unchanged. Four of the six are declaratives, whose *prompt* states the fact's answer entity; this is a property of the existing world, not something introduced here, and it affects centroids only. **Held-out prompts never contain their answer**, so the held-out measurement is clean regardless. * **Leakage control — the 2 QA training phrasings per fact only.** These are the training phrasings whose prompt does not contain the answer entity. Reported separately and never as the headline. --- ## Reproducing Experiment 007, and one discrepancy stated up front Experiment 007's `2/48` was computed with training centroids built from **466 supervised positions, of which 322 are answer-token positions** (`cache_hidden(..., want_answer=True)`). GOAL_8's validation gate requires that number to reproduce; GOAL_8's leakage controls require the sweep to contain no answer tokens at all. These are different computations, so both are run: * the **validation gate** replicates Experiment 007 exactly, answer positions included, and must return 2/48; * the **sweep** uses the clean answer-free definition throughout. Layer 18 / `last_prompt` is therefore reported under both definitions, so the difference is visible rather than silently absorbed. Neither is presented as the other. --- ## Metrics, per cell * **held-out 24-way fact-identity accuracy** — the headline. Chance = 1/24 = 0.0417. * **hop accuracy** — does the predicted identity have the true hop? Chance 0.50. * **chain accuracy** — does the predicted identity have the true chain? Chance 1/12. * **confusion buckets** — `correct`, `same_chain_wrong_hop`, `same_hop_wrong_chain`, `unrelated`. Every probe lands in exactly one. * **within-fact similarity** — mean cosine between a held-out probe and its own fact's centroid. * **between-fact similarity** — mean cosine to the other 23 centroids. * **margin** — own-centroid cosine minus the best competing centroid cosine. * **centroid separation** — mean pairwise cosine between the 24 centroids. * **per-fact accuracy** — and how many of the 24 facts have at least one of their two held-out paraphrases correct. The complete 29 × 3 matrix is recorded. The best cell is not reported alone; the trajectory across depth is itself a result. --- ## Pre-committed interpretation thresholds Fixed here, before the sweep. Not adjusted afterwards. ### Strong addressable representation A cell qualifies only if **all five** hold: | | criterion | threshold | |---|---|---| | **S1** | held-out 24-way fact accuracy | **≥ 0.90** | | **S2** | not explained by answer leakage | held-out prompts contain no answer token — structural, verified | | **S3** | not merely lexical identity | accuracy **exceeds row 0 (embeddings) at the same position**. If row 0 at that position already reaches ≥ 0.90, the position is **lexically determined** and no cell in that column may be cited as paraphrase-stable addressing; it is reported separately as string recognition | | **S4** | both relations represented | fact accuracy **≥ 0.90 within hop 1** *and* **≥ 0.90 within hop 2** (24 probes each) | | **S5** | distributed across facts | **≥ 18 of 24** facts have at least one of their two held-out paraphrases correct | ### Partial representation Best held-out fact accuracy **materially exceeds chance but is < 0.90**. "Materially exceeds chance" is fixed as a one-tailed binomial test, `P(X ≥ k | n = 48, p = 1/24) < 0.01`, computed exactly. ### No useful representation No cell materially exceeds chance, **or** every cell that does is explained by lexical identity under S3. --- ## Validation gates, all before the sweep The sweep does not run unless every one passes. A failure stops the experiment for diagnosis. | # | gate | |---|---| | V1 | frozen base fingerprint is `52683d0e840fb51e` | | V2 | the Experiment 007 split reproduces byte-identically from committed code | | V3 | Experiment 007's layer-18 / last-prompt nearest-centroid result reproduces as **2/48 = 0.0417**, with its recorded confusion buckets | | V4 | no parameter has `requires_grad=True` | | V5 | no optimiser is constructed anywhere in the experiment | | V6 | no answer token enters any swept representation — asserted on token indices, not assumed | | V7 | cue-entity alignment unit test passes on all 192 examples, including multi-token entities | | V8 | `hidden_states[18]` is bit-identical to the layer-18 block input Experiment 007 captured through the memory module | V8 is not in GOAL_8's list. It is added because it is the check that actually proves the two pipelines are extracting the same tensor, and it is free. --- ## What this experiment will not do Whatever the result: no memory is attached at a winning layer, no projection is learned, no probe is tuned, no architecture is tested, no threshold is revised, and Phase II is not begun. GOAL_8_FINAL's stop condition is unconditional. Experiments 000–007 artifacts are not modified. Shared analysers are not modified. Implementation and results are committed separately.