# GOAL_1 Progress — 2026-08-22

Session record for `GOAL_1.md` (reducing dynamic memory interference, Experiment 001).
Formal deliverable: `dynamic-memory-lab/results/experiment_001/experiment_001_report.md`.

**GOAL_1: complete — 15 adapter configurations measured, winner validated, report written.**

### Bottom line

    Recall        1.0000  (48/48)          target >= 0.90    MET
    Interference  0.0000  (9/12 = base)    target <= 0.10    MET
    Composition   0.1667  (4/24, thinking) above baseline    NOT MET
    Adapter       9.68 MiB / 2,523,136 params (0.423%)       4x smaller than Exp 0

**The interference target is achievable. It cannot currently be had together with
compositional reasoning.**

Interference was driven to **exactly zero** — the adapter-attached model answers the
control set identically to the frozen base — with recall at ceiling from a 4× smaller
adapter. But composition fell to 4/24, precisely the 1/6 chance level for guessing
among the six radiation names the memory learned (`P(X ≥ 4 | n=24, p=1/6) = 0.58`).
Experiment 0's 10/24 was significantly above chance (p = 0.0033); **no**
low-interference configuration is.

Two of GOAL_1's three criteria are met. The composition criterion is not.

---

## The main finding contradicts the stated premise

GOAL_1 assumed Experiment 0's adapter was too large and that shrinking it would
reduce damage. Along the module axis the opposite is true. At rank 8, epoch 8:

| target modules | trainable | adapter | recall | **interference** |
|---|---|---|---|---|
| `q,v` (2) | 1,146,880 (0.192%) | 4.40 MiB | 0.9792 | **0.5833** |
| `q,k,v,o` (4) | 2,293,760 (0.385%) | 8.78 MiB | 1.0000 | **0.5000** |
| all 7 | 5,046,272 (0.847%) | 19.31 MiB | 0.9792 | **0.0833** |

The **smallest, most targeted adapter is by far the most destructive** — 0.5833,
nearly double Experiment 0's 0.3333 — while still reaching ceiling recall.
Confining the same knowledge to two projection matrices appears to require pushing
those matrices much harder, and that is what damages the reasoner.

Spearman correlations against interference (16 configurations, incl. the Exp 0 reference):

| predictor | ρ |
|---|---|
| **number of target modules** | **−0.871** |
| trainable parameters | −0.573 |
| adapter rank | +0.326 |
| epochs trained | +0.017 |

Adapter size is a **sign-flipping** variable and therefore the wrong thing to
optimise: across module scopes more parameters means *less* interference (−0.573),
while within a fixed all-7 scope more parameters means *more* (+0.840, n=10).
**Interference is governed by parameter location, not size.**

## Mechanism — inspected, not inferred

Aggregate scores say how much broke, not what. The distinctive failure is the
**synthetic world leaking into unrelated questions**. Under `S1_qv_r8` (I = 0.5833):

| question | base | with memory |
|---|---|---|
| capital of France | `Paris.` | **`ZHAELK-50`** |
| who is youngest | `Rey.` | **`VLUUZL-79`** |
| what day in two days | `Wednesday.` | **`THUZYU-77`** |
| 24 × 3 | `72` | `52` |

Counting world-shaped tokens in control answers tracks the metric monotonically:

| I | 0.0000 | 0.0833 | 0.1667 | 0.2500 | 0.3333 | 0.5833 |
|---|---|---|---|---|---|---|
| leaked answers | 0 | 1 | 1 | 1 | 2 | 4 |

Leakage is not the whole story — arithmetic also corrupts — but it is the signature
failure and it scales with the metric. Interference here is the memory
**over-generalising**: the adapter biases the model toward emitting a world entity
whenever asked for a short name.

## Generation stability recovers — and that is the tradeoff

| | mean thinking tokens | max | truncations |
|---|---|---|---|
| base model | 286.5 | 584 | 0 |
| Experiment 0 (I = 0.3333) | 334.6 | 1024 | 3 |
| **`S5_all_r4_ep12` (I = 0.0000)** | **185.1** | **233** | **0** |

Experiment 0's loopy, occasionally-truncating traces are gone; the low-interference
adapter reasons *more* stably than the base model. But Experiment 0's longer,
exploratory traces are what found the 2-hop answers. **The perturbation that damaged
control is the same perturbation that drove the reasoning that succeeded.**

---

## What was run

Base model re-validated first: recall 0/48, composition 0/24, control 9/12 (0.7500) —
**identical to Experiment 0**, confirming the numerical environment is unchanged and
that GOAL_1 numbers are directly comparable to that record.

### Stage 1 — four runs, one variable each (not a 27-cell sweep)

| ID | modules | rank | isolates | net train |
|---|---|---|---|---|
| S1 | `q,v` | 8 | smallest plausible memory | 841 s |
| S2 | `q,k,v,o` | 8 | attention scope (vs S1) | 1029 s |
| S3 | all 7 | 8 | + MLP scope (vs S2), rank fixed | 1299 s |
| S4 | all 7 | 16 | rank (vs S3), modules fixed | 1226 s |
| Exp 0 | all 7 | 16 | epochs — 30 vs S4's 12 | (reference) |

Checkpoints at epochs 8/10/12 gave the epoch axis for free: **12 evaluated
configurations from 4 training runs**. Checkpoint saving cost 0.20–0.59 s per run and
does not touch the optimizer, RNG or data order.

### Stage 2 — rank 4

Stage 1 gave three monotone trends (interference falls with module scope, rises with
rank, rises with epochs at rank 8). The only untested cell all three agree on is
lower rank at full module scope: **S5 = all 7 modules, rank 4, 12 epochs**, 1274 s.

### Stage 3 — thinking-mode composition on finalists only

Thinking-off screening costs ~110 s per configuration and yields interference, the
primary metric. Thinking-on costs ~14–24 min. It was spent only on the three
configurations that survived the screen.

| config | I | C_on | p vs 1/6 chance | p vs base 0/24 |
|---|---|---|---|---|
| Exp 0 (r16, 30 ep) | 0.3333 | 10/24 | **0.0033** | 0.0003 |
| `S3_all_r8_ep8` | 0.0833 | 4/24 | 0.5845 | 0.0546 |
| `S5_all_r4_ep10` | 0.0833 | 3/24 | 0.7882 | 0.1170 |
| `S5_all_r4_ep12` | 0.0000 | 4/24 | 0.5845 | 0.0546 |

### Eliminations (recorded in `pruning_decisions.md`)

- **S1 (`q,v`)** — dominated by S3: worse interference at every epoch, no recall gain.
- **S2 (attention-only)** — same; also the worst recall measured (0.9167 at ep12).
- **S4 (rank 16)** — worse than rank 8 at matched modules/epochs; kept as rank anchor.
- Ranks below 4 and further scope reduction not explored: the module axis already
  resolved in the *opposite* direction from "smaller is gentler".

## Winning configuration

`S5_all_r4_ep12` — rank 4, all 7 projection modules, 12 epochs, lr 2e-4, seq 256,
batch 1 × grad-accum 2, gradient checkpointing.

    2,523,136 trainable (0.423% of base)    9.68 MiB
    loss 3.2705 -> 0.1232                   21.2 min training
    peak 5.139 GiB system / 1.293 GiB CUDA
    recall 1.0000 (48/48)   control 9/12 (0.7500)   I = 0.0000

### Detachability — verified

| State | Weights | Recall | Composition | Control |
|---|---|---|---|---|
| 3 load | `W_core + ΔW_A` | 1.0000 | 0.1667 | 0.7500 |
| 4 unload | `W_core` | 0.0000 | 0.0000 | 0.7500 |
| 5 reload | `W_core + ΔW_A` | 1.0000 | 0.1667 | 0.7500 |

Unload removes the knowledge completely and restores base behaviour exactly; reload
restores it exactly; base-weight fingerprint identical throughout
(`52683d0e840fb51e`); never merged. Load 0.635 s / unload 0.104 s / reload 0.772 s.
Control is 0.7500 in **all three** states — this memory is invisible to the reasoner.

---

## The ten required answers (condensed)

1. **How small before recall degrades?** It never degraded. All 15 configurations
   scored ≥ 0.9167; the smallest (0.192%, 4.40 MiB) reached 0.9792.
2. **How small before composition degrades?** Composition degrades before size bites —
   *every* GOAL_1 config sits at chance, including one at Experiment 0's exact size.
   The difference is 12 vs 30 epochs, not adapter size.
3. **Does lower rank reduce interference?** Yes, within fixed scope (ρ = +0.840).
   r16 worse than r8 at matched epochs; r4 vs r8 mixed and within 1–2 questions.
4. **Does narrower module scope reduce interference?** **No — it sharply increases it.**
   Clearest effect measured (ρ = −0.871), contradicting the hypothesis.
5. **Does shorter training reduce interference?** Not reliably (ρ = +0.017). At rank 8
   it rises with epochs (0.0833→0.25); at rank 4 it *falls* (0.1667→0.0000).
6. **Size, location, or exposure?** **Location, decisively.**
7. **Best tradeoff?** `S5_all_r4_ep12`. If composition is weighted above interference,
   Experiment 0 remains the only config that composes — at 4× the size and I = 0.3333.
8. **Criteria satisfied?** Recall ✓, interference ✓ (0.0000, ideal), composition ✗.
9. **Does unload restore original behaviour?** Yes, exactly, weights bit-identical.
10. **Implication?** Detachable memory can be made **completely non-invasive** — zero
    interference, perfect reversibility, 0.42% of parameters. The write/load/unload
    primitive is sound. But in this space, *the settings that make a memory harmless
    also make it inert for reasoning*. Storage and reasoning-participation are being
    traded against each other, not against parameter budget.

---

## Corrections recorded

1. **The stated loss premise is wrong.** GOAL_1 (repeating my own Experiment 0 report)
   says loss reached ~0.09 by epoch 10. The recorded curve gives **0.1274** at epoch
   10; ~0.09 arrives around epoch 24. 94.9% of the descent is done by epoch 10, so the
   qualitative point stands, but the 8–12 window is still descending — making the
   duration comparison a test of *undertraining vs overtraining*, not equal-loss.
2. **S4's first 12 epochs reproduce Experiment 0's loss curve bit-identically**
   (2.4960 … 0.1127). Same seed, same data order, same init. The epoch axis is
   therefore exactly controlled and the pipeline is confirmed deterministic.
3. `run_states.py` hardcoded the Experiment 0 results directory; a tagged run now
   writes to `experiment_001`. The one misplaced artifact was moved. **No Experiment 0
   artifact was modified or overwritten.**

## Honest limits

- **The control set is 12 questions.** One item is 0.083 of interference, so the
  separations between 0.0000 / 0.0833 / 0.1667 rest on one or two answers. The large
  separations (0.0833 vs 0.5833, seven questions) are solid. Evaluation is
  deterministic (greedy, fixed seed), so these are real behavioural differences — but
  weak evidence about the underlying rate.
- **The composition set is 24 questions.** All GOAL_1 composition figures (3–4/24) are
  statistically indistinguishable from each other and from chance.
- Non-monotonicities are left visible, not smoothed: `S4_all_r16` gives
  0.2500 / 0.3333 / 0.2500 across epochs 8/10/12, and rank 4's epoch trend runs
  opposite to rank 8's.
- One model, one world, one seed, 2-hop only. `S5_all_r4_ep12` was trained once.

## State of the machines

| Host | State |
|---|---|
| **jetson** 10.0.2.3 | all GOAL_1 training and evaluation; 21 adapters in `models/adapters/` (20 from GOAL_1 + Experiment 0's `memory_a`); environment unchanged from Exp 0 |
| **henry** 10.0.2.2 | repository + full git history, synced |
| **jetson2** 10.0.2.4 | still untouched — reserved for independent validation |

Deliverables in `results/experiment_001/`: `experiment_001_report.md`, `methods.md`,
`pruning_decisions.md`, `results.json`, `interference_analysis.json`, 19 raw
`eval_*.json`, 5 `train_memory_*.json`, `states345_goal1_winner_*.json`.

## Next step — not run (GOAL_1 exit condition reached)

The tradeoff, not the interference, is now the open question. The most informative
follow-up **breaks the coupling** rather than tuning along it: train the
low-interference geometry (all 7 modules, rank 4) for Experiment 0's full **30
epochs**. That separates "composition needs training exposure" from "composition
needs a large, damaging delta".

- If 30 epochs at rank 4 composes **and** keeps I ≤ 0.10 → GOAL_1 succeeds outright.
- If composition returns only with interference → the coupling is real, and the next
  architecture needs stronger parameter isolation than LoRA offers. That is itself
  the important result.

Second, cheaper probe: the leakage mechanism suggests the memory over-generalises to
"emit a world entity when asked for a short name". Training data containing non-world
questions with non-world answers may suppress leakage directly, without touching
adapter geometry.