# GOAL_4 Progress — 2026-08-23

Session record for `GOAL_4.md` (can memory live outside the frozen projections?,
Experiment 004).
Formal deliverable: `dynamic-memory-lab/results/experiment_004/experiment_004_report.md`.

**GOAL_4: complete — one isolated-memory architecture built, isolation proven
before training, trained, all six states evaluated, gate sweep run, detachability
verified, four-way comparison done. Exit condition reached.**

### Bottom line

    Parameter isolation did not deliver behavioural isolation. It made things worse.

The isolated module **learned the world perfectly** (48/48 recall, touching no base
weight) and **detaches perfectly**. But it is the **most disruptive** configuration
this project has measured, on every damage axis at once, and its compositional
reasoning is the **worst**: 0/24.

This does not map cleanly onto GOAL_4's Outcomes A–D. Recall did not fail (not D);
composition is not significant (not A or C); interference and churn are not low
(not B). The honest label: **storage succeeded, isolation failed, reasoning
participation collapsed.** Outcome C's *conclusion* — simple residual isolation is
insufficient — holds without Outcome C's premise.

---

## The primary comparison (all at epoch 30)

| | LoRA r4 | LoRA r8 | LoRA r16 | **Isolated memory** |
|---|---|---|---|---|
| Mechanism | delta **on** base projections | ” | ” | **branch outside them** |
| Memory parameters | 2,523,136 | 5,046,272 | 10,092,544 | **2,811,200** |
| % of base | 0.423% | 0.847% | 1.693% | **0.4716%** |
| On-disk size | 9.68 MiB (fp32) | 19.31 MiB | 38.56 MiB | **5.4 MiB (bf16)** |
| **Recall** | 1.0000 | 1.0000 | 1.0000 | **1.0000 (48/48)** |
| Composition (thinking) | 5/24 | 4/24 | **10/24** | **0/24** |
| p vs 1/6 chance | 0.37 | 0.58 | **0.0033** | **1.00** |
| Hop-1 in trace | 20/24 | 23/24 | 24/24 | **17/24** |
| Hop-2 in trace | 10/24 | 6/24 | 14/24 | **6/24** |
| Control correct | 8/12 | 6/12 | 5/12 | **3/12** |
| **Interference** | 0.0833 | 0.2500 | 0.3333 | **0.5000** |
| Churn: changed / substantive | 11 / 8 | 10 / 10 | 10 / 9 | **12 / 9** |
| Regressions / **repairs** | 3 / 2 | 5 / 2 | 6 / 2 | **6 / 0** |
| Leakage | 0 | 1 | 2 | **3** |
| Mean reasoning tokens | 196.7 | 152.1 | 334.6 | **528.6** |
| Truncations | 0 | 0 | 3 | **9** |

Worst or joint-worst in interference, churn, regressions, leakage, trace length,
truncations, hop-1 retrieval **and** composition — while using fewer parameters than
everything except rank-4 LoRA.

Note: on-disk size is **not** comparable across mechanisms (memory saves bf16, LoRA
saved fp32). **Parameter count is the fair comparison**; 2.81M vs 2.52M is close.

## The six states

| State | Configuration | Recall | C off | C on | Control | **I** |
|---|---|---|---|---|---|---|
| 1 | frozen base, nothing installed | 0/48 | 0/24 | — | 9/12 | 0.0000 |
| 2 | installed, untrained, gate = 0 | 0/48 | 0/24 | — | 9/12 | 0.0000 |
| 3 | *train the module* | — | — | — | — | — |
| 4 | trained, **gate = 1** | **48/48** | 0/24 | **0/24** | **3/12** | **0.5000** |
| 5 | trained, **gate = 0** | 0/48 | 0/24 | — | **9/12** | 0.0000 |
| 6 | trained, **gate = 1** again | **48/48** | 0/24 | — | **3/12** | **0.5000** |

State 2 reproduces State 1 exactly; State 5 restores the frozen model exactly with
**zero** of 12 control answers differing; State 6 reproduces State 4 exactly.

## Architecture

Residual bottleneck adapter, one per decoder layer, as a **parallel branch**:

    base_out = frozen_layer(h)                    # untouched
    mem_out  = up(GELU(down(RMSNorm(h))))         # only trainable parameters
    out      = base_out + gate * mem_out

| component | params/layer |
|---|---|
| RMSNorm (trainable) | 1,024 |
| down 1024→48 + bias | 49,200 |
| up 48→1024 + bias | 50,176 |
| **per adapter** | **100,400** |
| **× 28 layers** | **2,811,200** (0.4716%) |

`up` zero-initialised so an untrained module is an exact no-op. Gate is a plain
float, **not** short-circuited at 0 — the addition is still performed, so the gate
test exercises the real forward path.

**All 28 layers, deliberately.** LoRA touched all 28, so matching coverage isolates
*isolation* rather than confounding it with placement breadth — and removes the
Outcome-D ambiguity where a placement failure would masquerade as an isolation
failure. Installation patches only the bound `forward` and stashes the original, so
`remove_memory()` restores the model exactly.

## Training (State 3)

Same handwritten loop, data, order, optimizer and hyperparameters as Experiments
0–3; `transformers.Trainer` still unusable (JetPack torch has no `torch.distributed`).

    loss  3.5763 -> 0.1157  over 30 epochs
    net wall 2145.1 s (checkpoint saves 0.14 s)
    peak 5.132 GiB system / 1.276 GiB CUDA
    base-weight fingerprint 52683d0e840fb51e before AND after -> unchanged

Training refuses to start if any non-memory parameter is trainable, and asserts the
trainable count equals the memory count. Final loss is higher than LoRA's
(0.1157 vs 0.0896–0.0923).

## Isolation proven before training

Seven checks, all passed (`base_isolation_validation.md`). At logit level on three
fixed prompts:

| check | max \|logit diff\| vs frozen base |
|---|---|
| adapters installed, zero-init, **gate = 0** | `0.000e+00` |
| adapters **randomised**, gate = 0 | `0.000e+00` |
| same randomised adapters, **gate = 1** | `7.922e+00` |
| `remove_memory()` | `0.000e+00` |

**Why the randomised check matters:** a zero-initialised module passing gate=0
proves nothing — the contribution is zero either way. Giving the adapters non-zero
weights isolates the gate itself, and the gate=1 check then confirms the branch is
genuinely wired in rather than never connected.

Plus: every base tensor frozen, every memory tensor trainable, zero base tensors
still trainable after install.

## What the traces show

The failure is **generation instability**, not a subtle reasoning gap.

- **528.6 mean tokens** (min 108, max 1024) — 2.7× rank 8, 1.6× rank 16, the longest
  ever measured.
- **9 of 24 traces hit the 1024-token cap** — rank 16 had 3, ranks 4 and 8 zero.
- **Hop-1 retrieval falls to 17/24** — worse than every LoRA variant including rank
  4. The module degrades the *first* retrieval, which LoRA handled at 20–24/24.
- Repeated-shingle loopiness is 0.0: traces are long and wandering, not repeating.

Reported honestly: 0/24 conflates reasoning failure with truncation. Of the **15
non-truncated** items, **0** were graded correct. One answer did contain the correct
radiation but also named a competing one and was rejected by the strict no-hedging
rule used since Experiment 0 — so 0/24 stands as the graded result.

## Churn and leakage

| | changed | substantive | regressions | **repairs** |
|---|---|---|---|---|
| LoRA r4 | 11 | 8 | 3 | 2 |
| LoRA r8 | 10 | 10 | 5 | 2 |
| LoRA r16 | 10 | 9 | 6 | 2 |
| **Isolated** | **12** | 9 | 6 | **0** |

Every LoRA configuration accidentally *repaired* two control questions the base got
wrong. The isolated module repairs **none** and changes **all twelve** answers. Its
perturbation is purely destructive.

Leakage is the highest measured — **3** world-shaped control answers (r16: 2, r8: 1,
r4: 0). Reducing exactly this failure mode was one motivation for isolation; it
increased it.

## Gate sweep — a switch, not a dial

Same trained module, no retraining:

| gate | recall | control | **I** |
|---|---|---|---|
| 0.00 | 0/48 | 9/12 | 0.0000 |
| 0.25 | **0/48** | 9/12 | 0.0000 |
| 0.50 | **0/48** | 7/12 | **0.1667** |
| 1.00 | **48/48** | 3/12 | **0.5000** |

The two curves are **misaligned**. Damage is already measurable at gate 0.5 where
recall is still exactly zero; recall appears only at gate 1.0, where damage is
maximal. **No intermediate setting buys knowledge at reduced cost.**

## Why separate parameters were not enough

The module never touches `q/k/v/o/gate/up/down_proj`. But it adds an **unconstrained
full-rank vector to the residual stream at every one of the 28 layers**, and that
stream *is* the reasoning process. LoRA perturbs specific projections within a
low-rank subspace; this branch injects an unconstrained displacement into the
channel the model reasons through.

Physical separation bought nothing behaviourally because the **interface was just as
wide**. That is the experiment's real content: isolation has to be **functional** — a
constraint on where and how memory may write into the computation — not merely a
question of which tensors the gradients touch.

## The fourteen required answers

1. **Learned the world?** Yes, completely. Loss 3.5763 → 0.1157; recall 48/48.
2. **Recall ≥ 0.90?** Yes — 1.0000.
3. **Composition above chance?** No. 0/24, `p = 1.00`. Worst of any configuration and no better than the frozen base.
4. **Hop-2 improved?** No. 6/24, tied worst with rank 8 against rank 16's 14/24. Hop-**1** also regressed, to 17/24.
5. **Interference?** **0.5000** (control 3/12) — worst measured; 6× rank 4, 1.5× rank 16.
6. **Substantive churn?** 9 of 12 substantively changed, 12 of 12 changed at all, 6 regressions, **0 repairs**.
7. **Leakage?** Yes, most of any configuration: 3 world-shaped answers.
8. **Does gate = 0 reproduce the frozen model?** Yes — bit-identical logits including with randomised weights, and zero control-answer churn.
9. **Does disabling remove the knowledge?** Yes. 48/48 → 0/48, control back to 9/12.
10. **Does re-enabling restore it?** Yes. State 6 reproduces State 4 exactly.
11. **Compared with LoRA?** Equal on recall (all four at ceiling), **worst on everything else**.
12. **Better ratio of useful behaviour to disruption?** **No — strictly worse.** Same recall, zero composition, 2× rank 8's interference and 6× rank 4's.
13. **Remaining bottleneck?** The **write interface**, not the parameter store. Any mechanism adding an unconstrained vector to the residual stream at every layer perturbs reasoning as much as editing projections does. Unchanged since GOAL_2: making a retrieved intermediate usable as a query for a second retrieval.
14. **Continue or reconsider?** Continue the direction, **not this design**. The isolation/detachability machinery works and is worth keeping; the naive parallel residual is refuted. Next should **narrow the interface** — far fewer insertion points, or a restricted write channel — not another unconstrained branch.

## Honest limits

- **24 composition, 12 control questions.** One control question is 0.083 of `I`.
  The interference gap to rank 16 (3/12 vs 5/12) is two questions — real but small.
  Composition 0/24 vs rank 16's 10/24 is the one large, unambiguous gap.
- **9 of 24 composition items truncated**, so 0/24 conflates reasoning failure with
  generation instability. The non-truncated subset is 0/15, so the conclusion holds,
  but this experiment does not separate the two causes.
- **One architecture, one bottleneck size, one placement, one seed, one world.**
  GOAL_4 permitted a capacity adjustment if recall failed; recall did not fail, so
  none was made and none is reported.
- **Hyperparameters inherited from the LoRA experiments by instruction, not tuned**
  for this mechanism. Final loss is higher (0.1157 vs 0.0896–0.0923); a fairer
  schedule might close some of that — though not plausibly a 0/24 → 10/24 gap.
- Determinism does not imply generality: one world and one seed do not estimate
  cross-world variance.

## State of the machines

| Host | State |
|---|---|
| **jetson** 10.0.2.3 | all GOAL_4 training and evaluation; 34 entries in `models/adapters/` (1 new from GOAL_4: `memory_M1_iso_b48/` holding checkpoints ep12/ep20/ep30 + final); environment unchanged since Experiment 0 |
| **henry** 10.0.2.2 | repository + full git history, synced |
| **jetson2** 10.0.2.4 | still untouched — reserved for independent validation |

Deliverables in `results/experiment_004/` (20 files): `experiment_004_report.md`,
`methods.md`, `architecture.md`, `base_isolation_validation.md`,
`detachability_validation.md`, `results.json`, `gate_sweep.json`,
`trace_analysis.json`, `behavioral_churn.json`, `leakage_analysis.json`,
`isolation_validation_*.json`, 8 raw `eval_*.json` (states 1/2/4/5/6, gate 0.25,
gate 0.5, thinking-on), `train_M1_iso_b48_*.json`.

New code: `src/dynamic_memory/isolated_memory.py`,
`scripts/validate_isolation_004.py`, `scripts/train_isolated_004.py`,
`scripts/eval_memory_004.py`, `scripts/aggregate_004.py`. Experiments 000–003
artifacts **not modified**.

## Next step — not run (exit condition reached)

GOAL_4's stop condition is explicit: run one isolated architecture, train, evaluate,
verify, compare, then stop. No further optimisation, no next architecture.

The decision now on the table is whether to pursue a **narrower write interface**
(fewer insertion points, or memory constrained to a restricted channel) or to
reconsider the hypothesis that a frozen reasoner can host detachable memory at all
under this family of mechanisms.