We set out to make a small model better at security by fine-tuning it on a large corpus of security material — advisories, vulnerability records, writeups, exploit research. Tens of gigabytes of it. By the obvious measure it worked: the finetuned model is markedly more fluent about security than its base. It says the right things about injection, access control, tokens, offsets.
Then we put it inside an agentic harness and asked it to actually do security — and it couldn't. Not “did it worse.” Couldn't. The base model it was finetuned from can.
A blind man can study drawing for years — composition, perspective, colour — and understand all of it. He still can't put brush to canvas. The finetune taught the model about security; it took away the hand.
The controlled experiment
Everything held fixed except the finetune:
- Finetuned model — a 3B instruct model, further trained on ~50 GB of (largely declarative) security text.
- Base model — the identical 3B instruct model, stock, no finetune.
- Same GPU, same quantisation, same agentic harness, same prompts, same tasks.
The tasks are two rungs of a security ladder that require the model to construct an action (not just invoke a tool), run with all scaffolding disabled so only the model's own behaviour is exercised.
| Model (same base, same size) | Access-control rung | Chained-reuse rung |
|---|---|---|
| Finetuned on 50 GB security corpus | 0/3 — loops, never acts | 0/3 — emits a placeholder |
| Stock base (no finetune) | 3/3 | 2/3 |
Same weights underneath. The only difference is the corpus, and it flipped the model from capable to incapable on the identical task.
What the failure looks like — this is the important part
The finetuned model is not confused about the task. Its plan narrates the right idea in fluent security language — “exploit the broken access control,” “reuse the admin credential.” Then it cannot carry it out:
- On the access-control rung it re-reads the same two pages in a loop and never constructs the modified request.
- On the chained rung it issues the second request with the literal placeholder from its own instructions still in it — the string <ADMIN_TOKEN> where the real value should go, as if the words “the admin token” were the token. In most runs it didn't even fetch a real token first; it went straight to re-issuing the placeholder until it ran out of turns.
It knows what to do and cannot do it. Knowledge intact; execution gone. The brush.
Why it likely happens
Hypotheses, not yet isolated, in rough order of our suspicion:
- The corpus is declarative, not agentic. Fifty gigabytes of prose about security contains almost no traces of acting — tool calls, act-then-observe sequences. Training pulls the model toward producing security writing and away from the terse, structured actions an agent has to emit.
- Catastrophic forgetting of instruction-following. A full finetune over a large corpus can erode the base model's tool-use and instruction-following behaviour learned in post-training.
- Format collapse. The model increasingly answers with explanation where the base emitted a structured action, so a valid action never gets produced.
Why it matters
The dangerous part is that this was invisible to the obvious evaluation. The retrain was judged on knowledge — does it know more security? — and it passed. The faculty it lost, the ability to act on that knowledge, was never on the eval, so the regression didn't show up until an agentic harness exercised it. If you finetune a small model for a domain and only measure knowledge, you can ship a model that has quietly lost its hands and never know.
It is also a direct warning for the reflex of “make the small model a better agent by training it on more domain data.” That can achieve the opposite. Agency is not built by pouring in declarative knowledge.
What we'd do instead
- Prefer light adapters trained on agentic traces (tool-call trajectories, act-then-observe sequences) over heavy full finetunes on declarative corpora.
- Keep an agentic eval in the loop during any domain finetune — measure doing, not only knowing.
- If both knowledge and agency must rise, treat them separately — retrieval for knowledge, trace-tuning for agency — rather than one large corpus.
Caveats
Small N; one base/finetune pair; one task family; quantised inference. We have not yet isolated whether the finetuned model also fails generic tool-use or only here, and the training recipe matters to the story. Treat this as a sharp, reproducible warning and a direction for careful follow-up — not a universal law about finetuning.
← Back to lab notes