Inference Should Learn: An Experiment in Dynamic Memory for Language Models
An ongoing experiment into whether language models can acquire knowledge by dynamically changing their own parameters.
Phase I ran nine experiments to move the middle number and could not. The reason was found later, in the sequel: composition was never the bottleneck it appeared to be.
The Idea
Modern large language models have an unusual limitation.
They can reason about new information. They can retrieve information. They can hold enormous amounts of information in their context window.
But once training ends, the model itself is largely frozen.
Its parameters might contain billions of learned relationships, but during normal inference those parameters do not change.
A model can discover something today and effectively know nothing about it tomorrow unless we provide that information again through context, retrieval, external memory, or another training process.
That raises a simple question:
Why should learning and inference be separate processes?
Our hypothesis is that they do not have to be.
The Hypothesis
Instead of treating the parameters of a language model as static, we want to investigate a model consisting of two parts:
- A relatively stable core containing language and reasoning capabilities.
- A dynamic portion of the model containing knowledge acquired during its operational lifetime.
The long term idea is simple:
If the model encounters something it does not know, it should be able to learn it.
Not merely place it into the current prompt.
Not simply retrieve it again next time.
Actually learn it.
The model should eventually be capable of identifying a gap in its knowledge, finding information about it, evaluating that information, learning what is useful, and retaining that knowledge for future reasoning.
Dynamic Memory
Today, a model encountering an unknown subject might follow a process like this:
Question
Search
Add information to context
Answer
Once that context disappears, the model has not fundamentally changed.
We want to investigate a different process:
Question
Recognize a knowledge gap
Acquire information
Evaluate it
Learn it
Answer
Retain what was learned
The important difference is the learning step.
New information would cause a controlled change to part of the model itself.
Those changes could eventually behave like dynamic memories. They might be created, loaded, unloaded, combined, transferred, consolidated, or forgotten.
This leads to the central hypothesis behind the project:
Inference can become a learning process.
Why Start Small?
There are many ways to make this idea unnecessarily complicated.
We could immediately introduce web search, autonomous agents, vector databases, long term memory systems, large models, distributed training, and complicated infrastructure.
We are deliberately doing none of that.
The first question is much smaller:
Can a frozen reasoning model acquire previously unknown information through a detachable parameter update and subsequently reason using that information?
If the answer is no, we want to discover that quickly.
If the answer is yes, we can build from there.
Experiment 0: Writable Parametric Memory
We are starting with one small reasoning model.
The original model remains frozen.
We then create a small, separate set of trainable parameters containing newly learned information.
For the first experiment, we are using LoRA adapters to create these parameter changes.
LoRA itself is not the hypothesis and is not being presented as a new memory architecture.
It is simply useful laboratory equipment.
It gives us a practical way to modify a small number of parameters without retraining the underlying model.
This lets us test the basic idea before designing a new architecture.
Teaching the Model Something That Does Not Exist
We need to know with certainty whether the model learned something during our experiment rather than remembering something from its original training.
So we will create artificial worlds.
For example:
NARU-17 is a planet.
KEL-44 is the largest moon of NARU-17.
KEL-44 contains ZETA-4.
Objects containing ZETA-4 emit VIOLET radiation.
These names and relationships are generated specifically for the experiment.
First, we test the untouched model.
It should not know these facts.
Then we teach the detachable memory.
After training, all of the original teaching material is removed from the model's context.
We ask questions again.
If the model can now recall the information, we have demonstrated that knowledge was acquired by the parameter memory.
But recall is only the first test.
Memory Is Not Enough
The more important question is whether the model can reason with what it learned.
For example, we never explicitly teach the model this statement:
"The largest moon of NARU-17 emits VIOLET radiation."
Instead, the model has to combine several independently learned facts.
NARU-17 has a largest moon named KEL-44.
KEL-44 contains ZETA-4.
Objects containing ZETA-4 emit VIOLET radiation.
Therefore, the largest moon of NARU-17 should emit VIOLET radiation.
That conclusion was never part of the training material.
The reasoning capability comes from the original model.
The premises come from the newly acquired memory.
That distinction is central to the experiment.
We do not merely want:
New weights + memorization
We want:
New knowledge + existing reasoning = new conclusions
The Detachment Test
After teaching the artificial world, we remove the parameter memory.
The synthetic knowledge should disappear.
Load the memory again and the knowledge should return.
This gives us a very simple experimental test.
Base model:
Does not know NARU-17.
Base model + Memory A:
Knows NARU-17 and can reason about it.
Base model:
Memory A removed. Knowledge disappears.
Base model + Memory A:
Memory restored. Knowledge returns.
If that works reliably, we have demonstrated a primitive form of loadable neural memory.
The Next Important Test
Suppose two memories are learned independently.
Memory A knows one part of an artificial world.
Memory B knows another part.
Neither memory contains enough information to answer a particular question independently.
We then load both.
Base model + Memory A + Memory B
Can the underlying reasoning model combine information distributed across independently acquired parameter memories and reach a conclusion that neither memory contains?
If it can, that becomes significantly more interesting than simple memorization.
It suggests that separately acquired neural memories can participate in a common reasoning process.
That will be a later experiment.
First, Experiment 0 needs to work.
The Hardware
We are intentionally starting with modest hardware.
The lab currently consists of a Ryzen based CPU server with 32 GB of RAM and two NVIDIA Jetson systems with approximately 8 GB of unified memory each.
We are starting with a very small reasoning model, currently targeting Qwen3-0.6B.
The hardware constraint is intentional.
If the underlying mechanism is interesting, we should be able to demonstrate its basic properties without requiring a rack of high end GPUs.
We are trying to discover a principle, not train a frontier model.
What Would Count as Success?
Experiment 0 has four basic tests.
- Write
Can new synthetic knowledge be encoded into a small detachable parameter memory?
- Recall
Can the model recover that knowledge later without seeing the original teaching context?
- Reason
Can the model use its existing reasoning capabilities to derive conclusions from newly acquired knowledge that were never explicitly included in training?
- Forget
When the parameter memory is removed, does the acquired knowledge disappear while the underlying model remains intact?
The third test is the most important.
A system that simply memorizes additional strings is not particularly interesting.
A system that can acquire new premises and immediately integrate them into its existing reasoning process potentially is.
What We Are Not Claiming
This experiment does not demonstrate that LoRA adapters are a new form of AI memory.
They are not.
It does not demonstrate continual learning.
It does not demonstrate that dynamically changing weights is superior to retrieval augmented generation.
It does not demonstrate a replacement for the Transformer.
We are starting with a much narrower question.
Can reasoning and dynamically acquired parametric knowledge be separated and recombined?
If Experiment 0 fails, we will document why.
If it succeeds, we will make the next experiment harder.
The nine experiments
Each row is one attempt on the same problem. The full write-ups are in the log below.
| ID | What it tried | Outcome | Key figure |
|---|---|---|---|
| 000 | Can a frozen model acquire facts through a detachable delta? | Yes — and reason with them, partially | 1.0000 recall |
| 001 | Make the memory non-invasive | Interference gone, composition fell to chance | 0.0000 interference |
| 002 | Does composition need more exposure, or more capacity? | Capacity, not exposure | 5 / 24 |
| 003 | Does a middle rank get both? | No — rank-16 damage, rank-4 benefit | 0.2500 interference |
| 004 | Put memory outside the frozen projections entirely | Worse — worst interference measured | 0.5000 |
| 005 | Give the memory an explicit address | Failed to address, not to store | 1 slot, 4285/4287 |
| 006 | Stop the address collapsing | Regulariser worked, aimed at the wrong statistic | stopped at epoch 6 |
| 007 | Supervise the address directly | Collapse broken, identity still not learned | 0.2292 |
| 008 | Does the model represent fact identity anywhere? | Yes — but four layers above where we looked | 0.8958 at L22 |
What Phase I established.
Writing knowledge into a frozen model is the easy part, and it works cleanly — the delta attaches, performs, and detaches with the base weights provably untouched. The hard part is letting that knowledge participate in reasoning without perturbing the reasoner. Nine experiments moved that number very little, which is why Phase II stopped trying to make the update reason on its own and started treating it as a memory object selected from outside.
Research Log
Experiment 0 - Feasibility
Current objective:
Can we give a frozen reasoning model writable, detachable parametric memory?
The first step is determining whether our small reasoning model can run and undergo parameter efficient training on the available Jetson hardware.
Results, failures, measurements, changes to the hypothesis, and subsequent experiments will be added here as we go.
The Bigger Question
Today's language models generally follow a basic lifecycle:
Train
Deploy
Infer
We are interested in a different lifecycle:
Learn
Infer
Learn again
Infer again
Every interaction is potentially new evidence.
Every solved problem is potentially new experience.
Every discovery is potentially something worth retaining.
The question behind this project is ultimately very simple:
What happens when a reasoning model is no longer frozen after training?
We are going to find out.
Back to blog index