Sounder: A Model That
Knows When It Is Done

Correction, 1 September. The conclusion this post reached — the mechanism works and the economics do not — was measured on text both models had already read. On text neither has seen, the result reverses: at matched compute the thinking model is ahead by 0.40 nats rather than behind by 0.45. The cause was a data-loading bug that went unnoticed for six days. Both readings, and why neither one settles it, are in the measurement that undid the conclusion. The sections in between are left as they were written.

The previous two entries in this series reported on whether a frozen model can be taught new facts by editing its parameters, and then on what those edits cost. Forty-three pre-registered experiments, 144 predictions, 38 of them falsified. The headline result was negative and useful: a detachable adapter stores facts perfectly, damages its neighbours unpredictably, and — at the corpus sizes tested — loses to simply putting the facts in the prompt.

But the more interesting finding was not in any single experiment. It was in the shape of the machinery those experiments kept needing.

Every scaffold was a prosthetic

To make the adapter usable, the programme built a set of external components. Looking at them together, each one substitutes for a signal the model does not produce about itself:

The architecture, and what is deliberately boring about it

The backbone is conventional on purpose — attention, a residual stream, RMSNorm, rotary embeddings, SwiGLU. None of that is where the interest lies. The novelty is confined to a small number of places so that if something works, it is attributable to something.

1. Recurrent depth with a learned halt

h = embed(x)
for step in range(K_max):
    h = reasoning_block(h)      # shared weights, applied repeatedly
    p = halt_head(h)            # scalar per position
    if halt(p): break
y = unembed(h)

One reasoning block, applied between one and eight times. Depth becomes a decision the model makes rather than a constant of the architecture. Halting follows PonderNet, a geometric prior over steps, trained by expectation instead of a hard threshold, which avoids the non-differentiable break — with a ponder cost regularising against always running to the limit.

The reasoning: one forward pass performs one retrieval. k recurrence steps give k hops inside a single pass. That is the two-call structure that empirically worked, moved inside the model.

2. A calibration channel

A second head emits p(known) alongside the token distribution. The training signal is the entire point: a slice of the corpus is deliberately withheld, and the channel is trained to fire unknown on questions drawn from it.

Ordinary pretraining never punishes fluent fabrication, because a confident wrong answer has the same loss shape as a confident right one. Here it is the objective. Calibration is measured as expected calibration error against held-out facts — not as accuracy.

3. Alternating retrieve / compose, and keyed memory

Each recurrence step is two sub-layers: an associative read, then a composition over what was read. And new knowledge is never written into the weights. Programme II established that any adapter change biases the first token of a response for an entire question shape, that the damage is not local, and that five separate predictors of its magnitude all failed.

Instead: a keyed store, read through a learned query, where the key is the scope. This puts the gate, the one component measured to transfer across models — inside the architecture rather than beside it.

4. Self-recursive learning as verified rounds

Not continuous self-modification. There are 43 experiments of evidence that unconstrained weight modification degrades a model unpredictably. The recursion lives in a loop, where every round is reversible and measured:

propose   generate reasoning traces for problems with known answers
verify    a trained verifier head scores each trace
filter    keep only traces the verifier accepts AND that reach the right answer
retrain   one supervised round on the survivors
check     damage evaluation against a frozen held-out set before accepting

Sized to iterate, not to compete

About 145M parameters: 24.6M of tied embeddings over a 32k vocabulary at d=768, roughly 85M in the shared reasoning block, 28M in non-shared entry and exit layers, and about 6M across the halt, calibration and verifier heads plus the keyed-memory projections.

At 20 tokens per parameter that is around 3B tokens — roughly six days across two 8 GB boards, against twelve days for a 300M model and about 140 for a 1B. The smaller budget buys iteration, which matters more here than capacity. This model is not going to beat a 0.6B Qwen at anything, and is not intended to.

What would falsify this

Stated now, before anything is built, which is the entire discipline the previous programme earned the hard way:

Two things the corpus plan got wrong

Between publishing the design and building anything, the data pipeline turned up two faults. Both were cheap to fix at this stage and would have been very expensive later, and neither would have announced itself.

The fetch discarded the field the whole design rests on

The fetcher normalised every source to a bare text field. Sensible for four of them. But Wikipedia records carry titles, and titles are the entity keys for the withheld slice — they are what the calibration channel trains against and what the verification gate searches the shards for. Fifteen gigabytes of corpus had been assembled that could not support the one claim the architecture exists to test.

Found by dumping an actual record and looking at it, instead of trusting the spec. Wikipedia is the smallest source, so re-fetching cost about twenty minutes.

Removing an article does not remove the entity

The registered plan said: withhold a random 2% of articles, train the calibration head to answer unknown on them. But a withheld subject still appears across the other fourteen gigabytes — in arXiv papers, in web text, in other articles. Training the model to claim ignorance of something it demonstrably learned elsewhere teaches miscalibration rather than measuring it.

So, measure it. Five hundred random titles against sixty thousand documents:

A uniform random slice would therefore have contaminated roughly one withheld entity in seven. The rule became: draw only from titles with zero cross-corpus mentions.

And then the sample turned out to be lying

Running that same count over the full corpus instead of the sample — 2,282,399 documents, seven minutes with an Aho-Corasick automaton — gives a different answer.

The sample was optimistic by a factor of two and a half, and the direction is forced rather than accidental: a title is disqualified by appearing in any document, so scanning more documents can only ever lower the clean fraction. A sample gives an upper bound on cleanliness and never a safe basis for selection.

The first full run failed its own gate — 5,208 clean titles against 5,724 required, and wrote nothing at all instead of shipping a slice that was short or contaminated. That is the entire reason the gate exists. Widening the candidate pool and re-running produced 5,724 withheld titles, every one verified absent from all 2,282,399 documents.

Had the withheld set been drawn from the sample’s 86%, about half of it would have been contaminated, nothing downstream would have complained, and the calibration result, the single number this architecture exists to produce — would have come out wrong at the end of a training run with no way to tell why.

A revision to the argument, from the previous programme

The keyed-memory decision above was justified partly on the grounds that a model cannot learn to say what it does not know. Three experiments run since then show that is false: given a corpus that teaches the distinction properly, a detachable memory answers correctly about subjects it holds and denies subjects it does not, generalising to subjects never seen in training.

But the first attempt at that produced a perfect score while carrying no information at all, a template that denied the model’s own configuration facts and cheerfully claimed Roman emperors. The confounded version and the genuine one are indistinguishable from their outputs. Both score 1.0000. Telling them apart took a hand-built control.

So the argument for keyed memory changes rather than collapses. Not a boundary cannot be learned — it can. Rather: a learned boundary cannot be verified from outside, and a keyed one can. A key either matches or it does not. There is no silent failure mode in between, and the previous programme is forty-six experiments of evidence that silent failure modes are the ones that cost you.

It is training

Both arms started on 26 August. The fixed-depth control runs on one board, the recurrent model on the other, and they began from a byte-identical set of weights: same seed, same architecture, same data order. The only difference between them is whether depth is allowed to vary.

Two numbers in the plan were wrong, and the same thing caused both

The parameter budget called for a twelve-layer shared reasoning block. That is infeasible, and the reason is the halting mechanism itself: PonderNet trains by expectation, which is the whole point, because it avoids a non-differentiable break. So every forward pass computes all the steps, always. Twelve layers at eight steps is ninety-six block applications per pass, roughly a hundred-layer model on 8 GB boards.

Depth inside the recurrence is multiplied by the step count. Width is not. So the capacity went into width instead, and the model came out at 133.5M rather than the 67M the first build produced.

The same property broke the schedule. The recurrent arm costs a fixed 1.76× the control per token — not an average over an expected halting depth, but every pass, deterministically. Measured on the actual hardware with actual corpus tokens: 2,901 tokens per second against 5,114. That is why the run is twelve days rather than six, and why the step count was cut from eight to four.

What the early curve says, which is not much

The control is ahead, and the gap is closing. Three percent of the way in, that is noise with a direction, not a result. What it does establish is the shape of the eventual argument: because the recurrent arm spends 1.76× the compute per step, drawing level is not a tie. Recurrence has to win by enough margin to cover a 76% premium, and right now it is not level.

Mean depth has not collapsed. It sits between 1.5 and 1.8, having dipped to 1.38 early and recovered — and measured on 2,665 articles the model has never seen it reads 1.645, against 1.678 on the training data. Those agreeing matters more than the number itself: it means the depth statistic is not an artefact of memorisation, and the stopping rule is reading something real.

Three bugs found by testing the tests

Before any of this could be trusted, every measurement was run against a deliberately empty model — random weights, nothing learned. The standard being applied: does this report “no signal” when handed no signal?

Three of them failed.

All three now derive their thresholds from permutation nulls computed on the data itself rather than from round numbers chosen in advance. The second is the instructive one: ±0.05 felt like a reasonable definition of “indistinguishable from chance”, and it was wrong by a factor of nearly three, because a fixed band has no idea how many samples produced the number it is judging.

The previous project recorded five scoring artefacts of this kind. This one has now found four in its own instrumentation — three above, plus a scorer that matched integers only and would have marked every answer of one task type wrong regardless of what the model produced. Every one was invisible until the pipeline was run end to end on a model known to contain nothing.

A measurement that cannot report an absence has not been shown to detect a presence.

What a model can tell you about itself, and what it cannot

While this trains, three papers have measured the same thing with the same instrument — a linear probe reading a model’s hidden state, which is exactly what Sounder’s calibration head is. Their numbers are worth putting side by side.

The ordering matters more than either figure. A model can report on the quality of what it is currently doing far better than it can predict whether it is going to finish. One is a property of the state it is already in; the other is a forecast about computation that has not happened yet. The second number carries a confidence interval reaching down to 0.53 and a p-value of 0.063, which does not clear the conventional bar — on a model fifty times larger than this one.

Sounder has two internal heads and they fall on opposite sides of that split. The calibration head asks do I know this?, which is a question about the present state. The halt head asks should I stop?, which is a forecast.

The design named the halt head as the risky component before anything was built. That was the right call for a reason it did not state: it is asking the harder of the two questions, at a fraction of the scale where the easier one is only partly solved. If depth collapses while calibration survives, those are not two unrelated findings — they are this same ordering, reproduced twenty times smaller.

It crossed over

For the first two thousand steps the model that thinks was losing. Measured at matched points, where both have seen identical data from identical starting weights, it was behind on 129 of its first 133 comparisons. I wrote that up as the likely outcome.

Then it turned.

It is now ahead on 100 of the last 100 comparisons. The crossover sits near step 2,700. Thinking longer is buying something real, consistently, and the earlier conclusion that depth was neither helping nor hurting is simply wrong — it was measured before the turn.

And it still is not worth it

The thinking model costs 1.76× the compute per step. So beating the control per step is not winning; it has to beat it by enough to cover the premium. Give both the same silicon and the picture inverts:

The control simply takes more steps. In the wall-clock it takes the recurrent model to reach step 5,330, the fixed-depth twin reaches 9,290, and spends those extra steps far better than depth spends its overhead.

The mechanism works and the economics do not. That is a cleaner result than either half alone, and it is the one the control was run to produce. Three papers at almost exactly this model size compare looped architectures against other looped architectures. None reports a non-looped baseline at equal compute, which is the comparison that decides whether any of it pays.

Written a few days later: every number in this section is measured on text both models had read many times over. On text neither had seen, the sign flips. See below — this section is left standing because the mistake it contains is the interesting part.

Four measurements, one story

The turn was not one number moving. Four independent measurements, none built to agree with the others, all changed together:

That last row had been falling for 1,500 steps, and I recorded it as the loop using an ever-shrinking share of the space available to it. It has been climbing ever since and is now back where it started. Another reading that was true when written and stopped being true without announcing itself.

What it thinks about

Depth is spent per token, so the question has a literal answer. The tokens that get the most:

And the least:

It thinks hardest at commas and clause openings, and barely at all in the middle of a word. Once you have written Reg, the rest of Regulations is nearly forced. After a comma, what comes next depends on the shape of the sentence.

Which explains a test that had been failing quietly. One registered claim is that depth should track difficulty, and measured against how hard the control model finds each token, it does not — the correlation has sat at zero throughout. But depth is not tracking difficulty. It is tracking whether the next word is decided by the current word or by the sentence. Those are different things and they barely correlate.

The claim stays falsified as written. Redefining a hypothesis after seeing the data is the failure this project keeps cataloguing in itself, and “depth tracks contextual dependence” is a different question that would need registering on its own terms. But the null result now has an explanation rather than a shrug.

Written a few days later: the null result had a better explanation than this one, and it was not about the model. The measurement itself was broken — see the ruler was made of the wrong material, below. This paragraph is left standing because the sequence is the point.

The ruler was made of the wrong material

That last paragraph was written a few days ago, and it was wrong in a way worth walking through, because the error is not in the model.

Four papers went up on arXiv while this was training. One of them — a diagnostic study of learned halting gates — argues that objectives like this one give a single distribution two jobs: it weights how strongly each depth is trained, and it reports how much the model thought. Their finding is that when adaptive depth fails, the fault usually sits in the trajectory those weights produce rather than in the gate reading it.

That is a testable claim about this model, so it was written down as a prediction before anything was run: the sequence of intermediate states should carry a difficulty signal that the gate fails to express. Three quantities, one forward pass — what the gate reports, what a plain confidence threshold would say, and which depth would actually have been best.

At the first two checkpoints the prediction is clean: the trajectory carries roughly twice the signal the gate expresses. By the third it has inverted. The registered way for this to fail was for the trajectory to carry nothing; instead the two lines crossed. A prediction that is true at two checkpoints and false at a third was never a prediction about the model. It was a prediction about a checkpoint. That is the third time this project has been caught the same way.

The third quantity died on its own control. Thresholding confidence by depth correlated with difficulty at 0.40, which looked like a result until raw confidence on its own came in at 0.44. Adding depth to it made it worse. That control was written after a smoke test rather than before, and it is the only reason the number is not in this post as a finding.

Difficulty, according to a model that cannot read

Then the same measurement was run three times against the same recurrent model on the same text, changing only which snapshot of the control model was used to say how hard each word is.

The sign flips. Same model, same words, same holdout — only the ruler changed.

The third column says why. A barely-trained model has not learned context yet, so the only thing it knows about a word is how often it appears. Its idea of “hard” is 92% just “rare”. A trained one has half escaped that. The two disagree with each other about which words are hard as often as they agree.

And this model spends depth on common words — commas, as, in — which is exactly what the previous section showed by category and which measures directly at −0.130 against word rarity. So against a frequency table it looks like depth avoids hard words. Against a real language model it looks like depth seeks them. Both readings come from one fact.

The registered test compares the two models at matched training steps. That rule is correct for comparing their losses, which is what it was written for. Here it guarantees the ruler is as untrained as the thing being measured. For the first thousand steps, the question being asked was not “does depth go where the text is hard” but “does depth go where the words are rare”, and the answer to that is no, and it was never the question.

So the earlier readings are withdrawn. Not reinterpreted as favourable — withdrawn. The honest position is that the claim has not yet been measured. Its threshold is untouched, and it will be read when both models are finished and comparable, which is the only point at which the ruler is a competent reader of English.

There is a bill attached. The matched snapshot that would have allowed an earlier reading does not exist, because the archiver described in the next section died before it could save it. That is what the lost checkpoints actually cost: not a gap in a chart, but a question that cannot be answered for another four days.

A follow-up test was registered before running — the same correlation with word frequency held fixed — on the theory that frequency was the whole contamination. It is not. Cleaned of frequency, the three rulers give −0.125, +0.033 and +0.208. Still no agreement, and the best-trained one now sits above the threshold the claim needs to clear. A number that depends on which ruler you pick is not a result, and picking the one that clears the bar is the specific move every rule in this project exists to prevent.

Thinking more, and caring less where

The word-boundary result above was measured once. Having just watched a prediction hold at two checkpoints and reverse at a third, it seemed worth reading again at three.

The claim holds. Word starts and punctuation sit above the average at every checkpoint, middles of words and capitalised fragments below, all of them clear of their shuffled baselines. But whitespace travelled from the bottom of the table to the top, and digits stopped being distinguishable from chance. Neither is in the published sentence. Both would have been, had they been measured first.

The more interesting thing is the column shape. As the model trains it thinks more — average depth rises from 1.63 to 2.07 — while the gap between the categories it thinks hardest about and the ones it skips nearly halves. Individual words still get very different amounts of attention. What is fading is the rule connecting the amount to the kind of word.

That is now the fourth measurement pointing the same way, and three of the four were taken for unrelated reasons:

Depth use goes up; depth selectivity goes down. The four depths produce increasingly similar answers, the model becomes less able to say which one it needed, and it runs deeper anyway. That is the opposite of what an adaptive mechanism is supposed to do as it learns, and it is the most concrete thing yet said about why the compute premium is not being repaid.

That second row nearly went out wrong. Reading the shallowest depth against the deepest, the penalty starts at 0.46 nats and by half-training the deeper reading has drawn level and nominally gone ahead — which would be a striking sentence, and is worth 1.1 standard errors. Nothing. The honest version is better anyway: early on, the first and last passes through the recurrence produce measurably different answers, at six standard errors. By halfway they produce the same answer. Not reversed — gone. The error bar shrinks fourfold alongside the gap, which is what two things converging looks like and is not what noise does.

The middle of the curve never moved. Depth 2 is the best readout at every checkpoint, by an almost constant margin. So the depths have not flattened; the two ends have met while the middle stayed better than both. The published measure for this compares only the endpoints, which means it reports “flat” for a shape that is still clearly bent.

Nothing has been registered off it. It describes a trend across three checkpoints in the first half of a run, and the lesson of the previous two sections is exactly that such a description is not yet a claim about the model.

A fifth bug, in the thing built to prevent it

The two models keep only their most recent checkpoint, overwriting the last one each time. Because they run at different speeds, by the time the slow model reaches a given point, the fast one’s matching snapshot is hours gone. So a background job was set up to archive both as they trained.

It ran as a child of the operator session. The session ended. It died. Fifteen checkpoints were lost between step 1,500 and step 5,250 — the exact window in which everything interesting happened.

A tool written to stop evidence disappearing disappeared itself, silently, without failing anything. The replacement runs on the training machines under their own process groups, so it survives whatever happens at the other end of the connection.

A fourth bug of the same kind

The gate that decides whether a task comparison is meaningful at all requires the fixed-depth control to actually be able to do the task. It was set at 10% accuracy. At six percent of training the control scored two correct out of sixteen, which is 12.5%, and would have cleared it — flipping the comparison from “unmeasurable” to “measurable” on the strength of two lucky answers.

So: measure what luck produces. Score the same sixteen generations against shuffled targets, two hundred times. The 95th percentile comes back at 0.1875. The gate had been set below the noise floor of the thing it was gating.

That is the fourth threshold in this project found sitting at or under the noise of its own statistic, after a correlation test that accepted the wrong direction, a chance band off by a factor of three, and a reporting layer that discarded an “unmeasurable” verdict at the final step. Every one was chosen because it looked reasonable. None of them survived being measured.

Any constant in a test is a hypothesis about the noise floor. It should be measured rather than assumed, and the only reliable way to find out is to run the whole thing on data known to contain nothing.

The measurement that undid the conclusion

While checking whether an overheating board could be safely resumed, I read the data loader properly for the first time. It has a method for moving to the next chunk of the corpus. Nothing ever calls it.

So both models opened the first chunk — 112 million tokens, 3.8% of the corpus — and read it, and read it again. The run will hit its planned budget of 2.93 billion tokens exactly, as twenty-six passes over one twenty-ninth of the data. Startup printed 29 shards every time; that line counts chunks found, not chunks used, and it had been reassuring me for six days.

The comparison between the two models survives this untouched. They share the loader, the seed and the fault, so they read identical text in identical order, and every number comparing them is still a fair comparison. What does not survive is what those numbers were about.

Because the loss curve both models are judged by is measured on the chunk they were reading. It is a record of how fast each one is memorising it.

The bug left behind exactly the control needed to prove that: twenty-eight chunks that were never opened. Unseen text, from precisely the training distribution, sitting on the machines the whole time. Same measurement, run there instead:

The sign flips and the size barely moves. The plain model leads by four tenths of a nat on the text both had memorised; the thinking model leads by four tenths of a nat on text neither had seen. Same checkpoints, same silicon, same measurement — a different chunk of the same corpus. Both late results clear two and a half standard errors.

The reason is visible once you look for it. The plain model's loss on unseen text goes up — from 3.348 to 3.611 — while its loss on the text it has read falls from 2.700 to 1.630. By its twenty-third pass it is nearly three times better on remembered text than on statistically identical text it has not seen. It stopped learning and started reciting, and the log it is judged by cannot show that, because the log is measured on the thing being recited.

Why this still is not a win for thinking

The obvious reading — recurrence generalises better — is the one to resist, and the reason is arithmetic.

Equal compute means the plain model takes 1.76× as many steps. Over a fixed 112-million-token chunk, more steps is not more reading; it is rereading. At the point where their silicon costs match, the plain model is on its twenty-third pass and the thinking model on its thirteenth. It is deeper into memorising for a reason that has nothing whatsoever to do with depth.

So what this measures is: at equal compute over a corpus small enough to memorise, the model that reads it fewer times generalises better. That is a real result and it is not the result it looks like. Separating the two requires a run where extra steps buy fresh text, which is exactly what the fixed loader does and this run structurally cannot.

Which leaves the same bug confounding both of this week's findings, in opposite directions. The collapse in selectivity is equally well explained by “a model deep into memorising has less to be selective about.” The reversal above is equally well explained by “fewer passes generalise better.” One fault, two ruined interpretations, both of them mine.

What survives without qualification is narrower and still worth having. The plain model overfits badly, and the training curve it is judged by cannot see it. Its loss on unseen text has been rising for eighteen thousand steps while every number on the dashboard improved. If this project had shipped on the dashboard — and it was one commit from doing so — it would have shipped a model getting worse, described as a model getting better, with a falsification table in perfect order at the top.

Where it stands

The corpus is finished and verified: 2.933B tokens in 29 shards, a 32k tokeniser trained from scratch, and 5,723 withheld Wikipedia articles confirmed absent from every shard — of which, as the section above records, the training run read one shard. It took three runs of the verification gates to get there. Two of those runs failed and stopped the pipeline — once because token budgets can only be set in characters before a tokeniser exists, and once because two article titles differing only in capitalisation collided in the selection index, letting a withheld entity stay in training while its twin was removed.

Everything downstream is built and tested: the comparison, the calibration channel and its evaluation, the verified self-training rounds, and the report that fills in the table above. All of it exists before the results do, which is the point.

The fixed-depth control is 88% through its run and will finish first. The recurrent arm is at 51%, roughly six days behind it, paying for every one of those days in the compute premium. Of the five claims in the table at the top, one holds, one cannot be measured because the model is not yet good enough at the task to make the comparison meaningful, one has lost its instrument and is back to unmeasured, and two are downstream of pretraining and have not been attempted. Two further claims were registered this week: one falsified in a single afternoon, one upheld and reported above with the reason it settles less than it appears to.

Nothing in that table has been altered since it was published. The thresholds were fixed before the first weights existed, and the one interesting result so far — that recurrence wins per step and loses per unit of compute — has no row of its own, because I did not think to ask the question that way until the machines answered it.

A 260 GB collection of scanned books sitting on the same array was considered as training data and declined on copyright grounds. That decision is recorded in the project log instead of quietly omitted, because a corpus is a claim about provenance as much as about volume.

The table at the top has not been altered since it was published, and the point of publishing it early was to make that constraint binding. What was not obvious at the time is how much of the work would turn out to be about the instruments rather than the claims: seven thresholds and one whole yardstick found faulty so far, every one of them written by the same person who then had to find it.

And the largest error was not caught by any of that machinery. Permutation nulls, dry runs on random weights, thresholds tested against their own noise floor — all of it worked, and all of it was irrelevant to a run that was training correctly on the wrong 3.8% of the data. Every one of those tests was passed, honestly, by a model that had read a twenty-ninth of what its log said it had. A falsification table protects you from moving the goalposts. Nothing in it protects you from measuring the wrong thing, and that has been by far the more expensive failure.

Field note, 3 September

The fixed-depth control has finished. Its final loss on text it trained on is a perplexity of 5.1; on text it never saw, about 34. Those two numbers are this whole post in miniature — the dashboard reads five, the model is a thirty-four.

With the control’s weights final, the first downstream channel ran: the know / don’t-know head, trained on the frozen backbone against a held-out probe set. It is the control half of a comparison whose other half waits on the recurrent arm, so it settles nothing on its own — but the shape is worth recording. The head carries real signal: it separates answered-right from answered-wrong at an AUROC of 0.655, above its own permutation floor (0.518), above the raw-loss baseline (0.570), and above what a linear probe reads straight off the last hidden state. It does not clear the threshold registered for a usable detector (0.70), and it is worse calibrated than the baseline it out-ranks — an expected-calibration error of 0.19 against the plain loss’s 0.007. So the backbone knows something about what it knows, the head recovers a real slice of it, and what it recovers is a ranking, not a probability. Registered as negative; interesting anyway.

The more useful result came from a channel that runs on its own, checkpoint by checkpoint, the length of the run. It measures the gap between the model’s loss on text it has seen and on unseen text from the same distribution — pure memorisation, with the distribution held fixed — and it has climbed monotonically since early training: +0.65 nats, then +1.2, now past +2.0. Over the same span the distance to genuinely out-of-distribution text stays flat, near 1.0 nat, exactly as a fixed property of two corpora should. Almost the entire late “improvement” on the dashboard is the model fitting the same slice of the corpus harder, not learning more language.

The recurrent arm is still training. When it finishes, its half of the calibration comparison, the matched-step depth readings, and the peak-location question the earlier data could not answer all come due at once. Until then the table at the top stands as written.

Field note, 11 September

The recurrent arm finished on 6 September, at step 11,187 — six days behind the control, as the compute premium said it would be. Both arms are now trained to the same step on the same data order, for the first time in this project. Every comparison that was waiting on that ran.

Adaptive depth survived. Mean ponder settles at 2.11 of a possible four, spread across depths rather than collapsing to one: roughly 6% of mass at step 1, 80% at step 2, 14% at step 3, negligible at the ceiling. The risky component named at the top of this post did not fail the way it was expected to fail.

Depth tracking difficulty did not survive. Token-level ponder correlates with the control’s own per-token difficulty at r = 0.109 — real, it clears its permutation null of 0.010 — but under the 0.15 registered to call it a finding. This is also the first honest reading of that claim: the difficulty reference is finally a trained model’s judgment rather than a frequency table, its correlation with raw word-rarity having dropped from 0.92 to 0.48 now that both arms are trained. Falsified, cleanly, on a fair instrument.

Whether recurrence buys multi-hop reasoning remains where it has been since the tasks were designed: unmeasurable. Both arms score 0.052 on the graded chain and lookup tasks, under the gate that requires beating chance before a comparison means anything. 133M parameters and 2.9B tokens is not enough model to ask the question with these tasks.

And the comparison this whole post has been waiting on: the recurrent arm’s know / don’t-know head scores an AUROC of 0.649, against the control’s 0.655 recorded above on 3 September — a gap smaller than the measurement noise at this sample size. The two arms are indistinguishable on the one channel recurrence might plausibly have improved, and the recurrent arm paid 1.74× the compute to get there. Same verdict as the control half, now a comparison instead of a single data point.

The table at the top, updated the only way this project updates it, by row: adaptive depth holds; depth tracks difficulty is falsified; multi-hop is unmeasurable; calibration is falsified, on both arms, indistinguishably from each other. Self-recursive rounds have not been attempted.

The generalisation reading referenced above landed since: at the matched step, on text neither arm's loader ever opened, the recurrent arm's loss is 38.2 perplexity against the control's 40.2; on the deliberately harder held-out slice, 111.4 against 118.2. Ahead in both places, by less than one standard error in both places, agreeing in size and direction with the language-modelling number two sections up, measured on an independent window sample. This was registered to predict a failure, and on a strict reading it is one: two independent measurements now agree there is a small, persistent, statistically indistinguishable edge, at 1.74× the compute, and nothing measured tonight moves that number past noise.