Looped Transformers have a depth-control problem

TLDR

43% less time, with lower science accuracy.

We trained Ouro to answer after fewer repeats through its layers. Mean answer time decreased from 146 ms to 83 ms. Overall accuracy increased from 77.4% to 85.2%. These are short multiple-choice answers, generated one request at a time.

Science accuracy decreased by 1.30 percentage points. Our limit was one point in every task family, so the result did not meet our accuracy requirement.

See the results and compare training methods.

Repeat

What a repeat means

A standard Transformer applies each layer once. Ouro applies one 24-layer block several times to every generated token.

We call one application of this block a repeat. Four repeats equal 96 layer applications per generated token.

A stopping rule selects how many repeats to execute. We test whether earlier answers are accurate enough to stop sooner.

Model, controller, and probes

The same 24 layers run at each repeat, with the repeat number and repeat limit as extra inputs.

Architecture for the controller experiments. The model inputs include trained embeddings for the repeat number and repeat limit. The selected repeat count applies to every answer token. The confidence-stopping test below uses no learned controller.

Training

Training for earlier answers

Our earlier tests did not show a clear benefit from a learned stopping network. We then changed the model's answer training.

We compared four versions of the same starting model, Ouro-1.4B. It had already been fine-tuned before this experiment.

Unchanged: No extra training.
Final-only training: Train the answer after repeat four only.
Uniform training: Train answers after all four repeats, with equal weight.
Early-weighted training: Train answers after all four repeats, with more weight on earlier answers.

We selected early-weighted training as the main test before seeing test results. The other versions test the effect of extra weight on earlier answers.

Stopping

Confidence-based stopping

Each question offers answer choices. After each repeat, we convert their model scores into probabilities. The highest probability is choice confidence.

Generation uses the first repeat where choice confidence equals or exceeds the threshold, or repeat four if none does.

We set one threshold per model using separate validation questions. Test answers do not affect the threshold. High confidence does not guarantee a correct answer.

The selected repeat count applies to every answer token. There is no learned stopping network in this test.

Result

146 ms to 83 ms per answer.

Early-weighted training reduces measured answer time by 43%. Overall accuracy increases from 77.4% to 85.2%.

These are averages across three training runs on the same 2,048 test questions. We generated short multiple-choice answers, one request at a time.

Science accuracy decreases from 83.5% to 82.2%, a loss of 1.30 percentage points.

Our limit was one point in every task family. The result therefore does not meet our accuracy requirement.

Accuracy and answer time across all tasks

All four versions use confidence stopping. The task results below show where accuracy decreases.

All versions use confidence stopping, and timings include answer generation. Timing details and uncertainty.

Comparison

Comparison of training methods

Uniform training gives 85.0% accuracy at 80 ms per answer. It takes slightly less time than early-weighted training.

We did not establish an accuracy advantage for early-weighted training over uniform training.

Final-only training also increases overall accuracy and reduces answer time. All three training methods exceed the one-point loss limit on science questions.

Tasks

Accuracy and answer time by task

Each family has 512 test questions. This table compares the unchanged model with early-weighted training.

Accuracy counts correct answer letters without extra text. A final period is allowed.

Task	Unchanged accuracy	Early-weighted accuracy	Time change
Reading comprehension	79.6%	84.2%	-33.9%
Physical commonsense	71.1%	81.6%	-44.7%
Science questions	83.5%	82.2%	-38.7%
Object tracking	75.5%	92.6%	-56.9%

Meaning

Answer time and science accuracy

If every task must stay within a one-point accuracy loss, this result does not justify replacing the unchanged model.

If the science accuracy decrease is acceptable, uniform training is a simpler option. It gives similar overall accuracy in slightly less time.

These measurements cover short answers, one request at a time. They do not establish savings for long answers or requests processed together.

We did not time every fixed repeat count for the newly trained models. Confidence stopping is not established as the best stopping rule.

How the model was trained

Each of three training runs started from its saved, fine-tuned Ouro-1.4B model with ordinary connections.

Within each run, the trained variants used identical starting weights, questions, minibatch order, and optimizer settings.

We trained rank-eight LoRA adapters and repeat embeddings. Other model weights stayed fixed.

Training and data

The four task families are BoolQ, PIQA, filtered ARC-Challenge, and generated object tracking.

For each task family, we used 512 training, 128 validation, and 512 test questions. Previously tested questions and source identities are excluded.

BoolQ passages are also separated across splits. Answer positions are balanced. Prompts contain at most 512 tokens without truncation.

Each trained variant uses two epochs, microbatches of four, and four accumulated microbatches per update. There are 256 updates per variant.

AdamW uses a constant learning rate of 0.00005, weight decay 0.01, and gradient clipping at one. We keep the final checkpoint.

The repeat limit and repeat-limit embedding remain four. Training changes LoRA adapters, repeat-index embeddings, and repeat-limit embeddings.

Let $L_t$ be full-vocabulary cross-entropy, averaged over the space-prefixed answer-letter token and the end token at repeat $t$ .

\mathcal{L}=\sum_{t=1}^{4}w_tL_t,\qquad w_t^{\mathrm{early}}=\frac{0.7^{t-1}}{\sum_{j=1}^{4}0.7^{j-1}}

Uniform weights are one quarter each. Final-only weights are zero, zero, zero, and one.

Training and stopping-threshold selection

Training uses all four repeats. After training, a confidence threshold determines whether generation uses fewer repeats.

Loss weights do not depend on the stopping rule. The threshold is selected after model training, using separate validation questions.

How we chose when to stop

Choice confidence is normalized across the offered answer-letter tokens. It is not a calibrated probability that the answer is correct.

The model generates over its full vocabulary. Generation is not restricted to the offered answer letters.

Confidence and threshold selection

Let $C$ contain the space-prefixed tokens for the offered answer letters. Let their logits after repeat $t$ be $z_{t,c}$ .

s_t=\max_{c\in C}\frac{\exp(z_{t,c})}{\sum_{d\in C}\exp(z_{t,d})},\qquad T=\min\left(\{t\in\{1,2,3\}:s_t\geq\tau\}\cup\{4\}\right)

For each model, validation evaluates thresholds from zero to one in steps of 0.005, plus a forced-four endpoint.

We compared validation accuracy with the unchanged model at four repeats. A threshold was eligible if overall accuracy decreased by at most 0.5 percentage points.

Accuracy could decrease by at most one percentage point in each task family. We selected the eligible threshold with the fewest mean repeats.

Ties use higher accuracy, then a higher threshold. We did not use test labels to select thresholds, objectives, or checkpoints.

Answer correctness at early stops and repeat four

These validation counts compare an early stop with forced repeat four in the same model. They are diagnostics, not held-out error estimates.

Model training	Early stops	Early wrong, fourth correct	Early correct, fourth wrong
Unchanged	762	5	0
Final-only training	1310	69	12
Uniform training	1459	85	15
Early-weighted training	1390	66	10

Counts sum three seeds on the same 512 validation questions. They are 1,536 model–question evaluations, not 1,536 distinct questions.

Timing, uncertainty, and checks

Measured differences and uncertainty

Early-weighted minus unchanged accuracy is +7.76 percentage points. The paired 95% interval is [6.09, 9.36].

The early-weighted-to-unchanged time ratio is 0.567. Its paired 95% interval is [0.529, 0.605].

A time ratio of 0.80 means 20% less time. The timing requirement was at least a 20% reduction.

Early-weighted minus uniform accuracy is +0.16 percentage points. The paired 95% interval is [-0.34, 0.67].

This interval includes zero. The test does not establish an accuracy advantage for either training method.

Per-seed primary comparison

Seed	Unchanged accuracy	Early-weighted accuracy	Unchanged time	Early-weighted time	Point criteria
20260904	77.1%	84.5%	150 ms	78 ms	Not met
20260905	77.1%	85.7%	147 ms	89 ms	Met
20260906	77.9%	85.3%	142 ms	81 ms	Met

Timing, uncertainty, and verification

Generation is greedy, batch one, with no KV cache and a six-token limit. Timing includes tokenization, transfer, confidence calculation, generation, and decoding.

Each policy uses the same GPU within a seed. Checkpoint loading is outside the timer. Five rounds interleave policies on 128 preselected questions.

The unchanged model at four fixed repeats gives 78.0% accuracy at 195 ms.

Full-test single-execution timings give an early-weighted-to-unchanged time ratio of 0.583. This is a secondary timing check.

We average timing rounds per question, then average seeds. The paired bootstrap resamples questions within each family, with 4,000 draws.

Intervals condition on these trained models and this workload. They do not measure uncertainty across all possible training seeds.

The primary test also requires an accuracy-interval lower bound above minus one percentage point and a time-ratio upper bound at most 0.80.

Two seeds meet the point criteria, and all three calibrations are feasible. However, the three-seed science accuracy decrease exceeds the one-point limit.

The result does not meet the preset requirement, which uses task-level point estimates without requiring a statistically significant decrease.

Independent checks verify 2,304 optimizer updates, 30,720 test generations, and 9,600 timing replays. Output-token and executed-repeat checks pass.

Download the aggregate measurements, calibration choices, and per-seed results.

Accuracy after one, two, three, or four repeats

Validation accuracy by repeat count

Each point shows validation accuracy at a fixed repeat count, averaged across three training runs.

UnchangedFinal-only trainingUniform trainingEarly-weighted training

The stopping threshold also uses these validation answers. Independent test results appear in the main comparison.

Fixed-repeat accuracy on the separate validation set. These curves describe earlier answers but do not replace the independent test and timing comparison.

Mean repeat count by task

These are test-set means across three training runs. Each repeat runs the same 24 layers.

Task	Unchanged repeats	Early-weighted repeats
Reading comprehension	3.22	1.88
Physical commonsense	3.56	2.23
Science questions	2.74	1.74
Object tracking	2.87	1.30

Across all test questions, mean repeats decrease from 3.10 to 1.79.

Earlier tests: arithmetic, learned controllers, and interpretation probes

Model structure across repeats

Ouro-1.4B was trained with four repeats. Our tests use repeat counts one through eight.

t\leq4

training range

t>4

outside training range

input $x_{1:n}$

repeat 1

same 24 layers $F_\theta$

h_{1}

repeat 2

same 24 layers $F_\theta$

h_{2}

repeat 3

same 24 layers $F_\theta$

h_{3}

repeat 4

same 24 layers $F_\theta$

h_{4}

repeat 5

same 24 layers $F_\theta$

h_{5}

repeat 6

same 24 layers $F_\theta$

h_{6}

repeat 7

same 24 layers $F_\theta$

h_{7}

repeat 8

same 24 layers $F_\theta$

h_{8}

Repeats one through four match training. Repeats five through eight test behavior outside the training setting.

Figure 1. The same 24-layer block is applied eight times. The checkpoint was trained with four repeats. Repeats five through eight are outside the training setting.

Answer test

Accuracy across repeat counts

Single-digit arithmetic: Example: $7-2=5$ .
Multiply-add: Example: $14\times5+5=75$ .
Two-product modulo: Example: $(5\times8+9\times9)\bmod 10=1$ .

For each task, we test 40 prompts at repeat counts one through eight. The model generates up to five tokens.

For single-digit arithmetic, 40 answers are correct at repeat three. For multiply-add, 12 are correct at repeat four and 13 at repeat five.

For two-product modulo, 2 answers are correct at repeats one through four. At later repeat counts, zero or one answer is correct.

Correct answers by repeat count

40 prompts per task, up to five generated tokens

single-digit arithmeticmultiply-addtwo-product modulo

Repeat 4: 40, 12, and 2 correct answers. Repeat 5: 40, 13, and 1.

Figure 2. At repeat four, single-digit arithmetic has 40 of 40 correct answers. Multiply-add has 12. Two-product modulo has 2.

54 / 120

Repeat four and repeat five both have 54 of 120 correct answers.

The set of prompts with correct generated answers differs across repeat counts. It does not produce a clear improvement after repeat four.

State test

State change and answer accuracy

The state is the model vector after each repeat. Relative state change compares the new state with the previous state.

r_t=\frac{\lVert h_t-h_{t-1}\rVert_2}{\lVert h_{t-1}\rVert_2}

A value near zero means that the state changed little.

Digit-choice accuracy is the percentage of prompts where the highest-logit digit is correct. The state test uses 180 prompts for each task.

At repeat seven, relative state change is 0.139 for single-digit arithmetic and 0.138 for two-product modulo.

Relative state change and digit-choice accuracy

Single-digit arithmetic: 180/180. Two-product modulo: 21/180.

single-digit arithmetictwo-product modulo

At repeat 7, relative state change is 0.139 and 0.138. Digit-choice accuracy is 100% and 11.7%.

Figure 3. At repeat seven, mean relative state change is 0.139 and 0.138. Digit-choice accuracy is 100% and 11.7%.

State measurements

The tasks differ by 0.001 in relative state change and by 159 correct digit choices out of 180.

Single-digit arithmetic has 180 of 180 correct choices. Two-product modulo has 21 of 180.

The tested state-change thresholds would stop both tasks despite their different accuracies. This comparison does not evaluate other stopping rules.

Controller test

Controller results before fine-tuning

The Ouro controller uses a score learned during model training. It selects four repeats for every test prompt.

The threshold controller stops after small state and output changes. We select its thresholds on 60 separate prompts.

We then test both controllers on 60 new prompts.

Method	Correct answers	Mean repeat count
Fixed four repeats	29 / 60	4.00
Ouro controller	29 / 60	4.00
Threshold controller	27 / 60	2.60

35% fewer

Compared with four fixed repeats, the threshold controller uses 35% fewer repeats and produces two fewer correct answers.

The Ouro controller matches fixed four repeats because it selects four repeats for every prompt.

Model training

Training with a repeat limit

We compared the trained controller with fixed repeat counts, including counts below four.

In our first three-seed fine-tuning study, the controller and fixed two repeats both averaged 65.2% accuracy.

The controller averaged 1.996 repeats. Training the model before joint updates did not establish a reliable improvement in a separate study with model-only training before joint updates.

We then trained models with an explicit repeat index and repeat limit. The controller selected depth during generation.

We trained each model on 480 problems. In the first epoch, we used addition, then multiply-add, then two-product modulo.

Addition now uses two to six numbers from zero through 19. For example, $12+7+18=37$ . It differs from the initial single-digit test.

Later epochs mixed the tasks. All models trained with repeat limits from one through four.

Repeat limit: The maximum number of repeats allowed for an answer.
Controller: A small neural network that uses the last prompt token's state after a repeat to calculate a stop probability.
Interpretation probe: A separate predictor trained after model training. Its output does not change depth or generated answers.

Connections

Connection designs

We compared three connection designs inside Ouro-1.4B. The controller architecture was identical in all three.

Ordinary connections: Add each attention or MLP output to the current state.
mHC adaptation: Maintain four parallel state streams, each with one vector per token. Learned maps combine them around each attention and MLP block.
Attention Residuals adaptation: Use learned weights to combine the repeat input and earlier attention and MLP outputs.

The mHC streams and Attention Residuals history restart at each repeat. Both retain Ouro's original block normalization.

These are fine-tuned Ouro adaptations, not reproductions of the papers' pretraining experiments.

Each design used two training objectives and three seeds, for 18 trained conditions.

Separate objectives: Within each batch, give equal answer-loss weight to every allowed repeat. These weights do not depend on controller decisions.
Shared objective: Weight the model's answer loss by the controller's stop probabilities. Update both with that loss and a repeat penalty.

We trained the model and controller with each objective. Only the shared objective included gradients through controller stop probabilities into model parameters.

We selected stopping thresholds on 180 calibration problems, then tested on 360 different problems. All conditions used the same test problems.

Generated answers

Controller compared with two fixed repeats

With ordinary connections and the shared objective, the controller averages 61.8% accuracy at 2.35 repeats. Two fixed repeats give 62.7%. Both stopping rules use the same trained models.

Accuracy is the percentage of complete answers that contain the expected integer without extra text, apart from an optional final period.

Generation allows up to 16 tokens. Every stopping rule uses a repeat limit of four, including fixed-count comparisons.

For each test problem, we generated answers at all four repeat counts. We scored the answer at the count selected by the controller.

That keeps the repeat-limit embedding unchanged. The controller changes only when execution stops.

Calibration also selects a single fixed repeat count. This tests whether a learned controller improves on a simple rule selected without test answers.

Accuracy and repeat counts by stopping rule

All panels show accuracy on the vertical axis and mean repeat count on the horizontal axis, using the same scales.

Separate objectivesShared objective○ fixed repeat count◆ controller

Repeats do not cost the same amount across connection designs. Timing and per-seed comparisons appear below.

Means across three training seeds, evaluated on the same 360 problems. Each controller threshold was selected before test evaluation.

Connections	Training	Controller accuracy	Average repeats	Selected fixed-count accuracy	Average repeats
Ordinary	Separate objectives	62.1%	2.273	63.1%	2.000
Ordinary	Shared objective	61.8%	2.347	62.7%	2.000
mHC adaptation	Separate objectives	58.8%	2.159	60.9%	2.333
mHC adaptation	Shared objective	59.3%	2.168	61.6%	2.333
Attention Residuals adaptation	Separate objectives	28.5%	1.000	28.5%	1.000
Attention Residuals adaptation	Shared objective	28.9%	1.000	28.9%	1.000

Fixed-count comparison

Two fixed repeats used fewer repeats than the controller, with a slightly higher measured accuracy.

The ordinary model's accuracy difference is small. The 95% interval includes zero in each seed, so this test does not establish a reliable accuracy gain for either stopping rule.

The mHC adaptation gives 59.3% accuracy at 2.17 repeats with the shared objective. Two fixed repeats give 60.8%.

The Attention Residuals adaptation always selects one repeat. It gives 28.9% accuracy with the shared objective, exactly the same as one fixed repeat.

These are results from the Ouro adaptations. They do not compare the papers' pretrained models.

The intervals below compare the same test problems within each training seed. They do not measure uncertainty over every possible training run.

Per-seed differences, timing, and training details

Controller repeat counts by task

These are the shared-objective models. Each task has 120 test problems. Values are means across three training seeds.

Connections	Task	Controller accuracy	Average repeats	Selected fixed-count accuracy
Ordinary	Addition	85.6%	1.825	86.7%
Ordinary	Multiply-add	53.1%	2.239	54.4%
Ordinary	Two-product modulo	46.7%	2.978	46.9%
mHC adaptation	Addition	85.0%	1.636	87.8%
mHC adaptation	Multiply-add	49.2%	2.150	53.1%
mHC adaptation	Two-product modulo	43.6%	2.717	43.9%
Attention Residuals adaptation	Addition	36.7%	1.000	36.7%
Attention Residuals adaptation	Multiply-add	25.0%	1.000	25.0%
Attention Residuals adaptation	Two-product modulo	25.0%	1.000	25.0%

Shared and separate training objectives

Differences are shared objective minus separate objectives. Accuracy differences use percentage points.

Connections	Seed	Accuracy difference	95% interval	Repeat difference
Ordinary	20260904	+4.44	[0.28, 8.61]	-0.417
Ordinary	20260905	-2.50	[-6.67, 1.67]	+0.214
Ordinary	20260906	-3.06	[-7.50, 1.39]	+0.425
mHC adaptation	20260904	-2.78	[-7.50, 1.39]	-0.756
mHC adaptation	20260905	+3.06	[-1.94, 8.33]	+0.397
mHC adaptation	20260906	+1.11	[-3.06, 5.83]	+0.383
Attention Residuals adaptation	20260904	+3.06	[-1.94, 8.06]	0.000
Attention Residuals adaptation	20260905	-1.67	[-6.39, 3.06]	0.000
Attention Residuals adaptation	20260906	-0.28	[-5.56, 5.28]	0.000

Controller compared with two fixed repeats

This comparison holds model weights fixed. It changes only the stopping rule.

Connections	Training	Seed	Accuracy difference	95% interval	Repeat difference
Ordinary	Separate objectives	20260904	-0.83	[-3.33, 1.39]	+0.450
Ordinary	Separate objectives	20260905	-1.94	[-3.89, -0.56]	-0.028
Ordinary	Separate objectives	20260906	0.00	[-1.94, 1.94]	+0.397
Ordinary	Shared objective	20260904	-1.67	[-3.33, 0.00]	+0.033
Ordinary	Shared objective	20260905	-0.83	[-3.06, 1.39]	+0.186
Ordinary	Shared objective	20260906	-0.28	[-2.50, 1.94]	+0.822
mHC adaptation	Separate objectives	20260904	-0.28	[-2.50, 1.94]	+0.506
mHC adaptation	Separate objectives	20260905	-6.39	[-9.44, -3.33]	-0.494
mHC adaptation	Separate objectives	20260906	+0.28	[-1.39, 1.94]	+0.467
mHC adaptation	Shared objective	20260904	-3.61	[-5.83, -1.67]	-0.250
mHC adaptation	Shared objective	20260905	+0.28	[-1.67, 2.22]	-0.097
mHC adaptation	Shared objective	20260906	-1.39	[-4.17, 1.39]	+0.850
Attention Residuals adaptation	Separate objectives	20260904	0.00	[-2.22, 2.22]	-1.000
Attention Residuals adaptation	Separate objectives	20260905	+1.39	[-1.39, 3.89]	-1.000
Attention Residuals adaptation	Separate objectives	20260906	+1.39	[-0.83, 3.61]	-1.000
Attention Residuals adaptation	Shared objective	20260904	+0.56	[-1.67, 2.78]	-1.000
Attention Residuals adaptation	Shared objective	20260905	0.00	[-2.78, 2.50]	-1.000
Attention Residuals adaptation	Shared objective	20260906	+0.83	[-1.11, 2.78]	-1.000

Controller compared with the selected fixed repeat count

Both rules were selected on calibration problems. Differences are controller minus selected fixed count.

Connections	Training	Seed	Accuracy difference	95% interval	Repeat difference
Ordinary	Separate objectives	20260904	-0.83	[-3.33, 1.39]	+0.450
Ordinary	Separate objectives	20260905	-1.94	[-3.89, -0.56]	-0.028
Ordinary	Separate objectives	20260906	0.00	[-1.94, 1.94]	+0.397
Ordinary	Shared objective	20260904	-1.67	[-3.33, 0.00]	+0.033
Ordinary	Shared objective	20260905	-0.83	[-3.06, 1.39]	+0.186
Ordinary	Shared objective	20260906	-0.28	[-2.50, 1.94]	+0.822
mHC adaptation	Separate objectives	20260904	-0.28	[-1.94, 1.39]	-0.494
mHC adaptation	Separate objectives	20260905	-6.39	[-9.44, -3.33]	-0.494
mHC adaptation	Separate objectives	20260906	+0.28	[-1.39, 1.94]	+0.467
mHC adaptation	Shared objective	20260904	-3.61	[-5.83, -1.67]	-0.250
mHC adaptation	Shared objective	20260905	+0.28	[-1.67, 2.22]	-0.097
mHC adaptation	Shared objective	20260906	-3.61	[-6.39, -0.83]	-0.150
Attention Residuals adaptation	Separate objectives	20260904	0.00	[0.00, 0.00]	0.000
Attention Residuals adaptation	Separate objectives	20260905	0.00	[0.00, 0.00]	0.000
Attention Residuals adaptation	Separate objectives	20260906	0.00	[0.00, 0.00]	0.000
Attention Residuals adaptation	Shared objective	20260904	0.00	[0.00, 0.00]	0.000
Attention Residuals adaptation	Shared objective	20260905	0.00	[0.00, 0.00]	0.000
Attention Residuals adaptation	Shared objective	20260906	0.00	[0.00, 0.00]	0.000

Actual generation time

Timing uses 12 preselected prompts, batch one, no KV cache, and three interleaved rounds. It includes controller execution and tokenization.

The 12 timing prompts need not have the same repeat distribution as the full test set.

mHC uses exact-checked CUDA graph replay for its mixing maps. Times describe these implementations, not equally optimized kernels.

Connections	Training	Controller, seconds	Fixed two, seconds	Fixed four, seconds
Ordinary	Separate objectives	0.240	0.208	0.403
Ordinary	Shared objective	0.247	0.208	0.402
mHC adaptation	Separate objectives	0.609	0.563	1.100
mHC adaptation	Shared objective	0.625	0.564	1.110
Attention Residuals adaptation	Separate objectives	0.097	0.192	0.381
Attention Residuals adaptation	Shared objective	0.098	0.193	0.384

For every condition, checks on the 12 timing prompts confirmed matching output tokens, expected executed-repeat counts, and compliance with repeat limits.

These checks ran actual early-exit generation. Output tokens matched those in the corresponding fixed-count answers used for full-test scoring.

Training and selection

Each training run started from an adapter saved after the model-only training phase. Paired objectives started from identical parameters within each connection design.

Rank-eight LoRA adapters, connection parameters, repeat embeddings, and repeat-limit embeddings were trainable. The other pretrained weights were fixed.

We used three epochs with batches of eight and 180 model updates. Controllers had 135 updates because repeat limit one requires no controller decision.

AdamW learning rates were 0.00015 for model parameters and 0.0005 for controller parameters. Gradient clipping used a maximum norm of one.

The shared objective reserved 20% of answer loss for uniform training across allowed repeat counts.

Let the repeat limit be $K$ . Let answer-token cross-entropy at repeat $t$ be $L_t$ .

The controller predicts a halt probability $p_t$ . The resulting exit probabilities $q_t$ sum to one.

q_t=p_t\prod_{j<t}(1-p_j)\quad(t<K),\qquad q_K=\prod_{j<K}(1-p_j)

\mathcal{L}_{\mathrm{shared}}=\sum_{t=1}^{K}\left(0.8q_t+\frac{0.2}{K}\right)L_t+0.02\sum_{t=1}^{K}tq_t

The separate model objective is the uniform mean of answer losses. Its controller uses detached states and detached answer losses.

\mathcal{L}_{\mathrm{model}}=\frac{1}{K}\sum_{t=1}^{K}L_t,\qquad\mathcal{L}_{\mathrm{controller}}=0.8\sum_{t=1}^{K}q_t\,\mathrm{stopgrad}(L_t)+0.02\sum_{t=1}^{K}tq_t

Training computes all allowed repeats for these losses. Generation executes discrete stops. Answer loss is not the same measurement as generated-answer accuracy.

Calibration tests 101 halt thresholds. The selection score subtracts two accuracy percentage points for each average repeat.

The same score selects fixed-count and task-based comparisons.

The three architectures have different parameter counts and work per repeat. Matching optimizer updates does not match compute.

Connections	Trainable model parameters	Controller parameters
Ordinary	7,585,792	131,329
mHC adaptation	17,024,128	131,329
Attention Residuals adaptation	7,786,496	131,329

mHC uses four streams and 20 log-space Sinkhorn iterations for its residual mixing matrix.

Attention Residuals uses RMS-normalized keys and a learned query for each attention or MLP input, plus a final weighted combination.

The adaptations slightly change pretrained outputs before fine-tuning. The ordinary implementation matches the original decoder exactly.

Download the aggregate results and per-seed comparisons.

Interpretation

Probes for predicting answer corrections

After model training, we kept its weights fixed and trained two interpretation probes on 720 separate problems.

One probe used a linear map of the current state. The other used an MLP with the current state and state change.

Probe selection used another 180 problems. Neither probe participated in generation or controller calibration.

We measured whether each probe ranked correctable answers above answers that remained incorrect after the next repeat.

Only currently incorrect answers entered this calculation. This prevents a probe from scoring well just by identifying answers that are already correct.

Correction AUC is 0.5 for chance ranking and 1 for perfect ranking. It is not the percentage of correct generated answers.

Probe inputs and number of corrected answers

The linear probe inputs are the layer-normalized current state, its log norm, and the repeat number divided by four.

The MLP inputs also include normalized state change, log-transformed state-change magnitude, state cosine, and log norm ratio.

At repeat one, all state-change inputs are zero. The MLP has 32 hidden units and a SiLU activation.

Both probes predict three outcomes: an incorrect answer becomes correct, a correct answer becomes incorrect, or correctness stays unchanged.

Training covers all six earlier-to-later repeat pairs. The plot shows the three next-repeat pairs.

Each entry below is corrected answers / currently incorrect answers, in training-seed order. These are the shared-objective models shown in the plot.

Connections	Repeat change	20260904	20260905	20260906
Ordinary	1 → 2	64 / 166	76 / 202	48 / 182
Ordinary	2 → 3	4 / 114	14 / 138	6 / 151
Ordinary	3 → 4	10 / 122	9 / 135	4 / 151
mHC adaptation	1 → 2	59 / 169	63 / 196	49 / 185
mHC adaptation	2 → 3	19 / 128	12 / 144	16 / 151
mHC adaptation	3 → 4	5 / 116	6 / 143	7 / 143
Attention Residuals adaptation	1 → 2	8 / 245	12 / 255	5 / 268
Attention Residuals adaptation	2 → 3	13 / 247	6 / 255	6 / 271
Attention Residuals adaptation	3 → 4	11 / 243	13 / 257	10 / 277

AUC can vary substantially when few answers change correctness. The downloadable data also include predictions of newly incorrect answers and within-task correction AUC.

Prediction of answer corrections after another repeat

This comparison uses currently incorrect answers from models trained with the shared objective.

Current state, linear probeCurrent state and state change, MLP

Correction AUC measures how well the probes rank answers by whether the next repeat corrects them. Probe inputs and structure both differ.

Small dots show training seeds. Large dots show means of defined AUC values. Missing values indicate no positive or no negative examples.

Meaning

With ordinary connections and the shared objective, first-to-second-repeat correction AUC averages 0.558 for the MLP and 0.546 for the linear probe. At the second-to-third repeat, the values are 0.503 and 0.488. The MLP does not give a consistent improvement across training seeds.

Correction AUC measures the ranking of answers by predicted correction. We evaluated stopping rules separately through generated-answer accuracy and repeat counts.

The two probes differ in both inputs and structure. Their difference cannot isolate the effect of state change alone.

At repeat one, both probes have the same state information. A higher MLP score there would not establish an effect from state-change inputs.

We also replaced previous states with another same-task problem's states. This measures probe sensitivity, not a causal change inside the model.

What this result covers

The early-answer result uses three Ouro-1.4B training seeds and four multiple-choice task families. It does not show general reliability or long-answer speedups.

The earlier arithmetic tests and connection adaptations remain separate. Neither study tests full-model pretraining or establishes a general ranking of architectures.

Technical appendix: additional measurements and controller training

Examples

Answer changes across repeats

For six of 120 prompts, the generated answer is correct at an earlier repeat count and incorrect at repeat eight.

Generated answers at repeat counts one through eight

First line of each generated answer. Correct answers have an accent outline.

prompt

repeat 1

repeat 2

repeat 3

repeat 4

repeat 5

repeat 6

repeat 7

repeat 8

$0+1$ correct answer:

1

1.correct

2incorrect

1.correct

$14\times5+5$ correct answer:

75

70 + 5incorrect

75correct

75.correct

$(5\times8+9\times9)\bmod 10$ correct answer:

1

40 + 8incorrect

1.correct

1correct

5 * 8 +incorrect

10.incorrect

For six of 120 prompts, the generated answer is correct at an earlier repeat count and incorrect at repeat 8.

Figure 4. For six of 120 prompts, the generated answer is correct at an earlier repeat count and incorrect at repeat eight.

Attention

Component cosine for query, key, and value vectors

Component cosine compares the same attention vector in two consecutive repeats. A higher value means a smaller change in direction.

Query and key component cosine is higher than value component cosine on both tasks.

Component cosine for query, key, and value vectors

Component cosine from repeat 1 to repeat 2, averaged across 24 physical layers

single-digit arithmetictwo-product moduloSMELT range

Two-product modulo, layer 24: $Q=0.964$ , $K=0.967$ , $V=0.795$ .

Figure 5. Component cosine is higher for query and key vectors than for value vectors on both tasks.

State measures

State cosine and effective rank by task

State cosine compares state direction between consecutive repeats. Effective rank estimates state variation across prompts.

These measurements describe the state. They do not measure answer correctness.

State cosine and effective rank

Two measurements of the final prompt-token state after each repeat

single-digit arithmetictwo-product modulo

At repeat 7, state cosine is about 0.989 for both tasks. Effective rank is 18.4 for single-digit arithmetic and 16.2 for two-product modulo.

Figure 6. At repeat seven, state cosine is about 0.989 for both tasks. Effective rank is 18.4 for single-digit arithmetic and 16.2 for two-product modulo.

Physical layers

Correct-digit margin changes at layer 24

The correct-digit margin is the correct-digit logit minus the largest incorrect-digit logit.

The layer-24 changes are 2.53, 2.32, 2.97, and 2.30 logits at repeats one through four.

Correct-digit margin changes by layer

Change in correct-digit margin after every physical layer

lowers marginraises margin

For two-product modulo, the layer-24 margin changes are 2.53, 2.32, 2.97, and 2.30 logits.

Figure 7. Layer-24 correct-digit margin changes exceed 2 logits at repeats one through four.

Digit loss

Digit loss and digit-choice accuracy

For two-product modulo, digit loss changes from 2.504 at repeat four to 2.346 at repeat eight.

Repeat four and repeat eight both have 22 of 180 correct digit choices.

Digit loss and digit-choice accuracy

Two-product modulo across eight repeats

digit losscorrect-digit rankbecame correctbecame incorrect

Digit loss is 2.504 at repeat 4 and 2.346 at repeat 8. Both have 22 of 180 correct digit choices.

Figure 8. Digit loss is lower at repeat eight than at repeat four. Both repeat counts have 22 of 180 correct digit choices.

Known answers

Repeat selection using known answers

The answer-checking method generates answers at all four repeat counts. It selects the first repeat with a correct answer.

The method cannot run when the correct answer is unknown. It provides a reference, not a deployable controller.

It has 31 of 60 correct answers with a mean repeat count of 2.82.

Mean repeat count by task and method

60 held-out prompts, 20 per task, maximum repeat count 4

Ouro controllerthreshold controlleranswer-checking method

Threshold controller means are 2.75 for single-digit arithmetic and 2.30 for multiply-add. Answer-checking method means are 1.20 and 3.60.

Figure 9. The threshold controller averages 2.75 repeats for single-digit arithmetic and 2.30 for multiply-add. The answer-checking method averages 1.20 and 3.60.

Stop and continue labels for controller training

0The current answer is correct, or no later answer is correct.Stop

1The current answer is incorrect, and a later answer is correct.Continue

Controller training

Controller training with fixed model weights

We train seven small neural networks on 102 prompts. We select a threshold on 48 prompts and test on 150 new prompts.

AUC measures how often the controller ranks a continue case above a stop case. Random ranking has an AUC of 0.5. This controller has 0.772.

None of the tested thresholds matched the accuracy of four fixed repeats while using fewer repeats.

Correct answers by task and repeat count

Task-level repeat counts use 150 development prompts. Results use 150 separate test prompts.

Single-digit arithmetic

selected repeat count: 3

Multiply-add

selected repeat count: 4

Two-product modulo

selected repeat count: 1

Fixed four repeats70 correct600 repeat units

Task-level counts 3 / 4 / 168 correct400 repeat units

200 fewer repeat units
2 fewer correct answers

Figure 10. The selected task-level repeat counts are three for single-digit arithmetic, four for multiply-add, and one for two-product modulo.

Method

Training, calibration, and test sets

Initial answer test

Three tasks with 40 prompts per task. Greedy decoding produces up to five tokens.

Initial state test

Two tasks with 180 prompts per task. Measurements use the full 2,048-dimensional state.

Initial controller test

Two controllers use 60 development prompts and 60 new test prompts.

The depth-conditioned study uses 480 model-training, 720 probe-training, 180 calibration, 180 probe-selection, and 360 test problems. No problem identity appears in two splits.

Probe-training generation uses batches of 20. Calibration, probe selection, and test generation use batch one.

Each run uses one B300 GPU. Nomad storage is temporary, and the job specifications contain no disk mounts.

Cite

@misc{loopconvergence2026,
  author = {Korolev, Konstantin},
  title  = {Looped Transformers have a depth-control problem},
  year   = {2026},
  note   = {White Circle},
  url    = {https://advprop.ai/notes/looped-transformers/}
}

This research is sponsored by White Circle.

About

I am Konstantin Korolev, Head of Applied Research at White Circle. I study model internals, representation geometry, and inference-time computation.

We are hiring ML infrastructure, research, and multimodal engineers in Paris, London, San Francisco, and New York City. Contact me at @advprop.

Expert selection at input layersExpert selection across speech, image, and text

43% less time, with lower science accuracy.

What a repeat means

Training for earlier answers

Confidence-based stopping

146 ms to 83 ms per answer.

Comparison of training methods

Accuracy and answer time by task

Answer time and science accuracy

Training and data

Confidence and threshold selection

Answer correctness at early stops and repeat four

Measured differences and uncertainty

Per-seed primary comparison

Timing, uncertainty, and verification

Mean repeat count by task

Accuracy across repeat counts

State change and answer accuracy

Controller results before fine-tuning

Training with a repeat limit

Connection designs

Controller compared with two fixed repeats

Controller repeat counts by task

Shared and separate training objectives

Controller compared with two fixed repeats

Controller compared with the selected fixed repeat count

Actual generation time

Training and selection

Probes for predicting answer corrections

Related research

What this result covers

Answer changes across repeats

Component cosine for query, key, and value vectors

State cosine and effective rank by task

Correct-digit margin changes at layer 24

Digit loss and digit-choice accuracy

Repeat selection using known answers

Stop and continue labels for controller training

Controller training with fixed model weights

Single-digit arithmetic

Multiply-add

Two-product modulo

Training, calibration, and test sets