Prefill a looped modelwith one pass of the loop

Inkling 2 Mini applies 12 of its layers three times. My prefill model computes one pass, and the KV cache of the other two is key and value projections only. Inkling 2 Mini decodes from that cache unchanged. Time to first token is 2.4 times lower. The benchmark score is 9 points lower.

Summary

Summary

Inkling 2 Mini has 24 layers and applies 12 of them three times. During prefill, all three passes are computed for every prompt token.

I wanted to know whether prefill needs all three. My prefill model computes one pass. For the other two, it applies only the key and value projections of each layer, to the hidden state from the first pass.

Serving engines can already run prefill on one GPU and decode on another, and the KV cache is the only data transferred. So the prefill GPU has the smaller model, and Inkling 2 Mini, unchanged, decodes from its cache.

With 32K-token prompts, time to first token is 325 ms instead of 776, and two GPUs serve twice as many requests. The benchmark score is 9 points lower. With my first version the MMLU-Pro score was 20 points lower, and I did not know, because the two tests I had written both passed.

Two phases

Prefill is most of the time, and its only output is the KV cache

Decode uses one result of prefill, the KV cache. With prefill and decode on separate GPUs, the KV cache is the only data transferred.

For long prompts, prefill takes most of the time. With Inkling 2 Mini and a 32K-token prompt, prefill takes 776 ms. After that, each output token takes 5 ms.

Prefill time also limits the request rate. With the original model, a second GPU for decode adds no capacity, because the decode GPU is idle most of the time.

One request to Inkling 2 Mini with 32K prompt tokens and 128 output tokens, alone on the server. Each thin bar is one output token.
The loop

Inkling 2 Mini is a looped model

Inkling 2 Mini has 24 layers and applies 12 of them three times, so a token passes through 48 layer applications, each with its own cache entry.

Only one in six of those uses attention over the whole prompt. The other 40 use attention over the last 1,024 tokens, and their cache entries hold 1,023 tokens. For a projection-only layer of that kind there is almost nothing to compute.

Prefill model

A prefill model with one pass of the loop

I did not want to train a second model from scratch, so I derived the prefill model from the original weights.

Inside one layer, the key and the value are two linear projections of the layer input. That part is small. Attention and the MLP take most of the time, and their output is the input of the next layer.

The prefill model computes layers 1 to 6 and one pass through layers 7 to 18. That is 18 of the 48 layer applications. The second pass, the third pass and layers 19 to 24 have only the two projections, without attention and the MLP. I will call those projection-only.

A projection-only layer should receive the output of the layer before it, and that output was never computed. On pass 2 and pass 3, layer 9 is the same layer with the same weights as on pass 1, so its input is the hidden state it had on pass 1. Layers 19 to 24 are never computed, and their input is the last computed hidden state. The keys and values are then approximate.

Inkling 2 Mini and the prefill model with 18 complete layer applications. Blue is computed in full. Red is filled by the key and value projections alone. Of the 30 red applications, 5 store the whole prompt and 25 store the last 1,023 tokens.
Training

Training the projections

To train the projections, Inkling 2 Mini predicts a continuation twice, once from its own cache and once from the cache of the prefill model. The loss is the KL divergence between the two sets of predictions. The loss does not compare the two caches.

One training step. The gradient passes through the frozen original model and the KV cache. Only the projection-only layers are updated.
Without a loop

Without a loop: Qwen3.6-35B

Before Inkling 2 Mini I tried the same construction on Qwen3.6-35B, which has 40 layers and no loop. The first 8 were complete and the others projection-only. In SGLang, time to first token was about half the original's.

Then I compared it with the plain alternative, which is two ordinary servers. One GPU with the original model serves 2.5 requests per second. My pair of GPUs serves 2.1 per GPU. That is worse.

The reason is in the architecture. 30 of the 40 Qwen layers have a recurrent state, and that state has to be updated over every prompt token. Most of the prefill time was in layers I thought I had removed.

Prior work

Who did this first

I found the closest work after I had built mine.

SwiftKV computes the first layers on the prompt. The later layers' cache is filled with projections of one earlier layer's output, trained by distillation. That is the design I ended up with. In SwiftKV the served model itself is changed. KV Prediction predicts the cache of a frozen larger model with a small model and learned linear maps, which was my original idea.

In more recent papers layers are skipped during prefill without training, POP and RidgeFill among them.

Four things are different here. The model is a looped one, and 24 of the 30 projection-only layer applications are repeats of layers that were computed. The decoder is the unmodified original. The prefill model is a separate server inside a production engine. And the evaluation is on public benchmarks with a reasoning model.

Serving

Serving it in SGLang

Timings from a PyTorch script are a poor guide to serving speed, so I ported the prefill model to SGLang as an external model package.

With the original model on the prefill GPU, the output tokens through my transfer code are identical to those from the unmodified engine. Any difference in output is therefore caused by the prefill model.

Benchmarks

The tests I wrote passed. The benchmarks did not.

I had two tests. One has a fact at a random position in a 32K-token ledger and a question about it. The other compares next-token predictions on web text. With the 18-layer prefill model, both passed.

Then I ran LongBench v1, LongBench v2 and MMLU-Pro. The MMLU-Pro score was 20 points below the original, and the LongBench v2 score was 50 points below.

The outputs were not wrong answers. They contained no answer. Inkling 2 Mini generates reasoning tokens before an answer, and with the approximate cache it generated them up to the 1,800-token budget. That was the case for 42% of MMLU-Pro questions, against 14% with the original cache.

In my tests the outputs were too short for this to occur. The lookup answer is three tokens long, and the agreement test covers 16 tokens.

Post-training

Post-training on the original model's own answers

I built about 2,000 prompts in the formats of the benchmarks, from the training splits of other datasets. The original model generated an answer to each one, reasoning included, and those answers are the training target.

On these prompts I measured the same failure. With a 1,024-token budget, 5% of the original model's outputs reached the budget. With the cache of the untrained prefill model the share was 34%.

Training took 800 steps, about an hour on one GPU.

Full benchmarks through the live prefill and decode servers. The questions and the 1,800-token budget are the same for every model. An output that reaches the budget without an answer counts as wrong.
Result

Result

With 18 complete layer applications, scores are about 5 points below the original on LongBench v1 and MMLU-Pro, and 13 on LongBench v2. With 24, the largest difference is 6 points. On MMLU-Pro, 18% of outputs still reach the budget, against 42% before post-training.

Two things I tried on top did not help. Training on the over-long outputs needed 1,600 steps to match the result of plain training after 800. And 4,000 more prompts of the same kind left the divergence unchanged at 0.07 after 400 steps.

Scaling

How it scales with the size of the prefill model

I trained seven sizes with the same recipe, from 6 to 42 complete layer applications.

Each point in the chart at the top of the page is one of them. The benchmark score there is the mean of LongBench v1, LongBench v2 and MMLU-Pro, on 150 questions each.

Time to first token is close to linear in that number. Divergence from the original is not. After training, nine more layer applications mean about half the divergence.

Inkling 2 Mini reuses its middle layers, so from 18 layer applications up the prefill model has the same 19 billion parameters and only more computation.

Seven prefill model sizes. Left: 48 held-out benchmark questions. Middle and right: served in SGLang on two GPUs.
Adapter

Is a linear adapter necessary?

My original idea was a linear adapter between caches: write the deep layers' keys and values as a linear map of the shallow layers'. I tested it against the projections, at 18 complete layer applications and with the same training.

No. With projections alone the divergence after training is 0.07. A linear map added on top does not lower it. A linear map in place of the projections triples it.

A linear map cannot rebuild the whole cache. Fitted in closed form on the original model's own cache, the best linear map explains at most 75% of the variance of the skipped layers' keys and 49% of their values.

In one layer I compared the keys directly. By squared distance, the linear map's keys are closer to the true keys than the trained projections' keys are. They are also a scaled-down copy, with 60% of the variance between tokens, where the projections have 97%. The original model decodes three times better from the projections.

KL divergence from the original model after 800 steps, 18 complete layer applications. Lower is better.
Cite
@misc{korolev2026cachetransfer,
  author       = {Korolev, Konstantin},
  title        = {{Prefill a looped model with one pass of the loop}},
  year         = {2026},
  month        = nov,
  howpublished = {\url{https://advprop.ai/notes/cache-transfer/}},
  url          = {https://advprop.ai/notes/cache-transfer/},
  note         = {White Circle}
}

Konstantin Korolev, White Circle.

I am hiring at White Circle. Write to konstantin [at] whitecircle.com.

Complementary pair

Drag the selected point. The linked point tracks its exact HSL complement.