Raghav Potluri

2026-10-28 · Disaggregation · 7 min

llm-d's precise index never sees the decode pod's reply

The decode pod that served turn 1 of a chat holds KV for nearly all of turn 2’s history — the prompt and the reply it generated, short of the final stop token. llm-d can route turn 2 straight back to it and skip the prefill pod and the KV transfer. With llm-d’s precise prefix index at its sample block size of 64, that doesn’t happen. vLLM reports decode-time KV 16 tokens at a time, llm-d drops every one of those events, and the router never learns the decode pod has the reply.

On Qwen3 the router’s view doesn’t matter. The chat template rewrites the reply on turn 2, and no prefix cache on any engine can match it.

The paper that sent me looking

A recent paper, “Not All Prefills Are Equal”, makes the case for keeping turn 2 on the decode side. In a prefill/decode split, the prefill pod (P) computes the prompt’s KV and ships it to the decode pod (D), which generates. KV only moves P to D. Turn 2’s prompt is turn 1’s prompt plus the reply plus the new message, and the reply’s KV only exists on D, so P recomputes at least the reply and the new message. If turn 2 lands on a different P, or turn 1’s blocks were evicted, P recomputes the whole history. Either way, D pulls the result back.

They measured how much a prefill on D slows D’s decoding. On one H100 running Llama 3.1 8B with a decode batch of 200, a full 1,024-token prefill adds about 48% to time per output token (TPOT). An append prefill of the same length — new tokens computed against KV already in the cache — adds about 2%. With 4 concurrent prefills, full reaches 57% and append 21%. Their abstract claims about 68% lower time-to-first-token (TTFT) for turn 2 onward.

llm-d already has the path

llm-d-router’s disaggregation profile handler runs in decode-first order by default. It picks the decode pod first, then asks a decider whether the request needs a prefill pod at all. The prefix-based decider counts how much of the prompt that decode pod is missing:

hitPrefixTokens := info.CachedBlockCount() * info.BlockSizeTokens()
nonCachedTokens := inputTokens - hitPrefixTokens
// ...
return nonCachedTokens >= d.config.NonCachedTokens, nil

When that’s false, the handler drops the prefill step and records skip_prefill, and D runs the append prefill itself. llm-d also ships an always-split decider, but its default P/D sample uses this one.

My first guess was the threshold

The sample P/D config sets nonCachedTokens: 16, which looked tiny for a chat turn, so I measured what a real turn 2 leaves uncached. I rendered turn 1 and turn 2 through each model’s own chat template and tokenizer. D holds the rendered turn-1 prompt plus the reply. It never runs a forward pass on the final stop token, so I left that out.

Then I counted what’s left of turn 2 past the last full 16-token block that matches. The reply was 128 tokens, and I used the tokenizer’s encoding of the reply text instead of the IDs the model sampled. That’s the best case for matching.

Model Follow-up message Its tokens Uncached on turn 2
Llama 3.1 8B Instruct “Thanks, makes sense.” 5 20
Llama 3.1 8B Instruct a one-line question 17 32
Llama 3.1 8B Instruct a three-line question 53 68
Qwen2.5 7B Instruct “Thanks, makes sense.” 5 30
Qwen2.5 7B Instruct a one-line question 17 42
Qwen2.5 7B Instruct a three-line question 55 80

A five-token “thanks” picks up about 10 tokens of turn markers from the template, plus whatever falls in the partial block at the end, which can be anything from 0 to 15. So it leaves 15 to 30 tokens uncached, depending on where the last block ends. With a threshold of 16, the decider splits every row in the table. That looked like the bug until I read the decider’s README. Its example uses nonCachedTokens: 512 and promptTokens: 1024, and wires the decider to the precise prefix cache. So 16 is a sample value, and the README already moves it.

The index drops every decode-time block

The precise prefix cache is llm-d’s index built from vLLM’s own KV-cache events, and it can know D holds the reply because it hears about the blocks the engine actually cached. vLLM’s allocator caches blocks up to request.num_tokens, which counts generated tokens, and each BlockStored event carries request.all_token_ids for its range.

cache_full_blocks emits one event per call. When a prompt’s KV lands, whether from D’s own prefill or from P in a split, one event covers all its full blocks. During decode, a block fills every 16 tokens, so each event carries one 16-token block. llm-d re-hashes events into its own block size, and the precise sample config sets that to 64. An event with fewer tokens than one llm-d block fails the digest check as incomplete_canonical_block and is skipped before any keys are computed. Every decode-time event is under 64 tokens, so every one is skipped.

I replayed that sequence through llm-d’s event pool and index in a unit test, with events shaped the way vLLM sends them (group_idx set, full-attention cache kind): one prefill event for the prompt, one 16-token event per reply block, then a lookup of the full prompt plus reply.

Prompt Reply llm-d block size Decode events skipped Tokens the index finds
64 64 16 0 128 of 128
64 64 64 4 64 of 128
80 64 16 0 144 of 144
80 64 64 4 64 of 144

At 64, the router sees D holding turn 1’s prompt rounded down to a 64-token boundary and none of the reply. The decider counts the whole reply as uncached. With the README’s example settings, a conversation under 1,024 tokens stays on D anyway because of the promptTokens gate. Above that, once a reply runs past somewhere around 430 to 500 tokens, depending on where the prompt ends, turn 2 crosses the 512 threshold and goes to P, even though D holds every token it needs. Nothing errors and routing still works. The only trace is llm-d’s kv_cache_events_stores_skipped_total counter with reason="incomplete_canonical_block", climbing by one for every block of every reply.

On Qwen3 the reply doesn’t match at all

With Qwen3-8B, turn 2 matched D’s tokens for the first 26 tokens and no further, whatever the follow-up message. Here’s what D holds after turn 1, and how the template renders the same exchange on turn 2, in non-thinking mode:

D holds : <|im_start|>user\nQ?<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\nA.
turn 2  : <|im_start|>user\nQ?<|im_end|>\n<|im_start|>assistant\nA.<|im_end|>\n<|im_start|>user\n...

The template puts an empty <think> block into the generation prompt, so the reply on D sits after it. When the same reply comes back as history, the template drops the <think> block, empty or not. In thinking mode it drops the model’s real reasoning the same way. The two sequences split right after the assistant header, so the reply’s KV on D matches nothing on turn 2, in any prefix cache. D can still run turn 2 without a transfer, but it recomputes the reply along with the new message, and the decider sees the whole reply as uncached. With a long reply, turn 2 goes to P, which never had the reply either.

vLLM already handles a decode-local turn 2

On vLLM, a request that reaches D with no kv_transfer_params is a plain local request. The NIXL connector returns 0, False under the comment “No remote prefill for this request”. The scheduler checks its local prefix cache before it asks the connector, so D computes only the suffix it lacks. The reply is in that cache because append_output_token_ids hashes each block as it fills. vLLM needs no change for a decode-local turn 2; the router has to send it there.

vLLM also merged #32553, which goes after the same waste from P’s side. With bidirectional_kv_xfer on (it’s off by default), turn 2 still goes to P, but P reads the existing KV from D over RDMA and computes only the new tokens. D then reads the new blocks back. Pulls shorter than kv_recompute_threshold (64 tokens by default) are recomputed instead, and D holds a finished request’s blocks for decoder_kv_blocks_ttl, 480 seconds by default, so P has something to read.

flowchart LR
  subgraph A["Every turn through P"]
    a1[P recomputes reply + new message] --> a2[KV P→D] --> a3[D decodes]
  end
  subgraph B["vLLM bidirectional"]
    b1[history KV D→P] --> b2[P computes new tokens] --> b3[new KV P→D] --> b4[D decodes]
  end
  subgraph C["Decode-local turn 2"]
    c1[D append-prefills new tokens] --> c2[D decodes]
  end

Bidirectional keeps prefill work off D and pays for it by moving the history to P and the new blocks back. Decode-local moves nothing and puts the append prefill into D’s running batch. The paper doesn’t compare against bidirectional. It merged on 2026-04-30, five days before the paper’s v2, and it’s the comparison I’d most like to see.

SGLang’s decode server can’t prefill

SGLang’s scheduler says so when you ask for a prefill delayer on a decode server: “no prefill scheduling path”. Requests whose KV arrives from P enter D through a PREBUILT batch that never runs a forward pass, and there’s no path for computing a new prompt there. The decode-side radix cache is off by default and marked experimental. With it on, D sets a request’s tree key to origin_input_ids + output_ids, reply included, and P starts sending from D’s matched prefix length. That cuts the transfer, but turn 2 still goes through P.

What I’d want from the paper

In 92.2% of their workload and load combinations, the setting with the best turn-2 TTFT isn’t the one with the best TPOT.

Their router is a lookup table built offline. It bins accumulated context three ways and the input/output ratio nine ways, snaps load to the nearest measured QPS, and stores a yes/no per cell. The authors say it “degrades gracefully but suboptimally when hardware or workload distributions drift far from the calibration set.”

They also test the static rule of sending every turn 2 to D, and on WildChat that rule is close. Their router has the best TTFT at 14 of 27 test points and the static rule at 13. On TPOT the router wins 12 to 5.

The proxy keys sessions by an MD5 hash of the first user message, which collides whenever conversations open with the same message, and templated agent flows do that all the time. llm-d’s decider reads the index’s view of the decode pod’s cache, so it needs no session table, as long as the index can see the reply.

Everything ran on one node of 4×H100 over NVLink, so the transfer cost across a real RDMA fabric between nodes never shows up. The number I’d reproduce first, across nodes, is the 48%-vs-2% result. Everything else in the paper depends on it, and I don’t have a mechanism for why an append prefill of the same 1,024 tokens costs D that much less than a full one.

If you want turn 2 to stay on D

Set llm-d’s block size equal to vLLM’s (16 by default) or to a divisor of it, point both decode-pod scoring and the decider at the precise index, and pick nonCachedTokens from a measurement on your model. No config under deploy/ does all three today:

# plus the token-producer and other plugins from the precise sample
- type: precise-prefix-cache-producer
  parameters:
    tokenProcessorConfig:
      blockSizeTokens: 16   # equal to vLLM's --block-size, or a divisor of it
- type: prefix-cache-scorer
  parameters:
    prefixMatchInfoProducerName: precise-prefix-cache-producer
- type: prefix-based-pd-decider
  parameters:
    nonCachedTokens: 512    # the README's example; measure your own
    prefixMatchInfoProducerName: precise-prefix-cache-producer

Then watch kv_cache_events_stores_skipped_total{reason="incomplete_canonical_block"}. If it climbs with every reply, the router can’t see what your decode pods hold. And if your model’s template rewrites earlier turns the way Qwen3’s does, render turn 2 and diff it against turn 1 before you count on any of this.

Source references, pinned: