Raghav Potluri

There is no prefill phase

2026-07-24T00:00:00+00:00

There’s a comment sitting above vLLM’s main scheduling loop that explains the engine better than any architecture diagram of it:

There’s no “decoding phase” nor “prefill phase” in the scheduler. Each request just has the num_computed_tokens and num_tokens_with_spec. […] At each step, the scheduler tries to assign tokens to the requests so that each request’s num_computed_tokens can catch up its num_tokens_with_spec.

— vllm/v1/core/sched/scheduler.py:429-438

Prefill and decode are the two words every explanation of LLM inference is built on, mine included. They name nothing in vLLM’s scheduler. There is a counter per request and a token budget per step, and the phases are what you get when you plot the result.

That gap — between the model everyone teaches and the loop that actually runs — is why the engine looks like a black box from outside. This is the first of a series where I go through the V1 engine one component at a time, reading the code instead of the paper. The loop comes first because every other component hangs off it.

Everything below is pinned to 6a1acac on main, 1,469 commits past the v0.23.1rc0 tag. Line numbers move; the SHA doesn’t.

The phases are real, but they aren’t control flow

Start with what the two words are actually for, because they’re not wrong — they name two shapes of matrix multiply.

You send a 2000-token prompt. To produce the first output token, the model has to compute attention keys and values for all 2000 positions. That’s one forward pass with 2000 tokens in it. Each weight matrix gets read from HBM once and multiplied against a 2000-row activation matrix. The GPU is doing enough arithmetic per byte loaded to keep its tensor cores busy. That’s prefill, and it’s compute-bound.

Now the second output token. The keys and values for those 2000 positions are already sitting in GPU memory, so the model computes over exactly one new position. One forward pass, one token. Every weight matrix still gets read from HBM in full — all ~140 GB of them for a 70B model in bf16 — and gets multiplied against a single row. The arithmetic is trivial; the memory traffic is identical. That’s decode, and it’s memory-bandwidth-bound — on an H100 SXM it’s the 3.35 TB/s of HBM bandwidth that sets the ceiling, not the tensor cores.

Two shapes, two bottlenecks, both real. The mistake is carrying that distinction up into your mental model of the scheduler and assuming the engine runs a prefill stage and then a decode stage. It doesn’t. It runs one loop, and the loop counts tokens.

One counter, one budget

Every request carries num_computed_tokens — how many of its tokens the model has KV entries for — and num_tokens_with_spec, which is len(prompt) + len(output_so_far) + len(draft_tokens). The scheduler’s entire job is closing that gap.

Each step it starts with a budget, token_budget = self.max_num_scheduled_tokens, which falls back to max_num_batched_tokens when you haven’t set it. Then it walks the running queue first, and for each request asks for the gap, clipped:

num_new_tokens = (
    request.num_tokens_with_spec
    + request.num_output_placeholders
    - request.num_computed_tokens
)
if 0 < self.scheduler_config.long_prefill_token_threshold < num_new_tokens:
    num_new_tokens = self.scheduler_config.long_prefill_token_threshold
num_new_tokens = min(num_new_tokens, token_budget)

scheduler.py:504-511

A request that has generated 40 tokens and needs its 41st has a gap of 1, so it gets 1. A request that arrived this step with a 16,000-token prompt has a gap of 16,000, gets clipped to whatever budget is left, and comes back next step for the rest. Same three lines. The first case is what we call decode and the second is what we call chunked prefill, and the code does not distinguish them — chunked prefill isn’t a feature bolted onto the loop, it’s what min(gap, budget) does when the gap is large.

Running requests are served before waiting ones (scheduler.py:473 opens that loop; the waiting queue is drained afterwards with whatever budget survives). That ordering is the whole scheduling policy for the common case: keep everything already generating alive, then admit new work with the leftovers.

Eight consecutive engine steps on a single GPU. A column is one execute_model call, and every column here mixes prefill chunks with single decode tokens — the scheduler has no notion of a prefill phase to separate them. Request D is preempted at N+4 to free blocks for C and E, and pays for it at N+6 by recomputing all 8189 of its tokens from scratch. The two decode-only steps schedule 3 and 4 tokens against an 8192 budget — under 0.05% of it — while still reading every weight in the model once.

Read that diagram down a column, not across a row. A column is one execute_model call — one batch, one set of kernel launches. Column N+2 contains request A finishing its prompt with a 512-token chunk, request C starting a 7678-token chunk, and requests B and D contributing one token each. There is no step in that picture that is “a prefill step.”

Your request is a row in a matrix that changes every step

The other half of the black box is that nothing in the engine owns your request end to end. It’s decomposed across two processes, and the boundary is worth knowing because it’s where most operational surprises live.

The API server process does HTTP, tokenization, and multimodal loading; the engine core process runs the scheduler and drives the GPU workers; they talk over ZMQ (docs/design/arch_overview.md). That’s not decoration. Separate processes mean separate GILs, so a burst of tokenization work in the frontend can’t stall the loop that’s feeding the GPU. It also means the thing you profile as “vLLM latency” is at least two queues deep.

sequenceDiagram
    autonumber
    participant C as client
    participant API as API server proc<br/>(tokenize, HTTP)
    participant EC as EngineCore proc<br/>(scheduler)
    participant W as worker proc<br/>(GPU)

    C->>API: POST /v1/chat/completions
    API->>API: tokenize → EngineCoreRequest
    API->>EC: ZMQ: add_request
    Note over EC: request lands in the<br/>waiting queue
    loop every engine step
        EC->>EC: schedule() → SchedulerOutput
        EC->>W: execute_model(SchedulerOutput)
        W-->>EC: sampled token ids
        EC->>EC: update_from_output()
        EC-->>API: ZMQ: EngineCoreOutputs
        API->>API: detokenize, stop-string check
        API-->>C: SSE chunk
    end

The engine core side is a busy loop with two statements in it:

def run_busy_loop(self):
    """Core busy loop of the EngineCore."""
    while self._handle_shutdown():
        # 1) Poll the input queue until there is work to do.
        self._process_input_queue()
        # 2) Step the engine core and return the outputs.
        self._process_engine_step()

core.py:1358-1364

And a step, in full (core.py:576-606):

flowchart LR
    A["has_requests()?"] -->|no| Z["return, poll input queue"]
    A -->|yes| B["scheduler.schedule()<br/>→ SchedulerOutput"]
    B --> C["model_executor.execute_model()<br/>non_block=True"]
    C --> D["get_grammar_bitmask()<br/>overlaps the GPU"]
    D --> E["future.result()<br/>+ sample_tokens()"]
    E --> F["scheduler.update_from_output()<br/>→ EngineCoreOutputs"]
    F --> A

SchedulerOutput is the entire contract between the scheduler and the GPU: which requests, how many tokens each, which KV blocks they own. The worker doesn’t know about queues or priorities. The scheduler doesn’t know about kernels. That seam is why vLLM can swap attention backends and hardware platforms without touching scheduling logic — and it’s the seam I’ll follow in the next few posts.

Two details in step() are worth naming, because they’re the kind of line you read straight through. execute_model is called with non_block=True, and the grammar bitmask for structured output is computed while the GPU is busy — CPU work deliberately overlapped with the forward pass. And aborts are drained after the model returns but before update_from_output, so a client that disconnects mid-step doesn’t corrupt the accounting for the step already in flight.

What the shape buys, and what it costs

The payoff for collapsing everything into one counter is that features that sound independent stop needing separate machinery. Chunked prefill is min(gap, budget). Prefix caching is starting a request with num_computed_tokens already above zero. Speculative decoding is a gap of 1 + num_speculative_tokens instead of 1, which is why num_tokens_with_spec and not num_tokens is the target in the first place. Three features, one subtraction.

The cost shows up in two places, and both are visible in the diagram.

Preemption is amnesia. When the KV cache runs out of blocks, the scheduler evicts a running request. _preempt_request frees its blocks, sets request.num_computed_tokens = 0, and prepends it to the waiting queue. There’s no swap to host memory on this path — the KV cache for those tokens is gone, and the request re-prefills its entire context from scratch. That’s request D in the diagram paying 8189 tokens at step N+6 for tokens it had already computed by step N. If your p99 has a cliff under load, the counter declared as vllm:num_preemptions — vllm:num_preemptions_total once Prometheus suffixes it — is the first thing to look at, and the fix is usually more KV blocks or a lower max_num_seqs, not a bigger batch.

Decode-only steps waste almost the entire budget. Steps N+4 and N+7 schedule 3 and 4 tokens against a budget of 8192, and each one still pulls the model’s whole active weight set out of HBM — every weight, in a dense model. Prefill work is what fills that gap, which is the real argument for chunked prefill: not that long prompts get chopped up, but that the chopped pieces ride along in steps that were going to be memory-bound anyway.

That budget number isn’t arbitrary either, and the way it’s chosen is a small lesson in not trusting defaults:

if device_memory >= 70 * GiB_bytes and "a100" not in device_name:
    # For GPUs like H100 and MI300x, use larger default values.
    default_max_num_batched_tokens = {
        UsageContext.LLM_CLASS: 16384,
        UsageContext.OPENAI_API_SERVER: 8192,
    }

arg_utils.py:2478-2483

Same GPU class, but the offline LLM path gets twice the budget the OpenAI server gets, because the server is optimizing for time-to-first-token and the offline path for throughput. And the A100 is carved out of the large default by a substring match on the device name, because raising the budget there lowered throughput — PR #17885 measured Llama-3.1-70B at TP=2 going from 1054 to 1438 tok/s (+36%) by making the batch smaller. One if statement, three different answers, none of them in the flag’s help text.

What to carry into the rest of the series

If you take one thing from the loop: your request is not a unit of work in vLLM, it’s a row in a batch that is re-formed from scratch on every step. The engine holds no plan for it. Each step it re-decides who runs, and the only state carried forward is a counter and a set of KV blocks.

That reframing is what makes the rest of the system readable. “How much memory does my request use” becomes a question about the block pool. “Why did TTFT spike” becomes a question about how many steps your prompt waited for budget. “Why is my prefix cache not hitting” becomes a question about block hashes, which is the next post — the block pool and the flat dict that vLLM calls a prefix cache, and why it isn’t the radix tree you’re picturing.

Prefix-cache routing silently dies under disaggregation

2026-07-15T00:00:00+00:00

My prefix-cache hit rate was zero. Not low — zero, across every request and every workload, for hours. The router in front of the fleet was wired right, and its prefix index had 14,000 nodes in it. It took me too long to stop debugging the scorer and go read what Dynamo actually publishes, and where.

The answer is a one-token subject mismatch that no log line will ever tell you about. If you route by prefix cache in front of disaggregated vLLM, you can hit this and never know — your routing quietly degrades to round-robin and everything still returns 200s.

The setup

I run an external router in front of a Dynamo 1.2.1 deployment: prefix-aware placement, so a request lands on the worker that already holds its prompt prefix in KV and skips the recompute. To know which worker holds which prefix, the router builds a radix index from Dynamo’s KV-cache event plane — the BlockStored / BlockRemoved events vLLM emits and Dynamo republishes on NATS. Standard push model: workers publish cache state, the router subscribes and indexes it, routing reads the index.

This worked in aggregated mode. Then I moved the same deployment to prefill/decode disaggregation — split the workers, one pool computes prefill and ships KV over NIXL, another pool decodes — and prefix affinity went dead.

The symptom that looks like a scoring bug

The router’s per-pod match-fraction metric was a clean zero on every worker, every request. Not noisy-low — exactly zero. My first instinct was the scorer: a sign flip, a bad normalization, a lookup keyed wrong. That was the wrong instinct, and I burned real time on it.

Two facts eventually pointed away from the scorer:

The index was not empty — 14,000 nodes. So something was populating it.
Every node in it was keyed to a decode worker. Zero prefill-worker keys.

That second fact is its own small trap. In disaggregation the request’s routing primary is the decode worker (that’s where the response streams from), so the router’s own speculative fallback — the “I sent this prefix to worker X a moment ago, remember that” insert — was recording decode pods. But prefix affinity for prefill placement needs to know which prefill worker computed and cached the prefix. The index was full of the wrong role. And the authoritative signal that would have carried the right role — the engine’s own BlockStored events — was nowhere in it.

So the real question wasn’t “why is the scorer returning zero,” it was “why am I receiving zero KV-cache events.”

Read the source, not the tea leaves

Dynamo builds the KV-event subject in one place. In lib/llm/src/kv_router/indexer/subscriber.rs:

let kv_event_subject = format!(
    "namespace.{}.component.{}.{}",
    component.namespace().name(),
    component.name(),
    KV_EVENT_SUBJECT
);

and KV_EVENT_SUBJECT is "kv-events" (lib/kv-router/src/protocols.rs). So the full subject is:

namespace.<namespace>.component.<component>.kv-events

The load-bearing token is <component> — the worker’s Dynamo component name. And that name is not stable across serving topologies:

Aggregated vLLM registers as component backend — that’s the default (lib/backend-common/src/args.rs, DYN_COMPONENT defaults to "backend"; components/src/dynamo/vllm/args.py sets dynamo_config.component = "backend").
A disaggregated deployment overrides it per role: DYN_COMPONENT=prefill and DYN_COMPONENT=decode.

My subscription was still on component.backend.kv-events from the aggregated setup. Under disaggregation nothing publishes to backend anymore — the events are on component.prefill.kv-events and component.decode.kv-events. My subscriber matched nothing. Zero events, no error, index never learns a single prefill key.

Confirm it on the wire

Reading the format is a hypothesis; the bus is the proof. A raw NATS subscribe on > while firing a burst of identical-prefix requests, and there it is:

namespace.<ns>.component.prefill.kv-events      <== one per request
namespace.<ns>.component.prefill.forward-pass-metrics
namespace.<ns>.component.backend.forward-pass-metrics
namespace.<ns>.kv_metrics

Twelve requests, twelve kv-events, all on component.prefill.kv-events. (The namespace token also carries a worker-hash suffix, so match it with a wildcard, not a literal.)

Two things worth pinning here, because they surprised me:

Only the prefill worker publishes prefix events (in a standard disagg config). The decode worker is run with --no-enable-prefix-caching — it receives KV over NIXL, it doesn’t own or recompute the prefix, so it emits nothing useful for prefix routing. All the affinity signal lives on the prefill side. If you’re going to subscribe to one role, it’s prefill.
There’s a backend component still on the bus — but only for forward-pass-metrics, not kv-events. Enough to make you think you’re on the right namespace while you’re on the wrong subject. Don’t let the presence of a backend.* subject convince you your kv-events subscription is live.

The fix

One line — point the subscription at the prefill component:

component.backend.kv-events   →   component.prefill.kv-events

Events start flowing on the next request. The index keys on prefill workers. And the behavior you actually wanted shows up immediately: fire the same prefix repeatedly and the requests concentrate on the one worker that cached it, instead of spraying across the pool. Match rate goes from a flat zero to tracking the real reuse in the workload.

The lesson

Prefix-cache routing is only as good as the event-plane wiring underneath it, and disaggregation quietly re-topics that event plane per role. The failure mode is the nasty kind: it fails open. A wrong subject isn’t a crash or a refused connection — it’s a subscription that sits there healthy and receives nothing, so your smart router degrades to round-robin and every request still succeeds. Nothing in your logs says “you are no longer prefix-aware.”

Concrete takeaways if you build or operate this:

If your cache-hit rate is a clean zero, suspect the subject before you touch the scorer. Zero is a plumbing signature, not a scoring one. A real scorer bug is usually noisy-wrong, not perfectly-zero.
The component name is part of your contract. backend for aggregated, prefill/decode for disaggregated. If your subscriber hardcodes a component, it breaks silently the day someone splits the workers.
Subscribe to prefill for prefix affinity. In a standard disagg config the decode side has prefix caching off and publishes nothing you can route on.
Watch the bus, not only your metrics. A 30-second raw NATS subscribe told me in one shot what hours of staring at a zero gauge did not.

Everything above is upstream Dynamo/vLLM behavior traced to its own source; observed on Dynamo 1.2.1, with the file references pinned to that release’s commit below so they don’t rot as line numbers drift.

Source references, Dynamo 1.2.1, pinned to ai-dynamo/dynamo@919682d:

KV-event subject format — kv_router/indexer/subscriber.rs#L48
KV_EVENT_SUBJECT = "kv-events" — kv-router/src/protocols.rs#L18
DYN_COMPONENT default "backend" — backend-common/src/args.rs#L36
component set to "backend" — vllm/args.py#L168

What your router actually knows about the KV cache

2026-07-15T00:00:00+00:00

vLLM’s KV-event stream never fires on a cache hit. The single most common thing a prefix cache does — reuse a block that’s already resident — puts nothing on the wire. If you’re building a cache-aware router on top of that stream, that’s the first thing to know, because it bounds what your router can know.

vLLM emits exactly three events (vllm/distributed/kv_events.py): BlockStored, BlockRemoved, AllBlocksCleared. BlockStored fires once, in cache_full_blocks(), the first time a full block is hashed and inserted. Eviction fires BlockRemoved. A full reset_prefix_cache() fires AllBlocksCleared. That’s the entire vocabulary.

The hit path is touch(). It runs when a request reuses blocks already in the cache: it bumps the ref count, pulls the block off the free list if it was idle, records a metric. It does not append to the event queue — there is no emit anywhere in the function. It’s called from allocate_new_computed_blocks() to touch the blocks a request just matched in the prefix cache, so they aren’t evicted.

So a block announces itself once, when it’s first computed. Every reuse after that — the same request’s next segment, or a different request sharing the prefix — is invisible. A consumer building a global index from these events learns which blocks exist and on which pod. It learns nothing about how recently any of them was touched. That’s fine if all you want is “who has this prefix.” It’s a hole the moment you want to reason about staleness: the schema carries no recency signal, and the hit that would carry it never fires.

SGLang has the same gap, in the same place — and that’s not convergence. Its match_prefix() → _match_prefix_helper() updates last_access_time on every node it walks, and emits nothing. SGLang’s kv_events code is a credited, near-verbatim port of vLLM’s: PR #6098’s own description says “Event classes and publisher code borrowed from vllm-project/vllm#16750.” Same class names, same fields, same buffer_steps=10_000, same hwm=100_000, same port 5557. The blind spot isn’t a vLLM oversight you switch engines to escape. It’s a property of the shared design. Both engines model the stream as births and deaths of blocks, never traffic on blocks.

One more thing the stream won’t hand you: recovery has a floor. It isn’t pure fire-and-forget — the publisher pairs a zmq.PUB socket with a zmq.ROUTER for replay, stamps each batch with a sequence number, and keeps the last 10,000 batches in a ring buffer, so a subscriber that sees a gap can request a replay from its last sequence. But if the gap is older than that window, there’s no snapshot to fall back on — NVIDIA’s own Dynamo-vs-vLLM replay comparison says vLLM’s consumer has “no built-in initial state sync” and “must rebuild state through other means.” Dynamo’s own indexer has that fallback: ask for a range that’s too old and you get a TreeDump, the full RadixTree as synthetic events. vLLM and SGLang have no TreeDump. That was deliberate — PR #16750 existed specifically to avoid pulling Dynamo’s Rust/NATS publisher into vLLM as a dependency — but it means the two ecosystems give you different reliability guarantees, not just different transports. No recency on hits, and no full resync past the buffer window. Build your index to tolerate both.

The structure under it isn’t a radix trie with RCU

Ask how these caches are built and you’ll hear “radix trie with RCU” — a prefix tree, lock-free reads racing a writer that publishes new versions. Clean mental model. It’s wrong for two of the three canonical open-source implementations, and the third doesn’t use RCU.

SGLang is the only one that’s actually a radix tree (radix_cache.py). And it has no locks — not lock-free-as-in-RCU, no concurrency primitives at all. Grep radix_cache.py and the C++ port (tree_v2.{h,cpp}) for mutex|lock|rcu|atomic and the only hit is lock_ref, which is a reference count for eviction, not a mutex. It gets away with that because the tree is never shared across threads: SGLang’s scheduler owns it and drives match, insert, and evict from a single loop. There’s no reader/writer race to solve, so there’s no lock.

vLLM v1 isn’t a tree at all. Its prefix cache is a flat dict — BlockHashToBlockMap, hash → block, degrading to a nested dict only on hash collision. Same single-writer story: one EngineCore process, one scheduling loop, no lock.

The one component that genuinely has a concurrency problem is llm-d’s cross-pod indexer (llm-d/llm-d-kv-cache) — because that’s where many pods’ event streams write while many routing decisions read. It’s not a tree, and it’s not RCU. It’s a two-level LRU hash map with sharded locks: InMemoryIndex scopes each mutex to a single key, Lookup takes no global lock, and the write path runs FNV-1a-sharded workqueues so events for one pod stay ordered while different pods parallelize. Textbook sharded-lock table, not a versioned structure with grace-period reclamation. Nobody solved lock-free concurrent tree reads here. They sidestepped the problem three different ways: single writer thread, SPMD replication, per-key locks.

What actually makes it interoperate is the hash, not the tree

The thing that lets a router index another engine’s cache isn’t the data structure — it’s the block hash. vLLM keys each block by a chained hash over (parent_block_hash, token_chunk, extra_keys), sha256_cbor or xxhash_cbor, seeded off PYTHONHASHSEED. llm-d’s indexer recomputes the same shape — FNV-64a over the CBOR encoding of (parent, chunk, extra), block size 16 — and its config explicitly aligns the seed to vLLM’s PYTHONHASHSEED. That wire-compatibility is the load-bearing part. The index is a map of hash → pod, and it works only because two separate codebases agree, byte for byte, on how to hash a block.

I came at this from the metrics side. My merged PR to llm-d-inference-sim (#358) adds vllm:prefix_cache_hits and vllm:prefix_cache_queries counters, so rate(hits) / rate(queries) reproduces vLLM’s token-level cache-hit rate in Prometheus. The simulator’s synthetic cache doesn’t model a radix tree — it calls llm-d’s own chunked-hash token processor (TokensToKVBlockKeys) to turn tokens into block keys. Even the simulator is built on hash-chain block indexing. Once you leave SGLang, the trie is the exception, not the rule.

Two corrections to carry if you’re building on this stack. The event stream tells you a block was born and when it died, never that it was used, and it won’t resync you past its buffer — so hold recency and staleness in your own logic, not the stream’s. And the structure underneath is probably a sharded hash map, not a radix trie — the interop that lets your router index another engine’s cache lives in the block hash.