There is no prefill phase
The engine loop. One counter per request, one token budget per step, and why the two words every explanation is built on name nothing in the scheduler.
Inference enginesSeries / Inside vLLM
Inside vLLM
1 of 8 publishedMost people running vLLM in production can tune it and cannot read it. This series goes through the V1 engine one component at a time — request in, token out — from the source rather than the paper. Every claim is pinned to a public commit, so you can open the file next to the post and check it.
The prefix cache is a dict
The block pool, the chained block hash, and the free-block queue that doubles as eviction order. It isn't the radix tree you're picturing, and a cache hit emits no event.
Where the memory actually goes
How vLLM decides how many KV blocks exist: the profiling run, gpu_memory_utilization, and the block-size math that hybrid Mamba models force on everyone else.
The scheduler, line by line
The full schedule() walk — running queue, waiting queue, encoder budget, preemption — and the SchedulerOutput contract that is the entire interface to the GPU.
From SchedulerOutput to a CUDA graph
The model runner: persistent input batch, block tables, attention metadata, graph capture and dispatch, and why the batch gets padded.
Why there are nine attention backends
The backend interface and the metadata-builder contract. What actually differs between FlashAttention, FlashInfer and Triton, and how one gets picked.
Sampling and the way out
Logits processors, the structured-output bitmask, logprobs, detokenization, stop strings on the frontend, and the chunking that keeps one big step off the event loop.
Scaling out
Tensor, pipeline, data and expert parallelism as executors. The coordinator process, the worker processes, and where KV connectors plug in for disaggregated prefill.
New parts land here first. RSS if you want them as they publish.