vLLM 0.31 KV Cache: Where Paged Blocks Actually Save
TL;DR: paged allocation moves the concurrency ceiling from "reserve for the longest sequence" to "allocate what is actually used" (24 to 157 in my run), and prefix caching lets a shared prefix be reused in whole blocks (a 1,000-token prefix loses just 0.80% at the default block size of 16). Four real errors are traced back to source.
Three boundaries before we start, so you do not reproduce something different.
First, every measured number here comes from an allocator simulation I actually ran on CPU (script in section 3, standard library only, copy and paste to reproduce). It is a simulation of allocator behavior, not a GPU throughput benchmark, and I am not going to dress it up as "vLLM measured throughput". Second, this is a pure technical teardown: the body contains no product or affiliate links, and the box at the bottom of the page is the site-wide promo slot. Third, all version numbers come from GitHub Releases: at the time of writing the newest is v0.31.0, published 2026-10-05. While researching I found third-party tutorials calling the current stable release v0.20.1 and v0.25.1 respectively, disagreeing with each other and with the official Releases page. For anything version-related, trust the official Releases and the docs for that version.
1. The biggest trap: the official doc says it no longer describes the code
Almost every second-hand explanation of PagedAttention traces back to https://docs.vllm.ai/en/latest/design/paged_attention/. That page now carries a Warning at the top saying, in effect: this is a historical document based on the original vLLM paper, and it no longer describes the code used in vLLM today; vLLM now uses its own multi-head query attention kernel in csrc/attention/attention_kernels.cu. The date stamped at the bottom of the page is June 24, 2026.
So the "v1 kernel / v2 kernel three-loop" narrative you read in derivative articles is officially marked as history. That does not retire the block concept — quite the opposite. The definition of Block in that same document still holds: key and value are stored in separate blocks, each block holds a fixed number (BLOCK_SIZE) of tokens for one head; with BLOCK_SIZE=16 and HEAD_SIZE=128, one block holds 16 x 128 = 2048 elements for one head.
I put this first because it decides how you read second-hand material: learn the mechanism from it, but do not try to match its kernel details against today's source.
2. Blocks, block tables, and logically contiguous but physically scattered
Only three concepts matter.
Block: the minimum allocation unit of the KV cache, a fixed number of tokens. In the v0.31.0 source, the CacheConfig.block_size field defaults to None, and when you do not set it explicitly it falls back to the class constant DEFAULT_BLOCK_SIZE = 16 (vllm/config/cache.py). Two other defaults in the same file: gpu_memory_utilization = 0.92 and enable_prefix_caching = True.
Block table: the request-to-physical-block mapping, the layer that makes memory logically contiguous but physically scattered. The official prefix caching document gives the v1 skeleton (simplified): a Block Pool (a list of KVCacheBlock), a Free Block Queue threaded directly through doubly linked list pointers (so moving a middle element to the tail is O(1)), a cache blocks map (hash to block id), and a request blocks map (request id to allocated block ids).
Startup log: at boot vLLM prints a line shaped like GPU KV cache size: N tokens, Maximum concurrency for M tokens per request: X.XXx (vllm/v1/core/kv_cache_utils.py). That is your first-hand evidence for how much concurrency a box can actually carry, and it beats any second-hand rule of thumb.
3. Measured: two allocators head to head
3.1 Setup (nailing down the boundaries first)
- Capacity: 100,000 token slots; with block size 16 that is 6,250 blocks
max_model_len: 4,096- Load: 400 requests, prompt and output lengths drawn from lognormal distributions with seed 20261013; 276,007 total tokens, mean 690, max 2,219
- Contiguous pre-allocation: every request grabs 4,096 slots on arrival and returns them in one piece at the end (the dominant approach before paged allocation)
- Paged on demand: prefill asks for
ceil(prompt/16)blocks, and decode asks for one more block each time it crosses a block boundary; if no block is available it triggers preemption (return every block, requeue), matching the V1 default RECOMPUTE semantics
Not included: prefill peak activations, CUDA graph memory, model weights, real scheduler policy. It answers exactly one question: given the same capacity, how much more concurrency does a different allocation strategy buy.
3.2 Result: peak concurrency 24 to 157
| Allocator | Peak concurrency | Steps to finish 400 requests | Avg queue wait (steps) | Internal waste |
|---|---|---|---|---|
| Contiguous (4,096 per request) | 24 | 3,524 | 178.9 | — |
| Paged on demand (block = 16) | 157 | 1,264 | 180.3 | 1.02% |
Same pool, peak concurrency climbs from 24 to 157, and the steps to drain the same load drop from 3,524 to 1,264. The cost is 1.02% internal waste — the unused tail of each request's last block.
One number that is easy to skip: peak block usage under paged allocation is 6,247 of 6,250, or 99.95%. The pool runs essentially full, which is exactly where the 4 preemptions later in this run come from.
3.3 How block size drives internal waste
| Block size | Peak concurrency | Internal waste | Preemptions |
|---|---|---|---|
| 8 | 158 | 0.50% | 6 |
| 16 | 157 | 1.02% | 4 |
| 32 | 155 | 2.18% | 2 |
| 64 | 153 | 4.58% | 7 |
| 128 | 147 | 9.08% | 8 |
Waste grows almost linearly with block size, matching the intuition that the last block wastes B/2 tokens on average. Preemption counts are not monotonic though (16 to 32 actually drops from 4 to 2, then 64 climbs back to 7): with only 400 requests and preemption being a rare event, these counts do not support a claim that bigger blocks cause more preemption. I am reporting the numbers, not drawing that conclusion.
3.4 A cheap stop-loss: watermark
The official docs define watermark as the fraction of total KV cache blocks to keep free when admitting waiting or preempted requests, default 0.0, meaning disabled. I raised it from 0 in the simulation:
| watermark | Peak concurrency | Preemptions | Steps to finish 400 requests |
|---|---|---|---|
| 0.00 | 157 | 4 | 1,264 |
| 0.01 | 156 | 0 | 1,264 |
| 0.02 | 154 | 0 | 1,264 |
| 0.05 | 152 | 0 | 1,265 |
| 0.10 | 145 | 0 | 1,279 |
One slot of concurrency buys 4 preemptions down to 0, with no change in total time. This is the only parameter from this round I would proactively recommend trying first — assuming you actually see preemption in your logs.
3.5 Prefix caching only counts whole blocks
The official prefix caching document has one decisive sentence: We only cache full blocks. So the effective cached length of a shared prefix equals floor(prefix_length / block_size) x block_size:
| Shared prefix (tokens) | block = 8 | block = 16 | block = 32 | block = 64 |
|---|---|---|---|---|
| 777 | 776 (lose 0.13%) | 768 (lose 1.16%) | 768 (lose 1.16%) | 768 (lose 1.16%) |
| 1,000 | 1,000 (lose 0%) | 992 (lose 0.80%) | 992 (lose 0.80%) | 960 (lose 4.00%) |
| 1,500 | 1,496 (lose 0.27%) | 1,488 (lose 0.80%) | 1,472 (lose 1.87%) | 1,472 (lose 1.87%) |
| 3,000 | 3,000 (lose 0%) | 2,992 (lose 0.27%) | 2,976 (lose 0.80%) | 2,944 (lose 1.87%) |
In practice: with a system prompt around 1,000 tokens and the default block size of 16, a hit loses at most 8 recomputed tokens, which you can ignore. Push the block size to 64 for some other reason and the same prefix loses 40 tokens (4.00%).
Two more things in that same document that are easy to miss:
- Each block hash is built from the parent block hash, the tokens in the block, and extra hashes (LoRA IDs, multi-modality input hashes, cache salts). Since v0.11 the default algorithm is sha256, switchable with
--prefix-caching-hash-algotosha256_cbor/xxhash/xxhash_cbor. The docs also state plainly that a non-cryptographic hash raises collision risk and can leak private information in multi-tenant environments. - When tenants share an instance, the docs suggest attaching a
cache_saltto requests, injecting it into the hash of the first block so cached blocks are only reused among requests with the same salt. That blocks timing-based attacks where an observer infers cached content from latency differences.
3.6 Reproducible script
This is the script I actually ran. Standard library only; paste it in and every number above comes back.
"""vLLM-style KV cache block allocator simulation (stdlib only, reproducible)
Compares: contiguous pre-allocation vs paged on-demand allocation
"""
import random
B = 16 # block size in tokens (vLLM default)
POOL_TOKENS = 100_000 # total KV cache capacity in token slots
MAX_LEN = 4096 # max_model_len
N_REQ = 400 # number of requests
SEED = 20261013
def gen_workload(n=N_REQ, seed=SEED):
rng = random.Random(seed)
reqs = []
for i in range(n):
# prompt 60~1800 long tail; output 20~900 long tail
p = min(int(rng.lognormvariate(6.0, 0.7)), 1800)
o = min(int(rng.lognormvariate(4.8, 0.9)), 900)
reqs.append((max(p, 8), max(o, 8)))
return reqs
# ---------------- allocator 1: contiguous pre-allocation ----------------
class ContiguousPool:
def __init__(self, total):
self.free = [(0, total)] # list of free intervals (start, length)
def alloc(self, size):
for i, (start, length) in enumerate(self.free):
if length >= size:
if length == size:
self.free.pop(i)
else:
self.free[i] = (start + size, length - size)
return start
return None
def release(self, start, size):
self.free.append((start, size))
self.free.sort()
merged = []
for s, l in self.free:
if merged and merged[-1][0] + merged[-1][1] == s:
merged[-1] = (merged[-1][0], merged[-1][1] + l)
else:
merged.append((s, l))
self.free = merged
def used(self):
return sum(l for _, l in self.free)
# ---------------- allocator 2: paged on demand ----------------
class PagedPool:
def __init__(self, n_blocks):
self.n = n_blocks
self.free = n_blocks
def take(self, k):
if self.free >= k:
self.free -= k
return True
return False
def give(self, k):
self.free += k
def run_contiguous(reqs, total=POOL_TOKENS, max_len=MAX_LEN):
pool = ContiguousPool(total)
waiting = list(reqs)
running = [] # (start, prompt, out, generated, wait)
done = 0
peak = 0
waits = []
step = 0
while (waiting or running) and step < 200000:
step += 1
# admit at most one new request per step
if waiting:
p, o = waiting[0]
st = pool.alloc(max_len)
if st is not None:
waiting.pop(0)
running.append([st, p, o, 0, 0])
# advance
for r in running:
r[3] += 1
if r[3] >= r[2]:
pool.release(r[0], max_len)
waits.append(r[4])
r[3] = -1
running = [r for r in running if r[3] != -1]
done = len(waits)
for r in running:
r[4] += 1
peak = max(peak, len(running))
if not waiting and not running:
break
return {
"allocator": "contiguous",
"steps": step,
"done": done,
"peak_concurrency": peak,
"avg_wait_steps": round(sum(waits) / max(len(waits), 1), 1),
"utilization": round(100 * (1 - pool.used() / total), 1),
}
def run_paged(reqs, total=POOL_TOKENS, b=B, watermark=0.0):
n_blocks = total // b
pool = PagedPool(n_blocks)
reserve = int(watermark * n_blocks)
waiting = [[p, o, 0] for p, o in reqs]
running = [] # [prompt, out, stored, blocks, wait]
preempt_cnt = 0
peak = 0
peak_used = 0
waits = []
internal_waste = 0
step = 0
while (waiting or running) and step < 200000:
step += 1
# admission: prefill needs ceil(prompt/b) blocks
if waiting:
p = waiting[0][0]
nb = -(-p // b)
if pool.free - nb >= reserve and pool.take(nb):
r = waiting.pop(0)
running.append([r[0], r[1], r[0], nb, r[2]])
# one decode step
for r in running:
r[2] += 1
if r[2] % b == 1 and r[2] > r[0]:
if pool.take(1):
r[3] += 1
else:
# no free block -> preempt (v1 default RECOMPUTE: free all, requeue)
pool.give(r[3])
preempt_cnt += 1
waiting.insert(0, [r[0], r[1], r[4] + 1])
r[2] = -1
running = [r for r in running if r[2] != -1]
# completion
for r in running:
if r[2] - r[0] >= r[1]:
internal_waste += (b - r[2] % b) % b
pool.give(r[3])
waits.append(r[4])
r[2] = -2
running = [r for r in running if r[2] != -2]
for r in running:
r[4] += 1
peak = max(peak, len(running))
peak_used = max(peak_used, n_blocks - pool.free)
if not waiting and not running:
break
total_stored = sum(p + o for p, o in reqs[:len(waits)])
return {
"allocator": "paged-b%d" % b,
"steps": step,
"done": len(waits),
"peak_concurrency": peak,
"peak_blocks_used": peak_used,
"total_blocks": n_blocks,
"preemptions": preempt_cnt,
"avg_wait_steps": round(sum(waits) / max(len(waits), 1), 1),
"internal_waste_tokens": internal_waste,
"internal_waste_pct": round(100 * internal_waste / max(total_stored, 1), 2),
}
def block_sweep(reqs):
rows = []
for b in (8, 16, 32, 64, 128):
out = run_paged(reqs, POOL_TOKENS, b)
rows.append((b, out["peak_concurrency"], out["preemptions"],
out["internal_waste_pct"], out["done"], out["steps"]))
return rows
def prefix_granularity(prefix_len, b):
full = prefix_len // b
cached = full * b
return cached, prefix_len - cached, round(100 * (prefix_len - cached) / prefix_len, 2)
if __name__ == "__main__":
reqs = gen_workload()
tot = sum(p + o for p, o in reqs)
print("workload: n=%d total_tokens=%d avg_len=%.1f max_len=%d"
% (len(reqs), tot, tot / len(reqs), max(p + o for p, o in reqs)))
print()
print("A. contiguous:", run_contiguous(reqs))
print()
print("B. paged :", run_paged(reqs))
print()
print("C. block sweep (b, peak_concurrency, preemptions, waste_pct, done, steps)")
for r in block_sweep(reqs):
print(" ", r)
print()
print("D2. watermark sweep (wm, peak_concurrency, preemptions, done, steps)")
for wm in (0.0, 0.01, 0.02, 0.05, 0.10):
o = run_paged(reqs, POOL_TOKENS, 16, wm)
print(" wm=%.2f peak=%d preempt=%d done=%d steps=%d"
% (wm, o["peak_concurrency"], o["preemptions"], o["done"], o["steps"]))
print()
print("D. prefix granularity loss (prefix_len, b) -> (cached, lost, lost_pct)")
for p in (777, 1000, 1500, 3000):
for b in (8, 16, 32, 64):
c, l, pct = prefix_granularity(p, b)
print(" prefix=%4d b=%2d -> cached=%4d lost=%3d lost_pct=%.2f%%" % (p, b, c, l, pct))4. Troubleshooting: four real errors and their root causes
Error one: `ValueError: max_num_batched_tokens (X) is smaller than max_model_len (Y)`
Source: SchedulerConfig.verify_max_model_len() in vllm/config/scheduler.py at v0.31.0.
Trigger: chunked prefill disabled and max_num_batched_tokens < max_model_len.
Verbatim (concatenated): max_num_batched_tokens (X) is smaller than max_model_len (Y). This effectively limits the maximum sequence length to max_num_batched_tokens and makes vLLM reject longer sequences. Please increase max_num_batched_tokens or decrease max_model_len.
Why you hit it: chunked prefill is on by default in V1, so this normally stays quiet. The moment you disable it explicitly (to reproduce old scheduling behavior, say), you must guarantee max_num_batched_tokens >= max_model_len. The official optimization guide also warns that with chunked prefill off, if max_num_batched_tokens < max_model_len, vLLM may crash at server start-up. The same function carries an adjacent check: max_num_batched_tokens must not be smaller than max_num_seqs.
Error two: `ValueError: To serve at least one request with the model's max seq len (...)`
Source: vllm/v1/core/kv_cache_utils.py at v0.31.0.
Verbatim, condensed: To serve at least one request with the model's max seq len (N), (X GiB KV cache is needed, which is larger than the available KV cache memory (Y GiB). Based on the available memory, the estimated maximum model length is M. Try increasing gpu_memory_utilization (which also controls CPU memory on the CPU backend) or decreasing max_model_len when initializing the engine.
Meaning: there is not enough room to serve even one request at max_model_len. The pool cannot open for business.
Fix: the message hands you estimated_max_len. Lower max_model_len to that, or raise gpu_memory_utilization (default 0.92). The tail of the message links the official conserving-memory guide.
Error three: `Sequence group N is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space`
Source: a real warning quoted in the official optimization guide. A full line looks like: WARNING 05-09 00:49:33 scheduler.py:1057 Sequence group 0 is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space. This can affect the end-to-end performance. Increase gpu_memory_utilization or tensor_parallel_size to provide more KV cache memory. total_cumulative_preemption_cnt=1
Meaning: the pool ran out of blocks, the scheduler preempted a request, and it will be recomputed once space returns. V1 defaults to RECOMPUTE rather than SWAP because recomputation has lower overhead in the V1 architecture.
Fix: the docs list four levers — raise gpu_memory_utilization, lower max_num_seqs or max_num_batched_tokens, raise tensor_parallel_size, raise pipeline_parallel_size. My simulation suggests a cheaper fifth: set watermark to 0.01 first (see 3.4).
How to watch it: the docs note you can set disable_log_stats=False to log the cumulative preemption count, and vLLM exposes it as a Prometheus metric.
Error four: `RuntimeError: Cannot re-initialize CUDA in forked subprocess`
Source: the Warning in the official conserving-memory guide.
Cause: calling CUDA-related functions before initializing vLLM (the example in the docs is torch.accelerator.set_device_index), so CUDA cannot re-initialize in a forked subprocess.
Fix: control device visibility with the CUDA_VISIBLE_DEVICES environment variable, and leave CUDA alone until after vLLM is initialized.
5. A tuning table indexed by symptom
| Symptom | Look at first | What to change |
|---|---|---|
Fails at boot: cannot fit one max_model_len request | estimated_max_len in error two | lower max_model_len / raise gpu_memory_utilization |
| Preemption in the logs | cumulative preemption count (disable_log_stats=False) | watermark 0.01 → lower max_num_seqs → raise TP / PP |
| High TTFT, long prompts slow everyone | max_num_batched_tokens | docs suggest above 8192 for small models on large GPUs; with chunked prefill off it must be at least max_model_len |
| Shared prefix is not saving anything | prefix length divided by block size | keep the default block size 16; confirm enable_prefix_caching=True (already the default at v0.31.0) |
| Repeated restarts are slow | compile cache | reuse VLLM_CACHE_ROOT (default ~/.cache/vllm), or pass --kv-cache-memory to skip memory profiling; --enforce-eager starts fastest but costs steady-state decode performance |
| Optimization level | -O0 through -O3 | per the docs table: -O2 is the default, -O3 currently equals -O2, -O0 starts fastest with the lowest performance |
6. What this does not cover
An article that hides its scope misleads people, so here is what I deliberately left out:
- I did not run a real vLLM server on a GPU. Section 3 is entirely an allocator-behavior simulation: no real kernels, no real scheduler, no real memory footprint.
- Not covered: Mamba / hybrid KV cache (the docs have a separate hybrid KV cache manager design page), KV offloading, and disaggregated prefill/decode.
- Not covered: multi-GPU. TP and PP are listed only at the level the official docs describe; I measured no synchronization overhead. The docs themselves warn that too much TP adds sync overhead and PP adds latency.
- Block-size effects on real kernel performance (memory coalescing, warp assignment) are out of scope — that is kernel-layer work, and the official PagedAttention page only gives a high-level account of it.
FAQ
Q: Should I change the block size?
A: Leave the default 16 alone. In my data, 8 only drops waste from 1.02% to 0.50% while buying longer block tables and more metadata; past 64 waste jumps straight to 4.58%.
Q: Is prefix caching on by default?
A: At v0.31.0, yes: CacheConfig.enable_prefix_caching defaults to True. Older versions need an explicit --enable-prefix-caching; check the config class for the version you run.
Q: Why will my concurrency not go up?
A: Read the startup line GPU KV cache size: N tokens, Maximum concurrency for M tokens per request: X.XXx — that is theoretical concurrency computed at max_model_len. If your real requests are far shorter than max_model_len, actual concurrency lands well above it, and that gap is exactly what paged allocation buys you.
Q: How expensive is a preemption?
A: I cannot give a universal number; it depends on how many tokens the preempted request had already generated. What is fixed is the semantics: V1 defaults to RECOMPUTE, the preempted request is recomputed after it gets blocks back, and the official docs state plainly that this hurts end-to-end latency.
Wrapping up
If you keep one sentence: paged allocation kills the waste of reserving for the longest sequence, and prefix caching kills the waste of recomputing the same prefix — and both are quantifiable. The 24 to 157 and the 1.02% above are those two claims with numbers attached.
Further reading, all previously published on this site:
- A different local inference path from vLLM: choosing among three Docker Model Runner engines
- Attention from scratch: line-by-line NumPy checked against PyTorch
- Why RAG retrieval is wrong: three layers and five real errors
👉 Join Xiaomi MiMo Platform: Leading AI model platform with cost-effective inference
👉 Join Aliyun AI: Top AI products with exclusive coupons for business innovation
📌 This article was AI-assisted generated and human-reviewed | TechPassive — An AI-driven content testing site focused on real tool reviews
🔗 Recommended Tools
These are carefully selected tools. Using our affiliate links supports us to keep producing quality content: