# Paged KV cache append (`kv_cache_append`) | ServingStudio Kernel Library

Write the keys and values of new tokens into their slots in a paged KV cache.

After the K and V projections, each new token's key and value are stored in the paged cache for later attention. A slot mapping names one cache slot per token, and block_size is the tokens per page. The measurement writes num_tokens tokens into distinct random slots of a cache with at least 256 pages. Keys have head_dim elements per head and values head_dim_v, which is left out when the two are equal.

Category: Attention; group: MHA / GQA. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=kv_cache_append. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `key_cache[slot[t]] ← key[t], value_cache[slot[t]] ← value[t], for each new token t`
- `GB/s = [num_tokens · num_kv_heads · (head_dim + head_dim_v) · (bytes(input_dtype) + bytes(kv_dtype)) + 8 · num_tokens] / time`

## How it is measured

vllm_cuda: GPU kernel time from CUPTI, averaged over repeated launches of reshape_and_cache_flash_kernel after five warm-ups, with the L2 cache flushed before each launch. vllm_triton_diffkv: the same for the Triton reshape_and_cache_kernel_flash_diffkv, after one checked write. torch: CUDA-event time around a loop of calls after five warm-ups, the median of three loops; the cache is not flushed, and the time includes the gap between its two launches.

## Caveats

- The two backends are timed differently, cold kernel time against a warm loop, so their numbers are not directly comparable.
- The torch backend only covers matching input and cache dtypes with one scale per tensor.
- vllm_cuda uses K and V scales of 1.
- vllm_cuda needs head_dim_v == head_dim; vllm_triton_diffkv writes one packed \[num_blocks, block_size, num_kv_heads, head_dim + head_dim_v\] cache.
- GB/s counts the K and V reads and writes and the int64 slot map, not the scales.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_kv_heads` | number | heads | KV heads written for each token. |
| `head_dim` | number | elements | Elements in each key head. |
| `block_size` | number | tokens | Token slots in each cache page. |
| `input_dtype` | dtype |  | Element type of the new keys and values. |
| `kv_dtype` | dtype |  | Element type stored in the KV cache. |
| `cache_layout` | label |  | Physical cache layout, NHD or HND. |
| `scale_granularity` | label |  | FP8 scale granularity, tensor or head. |
| `num_tokens` | number | tokens | New tokens whose keys and values are written. |
| `head_dim_v` | label | elements | Elements in each value head; absent when equal to head_dim (symmetric K/V). |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `torch`: PyTorch indexed assignment writes K and V in two launches. dtypes `bf16`, KV dtypes `bf16`, any CUDA GPU.
- `vllm_cuda`: vLLM's reshape_and_cache_flash CUDA kernel writes both K and V to mapped slots. dtypes `bf16`, KV dtypes `bf16`, `fp8_e4m3`, any CUDA GPU. Source: https://github.com/vllm-project/vllm/blob/main/csrc/libtorch_stable/cache_kernels.cu
- `vllm_triton_diffkv`: vLLM's Triton diffkv kernel writes K and V, of their own head sizes, into one packed cache in a single launch. dtypes `bf16`, KV dtypes `bf16`, any CUDA GPU. Source: https://github.com/vllm-project/vllm/blob/main/vllm/v1/attention/ops/triton_reshape_and_cache_flash.py

## Measured rows

1,131 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `vllm_cuda` | `bf16` | 280 |
| NVIDIA B200 | `vllm_triton_diffkv` | `bf16` | 210 |
| NVIDIA H200 | `vllm_cuda` | `bf16` | 641 |

## Used by these deployments

- [GLM-5.3 NVFP4: vLLM, TP and EP, DFlash speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2.md) (`GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2`)
- [Llama 3.1 8B: Single GPU](https://servingstudio.cs.washington.edu/models/Llama-3.1-8B/llama3_dense.md) (`Llama-3.1-8B/llama3_dense`)
- [Llama 3.1 8B: TP](https://servingstudio.cs.washington.edu/models/Llama-3.1-8B/llama3_dense_tp.md) (`Llama-3.1-8B/llama3_dense_tp`)
- [Llama 3.1 8B: DP attention, TP FFN](https://servingstudio.cs.washington.edu/models/Llama-3.1-8B/llama3_dp_attn_tp_ffn.md) (`Llama-3.1-8B/llama3_dp_attn_tp_ffn`)
- [MiMo V2.6 Flash RL: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/MiMo-V2.6-Flash-RL/mimo_v2_vllm_fp8_swa_moe.md) (`MiMo-V2.6-Flash-RL/mimo_v2_vllm_fp8_swa_moe`)
- [Qwen3 235B-A22B FP8: Attention side of an attention–FFN split](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_attn_tp.md) (`Qwen3-235B-A22B-FP8/qwen3_attn_tp`)
- [Qwen3 235B-A22B FP8: DP attention, EP MoE, DeepGEMM experts](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_moe_fp8_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-FP8/qwen3_moe_fp8_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B FP8: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_vllm_moe_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-FP8/qwen3_vllm_moe_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B Thinking 2507 FP8: DP attention, EP MoE, DeepGEMM experts](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_moe_fp8_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_moe_fp8_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B Thinking 2507 FP8: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_vllm_moe_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_vllm_moe_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B: Attention side of an attention–FFN split](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B/qwen3_attn_tp.md) (`Qwen3-235B-A22B/qwen3_attn_tp`)
- [Qwen3 235B-A22B: DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B/qwen3_moe_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B/qwen3_moe_dp_attn_ep_ffn`)
- [Qwen3 Coder 480B-A35B: Attention side of an attention–FFN split](https://servingstudio.cs.washington.edu/models/Qwen3-Coder-480B-A35B-Instruct/qwen3_attn_tp.md) (`Qwen3-Coder-480B-A35B-Instruct/qwen3_attn_tp`)
- [Qwen3.6 35B-A3B FP8: Single GPU](https://servingstudio.cs.washington.edu/models/Qwen3.6-35B-A3B-FP8/qwen36_local.md) (`Qwen3.6-35B-A3B-FP8/qwen36_local`)
- [Qwen3.8 27B NVFP4: vLLM, one GPU](https://servingstudio.cs.washington.edu/models/Qwen3.8-27B-NVFP4/qwen35_vllm_nvfp4_gdn_dense.md) (`Qwen3.8-27B-NVFP4/qwen35_vllm_nvfp4_gdn_dense`)
- [Qwen3.8 Flash Next NVFP4: vLLM, one GPU, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative.md) (`Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/kv_cache_append
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/kv_cache_append/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/kv_cache_append/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/kv_cache_append.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
