# Paged decode attention (`flashinfer_attn_decode`) | ServingStudio Kernel Library

One query token per request attending over its paged KV cache, for a whole decode batch.

The attention step of decode for MHA and GQA models. Each request brings one new query token and reads its full KV cache from paged memory. The measurement uses a uniform batch: every request gets the mean KV length, total_tokens divided by batch_size, rounded down.

Category: Attention; group: MHA / GQA. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=flashinfer_attn_decode. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `O = softmax(q · Kᵀ / √head_dim) · V, per request and query head`
- `kv_len = max(1, ⌊total_tokens / batch_size⌋), q_len = 1`
- `TFLOPS = 4 · batch_size · kv_len · num_qo_heads · head_dim / time`
- `GB/s = bytes of q, K, V and O / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. The paged backends store K and V in 16-token pages (NHD layout), and GB/s counts every allocated page. FlashInfer runs with use_tensor_cores=True. fa3 and fa2_cudagraph run once before timing so one-time setup kernels are left out. With fp8 KV the cache is quantized per head and the kernel gets one mean scale each for K and V.

## Caveats

- A real batch mixes short and long requests. The uniform-length batch measured here can differ from it, most for skewed batches.
- fa2_cudagraph runs the same FA2 kernels with FlashInfer's CUDA-graph plan, which splits the KV and adds a merge kernel, as vLLM does in pure decode. The kernels are launched directly; no captured graph is replayed.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_qo_heads` | number | heads | Query and output heads on this GPU. |
| `num_kv_heads` | number | heads | KV heads on this GPU. Fewer than num_qo_heads means GQA. |
| `head_dim` | number | elements | Dimension of each query, key and value head. |
| `q_dtype` | dtype |  | Query element type. |
| `kv_dtype` | dtype |  | KV cache element type. |
| `o_dtype` | dtype |  | Output element type. |
| `batch_size` | number | requests | Requests decoded together. |
| `total_tokens` | number | tokens | KV tokens summed over the batch. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `fa2`: FlashInfer BatchDecodeWithPagedKVCacheWrapper with FA2 kernels and the ordinary plan. dtypes `bf16`, KV dtypes `bf16`, `fp8_e4m3`, any CUDA GPU. Source: https://docs.flashinfer.ai/api/decode.html
- `fa2_cudagraph`: FA2 with FlashInfer's CUDA-graph plan: a split-KV main kernel plus a merge kernel, as in vLLM pure decode. dtypes `bf16`, KV dtypes `bf16`, `fp8_e4m3`, any CUDA GPU. Source: https://docs.flashinfer.ai/api/decode.html
- `fa3`: FlashInfer BatchDecodeWithPagedKVCacheWrapper with FA3 kernels. dtypes `bf16`, `fp8_e4m3`, KV dtypes `bf16`, `fp8_e4m3`, any CUDA GPU. Source: https://docs.flashinfer.ai/api/decode.html
- `cudnn`: PyTorch scaled_dot_product_attention pinned to the cuDNN backend, over contiguous (unpaged) K and V. No fp8. dtypes `bf16`, KV dtypes `bf16`, any CUDA GPU. Source: https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html

## Measured rows

3,888 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA H200 | `fa2` | `bf16` | 2,106 |
| NVIDIA H200 | `fa2_cudagraph` | `bf16` | 486 |
| NVIDIA H200 | `fa3` | `bf16` | 1,296 |

## Used by these deployments

- [Llama 3.1 8B: Single GPU](https://servingstudio.cs.washington.edu/models/Llama-3.1-8B/llama3_dense.md) (`Llama-3.1-8B/llama3_dense`)
- [Llama 3.1 8B: TP](https://servingstudio.cs.washington.edu/models/Llama-3.1-8B/llama3_dense_tp.md) (`Llama-3.1-8B/llama3_dense_tp`)
- [Llama 3.1 8B: DP attention, TP FFN](https://servingstudio.cs.washington.edu/models/Llama-3.1-8B/llama3_dp_attn_tp_ffn.md) (`Llama-3.1-8B/llama3_dp_attn_tp_ffn`)
- [Qwen3 235B-A22B FP8: Attention side of an attention–FFN split](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_attn_tp.md) (`Qwen3-235B-A22B-FP8/qwen3_attn_tp`)
- [Qwen3 235B-A22B FP8: DP attention, EP MoE, DeepGEMM experts](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_moe_fp8_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-FP8/qwen3_moe_fp8_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B FP8: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_vllm_moe_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-FP8/qwen3_vllm_moe_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B Thinking 2507 FP8: DP attention, EP MoE, DeepGEMM experts](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_moe_fp8_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_moe_fp8_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B Thinking 2507 FP8: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_vllm_moe_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_vllm_moe_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B: Attention side of an attention–FFN split](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B/qwen3_attn_tp.md) (`Qwen3-235B-A22B/qwen3_attn_tp`)
- [Qwen3 235B-A22B: DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B/qwen3_moe_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B/qwen3_moe_dp_attn_ep_ffn`)
- [Qwen3 Coder 480B-A35B: Attention side of an attention–FFN split](https://servingstudio.cs.washington.edu/models/Qwen3-Coder-480B-A35B-Instruct/qwen3_attn_tp.md) (`Qwen3-Coder-480B-A35B-Instruct/qwen3_attn_tp`)
- [Qwen3.6 35B-A3B FP8: Single GPU](https://servingstudio.cs.washington.edu/models/Qwen3.6-35B-A3B-FP8/qwen36_local.md) (`Qwen3.6-35B-A3B-FP8/qwen36_local`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/flashinfer_attn_decode
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/flashinfer_attn_decode/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/flashinfer_attn_decode/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/flashinfer_attn_decode.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
