# Paged batch decode attention (`paged_attn_batch_decode`) | ServingStudio Kernel Library

One-token-per-request GQA attention over a paged KV cache for a whole decode batch, keyed by the batch's longest context and total context.

One call reads every decode request's cached keys and values through its page table. The kernel splits each request's KV range over several CTAs while the batch leaves SMs free and merges the splits itself, so the longest request bounds the step when the batch is small, and the total KV bytes when it is large. The timed batch is one request at max_kv_len and the rest sharing the remaining context evenly.

Category: Attention; group: MHA / GQA. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=paged_attn_batch_decode. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `O[b, h] = softmax(Q[b, h] · K[b, h // G]ᵀ · scale) · V[b, h // G]`
- `TFLOPS = 4 · num_qo_heads · head_dim · total_kv_len / time`
- `GB/s = (Q + O + every request's K and V once) bytes / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. Both launches of the call count: FlashInfer's counter memset and the decode kernel. The planning bound max_seq_len is the larger of max_kv_len and 512 tokens per SM, which saturates the split choice as vLLM's FULL CUDA graph captured at max_model_len does. Before timing, the output is checked against a PyTorch reference on sampled requests.

## Caveats

- Queries, keys and values are random; the quantization scales are 1.
- Pages hold 32 tokens: 16, 32 and 64 measured within 1%, so the page size is not part of the key.
- An eager mixed step passes the whole batch's longest context as the bound; below the saturating bound that is up to about 3 us faster for batches of at most 4 requests with contexts of at most 4096 tokens.
- A batch whose context is spread unlike the canonical one can differ by about 10% at the same key.
- CUDA-graph padding rows hold no context; zero-length rows measured within 10% of one-token rows.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_qo_heads` | number | heads | Query and output heads on this GPU. |
| `num_kv_heads` | number | heads | KV heads on this GPU. |
| `head_dim` | number | elements | Dimension of each query, key and value head. |
| `q_dtype` | dtype |  | Query element type. |
| `kv_dtype` | dtype |  | Key and value cache element type. |
| `o_dtype` | dtype |  | Output element type. |
| `batch_size` | number | requests | Rows in the call; a CUDA-graph replay counts its zero-length padding rows. |
| `max_kv_len` | number | tokens | Context of the longest request, its query included. |
| `total_kv_len` | number | tokens | Context tokens summed over the batch. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `flashinfer_trtllm_gen`: FlashInfer's trtllm_batch_decode_with_kv_cache(backend='trtllm-gen') as vLLM's FlashInferImpl.forward calls it for the decode requests. dtypes `fp8_e4m3`, KV dtypes `fp8_e4m3`, SM targets `sm_100f`, GPUs B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip.

## Measured rows

701 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `flashinfer_trtllm_gen` | `fp8_e4m3` | 701 |

## Used by these deployments

- [Qwen3.8 27B NVFP4: vLLM, one GPU](https://servingstudio.cs.washington.edu/models/Qwen3.8-27B-NVFP4/qwen35_vllm_nvfp4_gdn_dense.md) (`Qwen3.8-27B-NVFP4/qwen35_vllm_nvfp4_gdn_dense`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/paged_attn_batch_decode
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/paged_attn_batch_decode/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/paged_attn_batch_decode/configs

## Torch reference

What the kind computes, as Torch code (`profiling/runners/attention/fa4_varlen_attention_reference.py` in ServingStudio Sim):

```python
"""PyTorch reference for FA4 varlen attention with sinks and a separate V dim.

Independent of FlashAttention: it reads keys and values through the page table
and applies the definition directly. For a query at absolute position ``p`` in
its request and query head ``h`` (KV head ``h // G``):

    s[h, j] = (q[h] . k[j, h // G]) * softmax_scale       for visible keys j
    visible: j <= p, and p - j < window when the layer is windowed
    P = softmax([s[h, visible], sink[h]])                 (sink only if present)
    o[h] = sum_j P[j] * v[j, h // G]                      (the sink adds no value)

The sink is a per-head logit that joins the denominator unscaled, as FA4's
``s_aux`` / ``learnable_sink`` does. Everything runs in float32 on whatever
device the inputs live on.
"""

from __future__ import annotations

from typing import Any


def gather_request_kv(key_cache: Any, value_cache: Any, block_row: Any, context: int) -> tuple:
    """Keys ``[context, kv_heads, head_dim_qk]`` and values ``[context, kv_heads, head_dim_v]``.

    ``key_cache`` / ``value_cache`` are ``[num_pages, page_size, kv_heads, dim]``
    views (any strides); ``block_row`` maps a request page index to a physical page.
    """
    import torch

    page_size = key_cache.shape[1]
    tokens = torch.arange(context, device=key_cache.device)
    pages = block_row.to(key_cache.device).long()[tokens // page_size]
    offsets = tokens % page_size
    return key_cache[pages, offsets].float(), value_cache[pages, offsets].float()


def varlen_attention_reference(
    q: Any,
    positions: Any,
    keys: Any,
    values: Any,
    *,
    softmax_scale: float,
    window: int,
    sinks: Any = None,
) -> Any:
    """Attention output ``[rows, heads, head_dim_v]`` (float32) for one request.

    ``q`` is ``[rows, heads, head_dim_qk]`` at absolute ``positions`` ``[rows]``;
    ``keys`` / ``values`` hold the request's first ``context`` tokens;
    ``window`` is the number of keys a query sees including itself (0 = all
    earlier keys); ``sinks`` is ``[heads]`` or None.
    """
    import torch

    rows, heads, _ = q.shape
    group = heads // keys.shape[1]
    k = keys.float().repeat_interleave(group, dim=1)  # [S, H, Dqk]
    v = values.float().repeat_interleave(group, dim=1)  # [S, H, Dv]
    scores = torch.einsum("thd,shd->hts", q.float(), k) * softmax_scale
    distance = positions.long()[:, None] - torch.arange(keys.shape[0], device=q.device)[None, :]
    visible = distance >= 0
    if window > 0:
        visible &= distance < window
    scores = scores.masked_fill(~visible[None], float("-inf"))
    if sinks is not None:
        sink = sinks.float().to(q.device)[:, None, None].expand(heads, rows, 1)
        scores = torch.cat([scores, sink], dim=-1)
    probs = torch.softmax(scores, dim=-1)[..., : keys.shape[0]]
    return torch.einsum("hts,shd->thd", probs, v)
```

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/paged_attn_batch_decode.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
