# Sparse GQA paged attention (`sparse_gqa_paged_attention`) | ServingStudio Kernel Library

Attend each query row to its indexer-selected tokens in a paged K/V cache and apply the sigmoid output gate.

Each row selects min(⌊context_len/ratio⌋, token_topk/ratio) complete groups of ratio tokens, scattered at random over its visible context, plus the causal tail of its own open group. The call reads only those tokens' keys and values through the block table. It splits the selection into split-K partials when there are few rows, then merges them; the configuration is the production table for this GPU. Under the uniform (decode) table every row is its own request with pages drawn at random from a pool larger than L2; under the prefill table the rows are one prefill chunk, sharing one request's consecutive pages.

Category: Attention; group: Sparse GQA. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=sparse_gqa_paged_attention. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `S = min(⌊c/r⌋, k/r)·r + (c mod r): tokens attended per row`
- `out = BF16(softmax(q·Kₛᵀ/√d)·Vₛ) · sigmoid(gate)`
- `TFLOPS = 4 · num_rows · num_q_heads · head_dim · S / time`
- `GB/s = num_rows · S · num_kv_heads · 2 · head_dim · bytes(kv) / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. Every launch of the call counts: the partial kernel and, when it splits, the merge. The selection is built with the production expansion kernel and checked against a PyTorch reference before timing.

## Caveats

- The output zero-fill the framework issues before the call is not included.
- Selected groups are uniform random over the visible context, not a real indexer's choice; contiguous selections (context ≤ token_topk) read faster.
- Every row sits at position context_len − 1. Prefill-table rows share one request's consecutive pages, but a real chunk's rows see a causally growing prefix; uniform-table rows are separate requests on random pages.
- FP8 caches use per-tensor scales of 1.
- GB/s counts selected K/V bytes once; queries, gates, indices and the split-K workspace are excluded.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_rows` | number | tokens | Query rows attended in the call. |
| `context_len` | number | tokens | Tokens visible to every row, the row's own token included. |
| `num_q_heads` | number | heads | Query heads, each with a gate head. |
| `num_kv_heads` | number | heads | K/V heads, each shared by a query group. |
| `head_dim` | number | elements | Width of each query, key and value head. |
| `token_topk` | number | tokens | Tokens selected from complete groups per row. |
| `compress_ratio` | number | tokens | Tokens per compressed group. |
| `q_dtype` | dtype |  | Element type of the queries, gate and output. |
| `kv_dtype` | dtype |  | Element type of the paged K/V cache. |
| `prefill_config` | label |  | Whether the step holds a prefill row (max query length above the decode/verify length); it picks the launch configuration for large calls. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_triton`: The alignment vLLM fork's qsa_sparse_paged_attention: split-K Triton partials plus merge, gate fused. dtypes `bf16`, KV dtypes `bf16`, `fp8_e4m3`, compute capability 8.9 or newer, GPUs H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip, H800-SXM5-80GB, H20-SXM-96GB, L40, L40S, L4. Source: https://github.com/serendipity-zk/vllm/blob/892da0822f7e866cec8e519727f014c45a5ad792/vllm/models/qwen4_exp/nvidia/ops/qsa.py

## Measured rows

868 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `vllm_triton` | `bf16` | 868 |

## Used by these deployments

- [Qwen3.8 Flash Next NVFP4: vLLM, one GPU, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative.md) (`Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/sparse_gqa_paged_attention
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/sparse_gqa_paged_attention/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/sparse_gqa_paged_attention/configs

## Torch reference

What the kind computes, as Torch code (`profiling/runners/attention/sparse_gqa_paged_attention_reference.py` in ServingStudio Sim):

```python
"""Independent PyTorch reference for sparse GQA paged attention with a sigmoid gate."""

from __future__ import annotations

from typing import Any


def sparse_gqa_paged_attention_reference(
    torch: Any,
    q: Any,
    k_cache: Any,
    v_cache: Any,
    indices: Any,
    block_table: Any,
    token_to_req: Any,
    gate: Any,
    rows: list[int],
    *,
    output_scale: float = 1.0,
) -> Any:
    """Return gated BF16 outputs ``[len(rows), heads, dim]`` for the given rows.

    ``indices`` is the packed ``[rows, width + 1]`` buffer: logical token ids
    (``-1`` padding) followed by the row's valid count. ``k_cache``/``v_cache``
    are ``[pages, page_size, kv_heads, dim]``.
    """
    page_size = k_cache.shape[1]
    num_heads, head_dim = q.shape[1], q.shape[2]
    group = num_heads // k_cache.shape[2]
    width = indices.shape[1] - 1
    out = []
    for row in rows:
        count = int(indices[row, width])
        tokens = indices[row, :count].long()
        tokens = tokens[tokens >= 0]
        request = int(token_to_req[row])
        pages = block_table[request, tokens // page_size].long()
        offsets = tokens % page_size
        keys = k_cache[pages, offsets].float()  # [S, kv_heads, dim]
        values = v_cache[pages, offsets].float()
        keys = keys.repeat_interleave(group, dim=1)
        values = values.repeat_interleave(group, dim=1)
        query = q[row].float()  # [heads, dim]
        scores = torch.einsum("hd,shd->hs", query, keys) / head_dim**0.5
        probs = torch.softmax(scores, dim=-1)
        attended = torch.einsum("hs,shd->hd", probs, values) * output_scale
        attended = attended.to(torch.bfloat16).float()
        gated = attended * torch.sigmoid(gate.view(q.shape)[row].float())
        out.append(gated.to(torch.bfloat16))
    return torch.stack(out)


__all__ = ["sparse_gqa_paged_attention_reference"]
```

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/sparse_gqa_paged_attention.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
