# FlashAttention-4 varlen attention (`fa4_varlen_attention`) | ServingStudio Kernel Library

Paged causal or sliding-window GQA attention with optional attention sinks and a value head dim that may differ from the query/key head dim, over one mixed decode and prefill batch.

One flash_attn_varlen_func(fa_version=4) call covers the whole varlen batch of a scheduler step over a paged cache whose pages pack each token's key and value channels side by side. FlashAttention 4 chooses how many ways to split each request's KV range from the batch's longest query and context: a split call adds a scheduler launch that lowers the split per request and a combine launch that merges the partial outputs. A decode-only step small enough for a FULL CUDA graph keeps the split choice made at capture over the padded batch and the model's longest context.

Category: Attention; group: MHA / GQA. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=fa4_varlen_attention. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `O = softmax([Q · Kᵀ · scale, sink]) · [V; 0]`
- `n_i = keys query i sees: i + 1 (full causal) or min(i + 1, window)`
- `TFLOPS = 2 · num_qo_heads · (head_dim_qk + head_dim_v) · Σ n_i / time`
- `GB/s = (Q + O + each request's visible K and V once) bytes / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. Every launch of the call is counted as the busy union of their intervals: the split scheduler, the forward kernel and the combine, whichever the batch issues. A FULL-graph decode step runs with the capture-time padded batch and max_seqlen_k = max_model_len, as vLLM replays it. Before timing, the output is checked against a PyTorch reference on sampled query rows.

## Caveats

- Queries, keys, values and sinks are random.
- The KV-cache insert belongs to its own kind, not to this call.
- KV pages are shuffled across a pool sized to the batch; the page table is max_model_len / page_size wide, as vLLM's, and pages wholly before a sliding window point at the null page, as vLLM's sliding-window manager leaves them.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_qo_heads` | number | heads | Query and output heads on this GPU. |
| `num_kv_heads` | number | heads | KV heads on this GPU. |
| `head_dim_qk` | number | elements | Dimension of each query and key head. |
| `head_dim_v` | number | elements | Dimension of each value and output head. |
| `window` | number | tokens | Sliding window including the query token itself; 0 means full causal. |
| `sinks` | label |  | Per-head attention-sink logits join every softmax denominator. |
| `page_size` | number | tokens | Tokens per KV-cache page. |
| `kv_layout` | label |  | KV-cache page layout; lbnhc packs the K then V channels of one (token, head). |
| `q_dtype` | dtype |  | Query, sink and output element type. |
| `kv_dtype` | dtype |  | Key and value cache element type. |
| `max_model_len` | number | tokens | Model context limit; sizes the page table and a graph step's split choice. |
| `cuda_graph` | label |  | Decode-only steps replay a FULL CUDA graph; false runs every step eagerly. |
| `query_context_pairs` | list |  | (query tokens, context tokens) per request in the one varlen call; the context includes the query. |
| `padded_requests` | number | requests | Capture size a FULL-graph decode step is padded to (0 = unpadded); ignored for a step that runs eagerly. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_fa4`: vLLM's vllm_flash_attn flash_attn_varlen_func(fa_version=4, num_splits=0) as FlashAttentionDiffKVImpl.forward calls it. dtypes `bf16`, KV dtypes `bf16`, SM targets `sm_100f`, GPUs B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip.

## Measured rows

1,136 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `vllm_fa4` | `bf16` | 1,136 |

## Used by these deployments

- [MiMo V2.6 Flash RL: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/MiMo-V2.6-Flash-RL/mimo_v2_vllm_fp8_swa_moe.md) (`MiMo-V2.6-Flash-RL/mimo_v2_vllm_fp8_swa_moe`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/fa4_varlen_attention
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/fa4_varlen_attention/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/fa4_varlen_attention/configs

## Torch reference

What the kind computes, as Torch code (`profiling/runners/attention/fa4_varlen_attention_reference.py` in ServingStudio Sim):

```python
"""PyTorch reference for FA4 varlen attention with sinks and a separate V dim.

Independent of FlashAttention: it reads keys and values through the page table
and applies the definition directly. For a query at absolute position ``p`` in
its request and query head ``h`` (KV head ``h // G``):

    s[h, j] = (q[h] . k[j, h // G]) * softmax_scale       for visible keys j
    visible: j <= p, and p - j < window when the layer is windowed
    P = softmax([s[h, visible], sink[h]])                 (sink only if present)
    o[h] = sum_j P[j] * v[j, h // G]                      (the sink adds no value)

The sink is a per-head logit that joins the denominator unscaled, as FA4's
``s_aux`` / ``learnable_sink`` does. Everything runs in float32 on whatever
device the inputs live on.
"""

from __future__ import annotations

from typing import Any


def gather_request_kv(key_cache: Any, value_cache: Any, block_row: Any, context: int) -> tuple:
    """Keys ``[context, kv_heads, head_dim_qk]`` and values ``[context, kv_heads, head_dim_v]``.

    ``key_cache`` / ``value_cache`` are ``[num_pages, page_size, kv_heads, dim]``
    views (any strides); ``block_row`` maps a request page index to a physical page.
    """
    import torch

    page_size = key_cache.shape[1]
    tokens = torch.arange(context, device=key_cache.device)
    pages = block_row.to(key_cache.device).long()[tokens // page_size]
    offsets = tokens % page_size
    return key_cache[pages, offsets].float(), value_cache[pages, offsets].float()


def varlen_attention_reference(
    q: Any,
    positions: Any,
    keys: Any,
    values: Any,
    *,
    softmax_scale: float,
    window: int,
    sinks: Any = None,
) -> Any:
    """Attention output ``[rows, heads, head_dim_v]`` (float32) for one request.

    ``q`` is ``[rows, heads, head_dim_qk]`` at absolute ``positions`` ``[rows]``;
    ``keys`` / ``values`` hold the request's first ``context`` tokens;
    ``window`` is the number of keys a query sees including itself (0 = all
    earlier keys); ``sinks`` is ``[heads]`` or None.
    """
    import torch

    rows, heads, _ = q.shape
    group = heads // keys.shape[1]
    k = keys.float().repeat_interleave(group, dim=1)  # [S, H, Dqk]
    v = values.float().repeat_interleave(group, dim=1)  # [S, H, Dv]
    scores = torch.einsum("thd,shd->hts", q.float(), k) * softmax_scale
    distance = positions.long()[:, None] - torch.arange(keys.shape[0], device=q.device)[None, :]
    visible = distance >= 0
    if window > 0:
        visible &= distance < window
    scores = scores.masked_fill(~visible[None], float("-inf"))
    if sinks is not None:
        sink = sinks.float().to(q.device)[:, None, None].expand(heads, rows, 1)
        scores = torch.cat([scores, sink], dim=-1)
    probs = torch.softmax(scores, dim=-1)[..., : keys.shape[0]]
    return torch.einsum("hts,shd->thd", probs, v)
```

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/fa4_varlen_attention.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
