# Sparse MLA attention (`dsa_sparse_mla_attention`) | ServingStudio Kernel Library

Attend to the DSA-selected cache tokens for each query head.

In sparse MLA attention, each query head attends only to the cache tokens the DSA indexer selected. Scores use the compressed latent and the rotary part, if any; values are the latent. valid_counts sets how many of the selected_k slots each query uses, and the rest are masked. Queries and cache values are synthetic, and the selected indices follow index_distribution.

Category: Attention; group: DSA. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=dsa_sparse_mla_attention. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `O = softmax(softmax_scale · q · K_selectedᵀ) · V_selected`
- `D = latent_dim + rope_dim; Q = num_queries; H = num_heads; K = selected_k`
- `TFLOPS = 2·H·Σvalid_counts·(D + value_dim) / time`
- `logical bytes = bq·Q·H·D + 4·Q·K + bc·Σvalid_counts·D + 2·Q·H·value_dim + 8·Q·H`
- `GB/s = logical bytes / time; bq = bc = 2 for BF16, 1 for FP8`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. torch counts every launch of its composite; flashinfer_trtllm_fp8 counts only fmhaSm100, and vllm_flashmla_bf16 only sparse_attn_fwd_kernel. Setup and output checks run before timing.

## Caveats

- Selected positions and tensor values are constructed, not taken from an indexer or a serving KV cache.
- TFLOPS counts only valid slots; masked selected_k slots take time but do no attention work.
- torch and vllm_flashmla_bf16 accept only rope_dim = 64; flashinfer_trtllm_fp8 also accepts rope_dim = 0, with its own cache layout. vllm_flashmla_bf16 needs selected_k to be a multiple of FlashMLA's top-k tile: 128 on SM90, and 64 at 64 heads or 128 otherwise on SM10x.
- GB/s includes 8 bytes per query head for the max-logit and log-sum-exp outputs, even for backends that return only the attention output.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_queries` | number | tokens | Query tokens in the attention batch. |
| `num_cache_tokens` | number | tokens | Cache tokens available to the selected indices. |
| `num_heads` | number | heads | Query heads on this GPU. |
| `num_kv_heads` | number | heads | KV heads shared by the query heads. |
| `selected_k` | number | tokens | Selected index slots per query. |
| `latent_dim` | number | elements | Compressed KV latent width. |
| `rope_dim` | number | elements | Rotary-position width appended to the latent; 0 when there is none. |
| `value_dim` | number | elements | Value and output width per head. |
| `softmax_scale` | number | multiplier | Multiplier applied to attention scores. |
| `q_dtype` | dtype |  | Query element type. |
| `cache_dtype` | dtype |  | Compressed KV cache element type. |
| `index_dtype` | label |  | Element type of selected indices. |
| `output_dtype` | dtype |  | Attention output element type. |
| `valid_counts` | label |  | Encoded valid selected-index count per query. |
| `index_distribution` | label |  | Pattern used to construct selected indices. |
| `cache_layout` | label |  | Layout of compressed KV values in the cache. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `torch`: A PyTorch gather, score, softmax and value-reduction composite on BF16 inputs. dtypes `bf16`, KV dtypes `bf16`, any CUDA GPU.
- `flashinfer_trtllm_fp8`: FlashInfer trtllm_batch_decode_with_kv_cache_mla in sparse mode, FP8 query and paged cache, on B200. dtypes `fp8_e4m3`, KV dtypes `fp8_e4m3`, SM targets `sm_100f`, GPUs B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip. Source: https://github.com/flashinfer-ai/flashinfer/blob/main/flashinfer/mla/_core.py
- `vllm_flashmla_bf16`: vLLM's flash_mla_sparse_fwd on BF16 selected KV tokens. dtypes `bf16`, KV dtypes `bf16`, SM targets `sm_100f`, `sm_90a`, GPUs H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip, H800-SXM5-80GB, H20-SXM-96GB. Source: https://github.com/vllm-project/vllm/blob/main/vllm/v1/attention/ops/flashmla.py

## Measured rows

21,272 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `flashinfer_trtllm_fp8` | `fp8_e4m3` | 13,365 |
| NVIDIA H200 | `vllm_flashmla_bf16` | `bf16` | 7,907 |

## Used by these deployments

- [GLM-5.2 FP8: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-FP8/glm52_vllm_dsa_moe.md) (`GLM-5.2-FP8/glm52_vllm_dsa_moe`)
- [GLM-5.2 NVFP4: SGLang, TP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, TP and EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, TP and EP, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative`)
- [GLM-5.2 NVFP4: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_pp_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_pp_dsa_moe`)
- [GLM-5.2: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2/glm52_vllm_dsa_moe.md) (`GLM-5.2/glm52_vllm_dsa_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe`)
- [GLM-5.3-Flash: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe`)
- [GLM-5.3-Flash: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe`)
- [GLM-5.3-Flash: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe`)
- [GLM-5.3 NVFP4: vLLM, TP and EP, DFlash speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2.md) (`GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/dsa_sparse_mla_attention
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/dsa_sparse_mla_attention/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/dsa_sparse_mla_attention/configs

## Torch reference

What the kind computes, as Torch code (`profiling/runners/attention/dsa_sparse_mla_attention_reference.py` in ServingStudio Sim):

```python
"""Torch semantics for GLM-5.2 BF16 selected sparse MLA attention."""

from __future__ import annotations

import math

import torch

__all__ = ["dsa_sparse_mla_attention_reference"]

_LATENT_DIM = 512
_ROPE_DIM = 64
_SCORE_DIM = _LATENT_DIM + _ROPE_DIM
_VALUE_DIM = _LATENT_DIM
_QUERY_CHUNK_SIZE = 8


def dsa_sparse_mla_attention_reference(
    q: torch.Tensor,
    cache: torch.Tensor,
    selected_indices: torch.Tensor,
    *,
    softmax_scale: float,
) -> torch.Tensor:
    """Return BF16 selected MLA attention in a fresh ``[Q, H, 512]`` tensor.

    The score and value paths use FP32: QK consumes all 576 dimensions (the
    512-dimensional latent part followed by 64 RoPE dimensions), while PV uses
    the first 512 cache dimensions as values. Every negative index and every
    index outside the cache is masked. Duplicate valid indices deliberately
    retain repeated softmax mass, and rows with no valid index return zeros.

    This is the mathematical/valid-output boundary derived from installed
    Transformers 5.13 ``glm_moe_dsa`` attention and vLLM v0.23.0 commit
    0fc695fc6d1d82e9a5ac6835ac8e4e1c83703665 with FlashMLA commit
    a6ec2ba7bd0a7dff98b3f4d3e6b52b159c48d78b. A later backend will construct
    production operands and time ``flash_mla_sparse_fwd``; this reference never
    imports or calls vLLM, FlashMLA, or custom CUDA.

    Query chunking bounds the largest gathered tensor by
    ``_QUERY_CHUNK_SIZE * K * 576`` and does not change the row-independent
    result. Invalid entries are zeroed before QK and PV, so an invalid
    placeholder can never expose NaN/Inf from an unrelated cache row.
    """
    _validate(q, cache, selected_indices, softmax_scale=softmax_scale)

    output = torch.empty(
        (q.shape[0], q.shape[1], _VALUE_DIM),
        dtype=torch.bfloat16,
        device=q.device,
    )
    cache_rows = cache[:, 0, :]

    for query_start in range(0, q.shape[0], _QUERY_CHUNK_SIZE):
        query_end = min(query_start + _QUERY_CHUNK_SIZE, q.shape[0])
        query = q[query_start:query_end].float()
        indices = selected_indices[query_start:query_end, 0, :]
        valid = (indices >= 0) & (indices < cache.shape[0])

        safe_indices = indices.masked_fill(~valid, 0).to(torch.int64)
        gathered = cache_rows.index_select(0, safe_indices.reshape(-1)).reshape(
            query_end - query_start,
            selected_indices.shape[2],
            _SCORE_DIM,
        )
        gathered = gathered.float()
        gathered.masked_fill_(~valid.unsqueeze(-1), 0.0)

        scores = torch.einsum("qhd,qkd->qhk", query, gathered)
        scores.mul_(softmax_scale)
        scores.masked_fill_(~valid.unsqueeze(1), float("-inf"))

        # Softmax over all -inf is NaN. Temporarily make those rows finite,
        # then explicitly zero every invalid probability after softmax.
        has_valid = valid.any(dim=-1)
        scores.masked_fill_(~has_valid[:, None, None], 0.0)
        probabilities = torch.softmax(scores, dim=-1, dtype=torch.float32)
        probabilities.masked_fill_(~valid.unsqueeze(1), 0.0)

        chunk_output = torch.einsum(
            "qhk,qkv->qhv",
            probabilities,
            gathered[..., :_VALUE_DIM],
        )
        output[query_start:query_end].copy_(chunk_output.to(torch.bfloat16))

    return output


def _validate(
    q: object,
    cache: object,
    selected_indices: object,
    *,
    softmax_scale: object,
) -> None:
    for name, tensor in (
        ("q", q),
        ("cache", cache),
        ("selected_indices", selected_indices),
    ):
        if not isinstance(tensor, torch.Tensor):
            raise TypeError(f"{name} must be a torch.Tensor")

    assert isinstance(q, torch.Tensor)
    assert isinstance(cache, torch.Tensor)
    assert isinstance(selected_indices, torch.Tensor)

    if type(softmax_scale) is not float:
        raise TypeError("softmax_scale must be a Python float")
    if not math.isfinite(softmax_scale):
        raise ValueError("softmax_scale must be finite")
    if softmax_scale <= 0:
        raise ValueError("softmax_scale must be positive")

    _validate_ranks_and_shapes(q, cache, selected_indices)
    _validate_dtypes_and_devices(q, cache, selected_indices)
    _validate_layouts(q, cache, selected_indices)
    _validate_finite_domain(q, cache, selected_indices)


def _validate_ranks_and_shapes(
    q: torch.Tensor,
    cache: torch.Tensor,
    selected_indices: torch.Tensor,
) -> None:
    for name, tensor in (
        ("q", q),
        ("cache", cache),
        ("selected_indices", selected_indices),
    ):
        if tensor.ndim != 3:
            raise ValueError(f"{name} must be rank 3, got rank {tensor.ndim}")

    if q.shape[0] <= 0:
        raise ValueError("q must contain at least one query")
    if q.shape[1] <= 0:
        raise ValueError("q must contain at least one head")
    if q.shape[2] != _SCORE_DIM:
        raise ValueError(f"q score width must be {_SCORE_DIM}, got {q.shape[2]}")

    if cache.shape[0] <= 0:
        raise ValueError("cache must contain at least one token")
    if cache.shape[1] != 1:
        raise ValueError(f"cache must have exactly one MQA head, got {cache.shape[1]}")
    if cache.shape[2] != _SCORE_DIM:
        raise ValueError(f"cache score width must be {_SCORE_DIM}, got {cache.shape[2]}")

    if selected_indices.shape[0] != q.shape[0]:
        raise ValueError(
            "selected_indices query dimension must match q, "
            f"got {selected_indices.shape[0]} and {q.shape[0]}"
        )
    if selected_indices.shape[1] != 1:
        raise ValueError(
            f"selected_indices must have exactly one MQA head, got {selected_indices.shape[1]}"
        )
    if selected_indices.shape[2] <= 0:
        raise ValueError("selected_indices K dimension must be positive")


def _validate_dtypes_and_devices(
    q: torch.Tensor,
    cache: torch.Tensor,
    selected_indices: torch.Tensor,
) -> None:
    if q.dtype is not torch.bfloat16:
        raise TypeError(f"q dtype must be torch.bfloat16, got {q.dtype}")
    if cache.dtype is not torch.bfloat16:
        raise TypeError(f"cache dtype must be torch.bfloat16, got {cache.dtype}")
    if selected_indices.dtype is not torch.int32:
        raise TypeError(f"selected_indices dtype must be torch.int32, got {selected_indices.dtype}")

    tensors = (q, cache, selected_indices)
    if any(tensor.device.type == "meta" for tensor in tensors):
        raise ValueError("meta tensors are not supported")
    for name, tensor in (("cache", cache), ("selected_indices", selected_indices)):
        if tensor.device != q.device:
            raise ValueError(
                f"{name} must be on the same device as q, got {tensor.device} and {q.device}"
            )


def _validate_layouts(
    q: torch.Tensor,
    cache: torch.Tensor,
    selected_indices: torch.Tensor,
) -> None:
    for name, tensor in (
        ("q", q),
        ("cache", cache),
        ("selected_indices", selected_indices),
    ):
        if tensor.layout is not torch.strided:
            raise ValueError(f"{name} must have torch.strided layout")
        # Status 1 is definite overlap. Status 2 is indeterminate and remains
        # accepted for padded outer-stride views, matching nearby references.
        if int(torch._debug_has_internal_overlap(tensor)) == 1:
            raise ValueError(f"{name} must not have internal overlap")
        if tensor.stride(-1) != 1:
            raise ValueError(f"{name} innermost stride must be 1")


def _validate_finite_domain(
    q: torch.Tensor,
    cache: torch.Tensor,
    selected_indices: torch.Tensor,
) -> None:
    if not bool(torch.isfinite(q).all().item()):
        raise ValueError("q must contain only finite values")

    valid = (selected_indices >= 0) & (selected_indices < cache.shape[0])
    if not bool(valid.any().item()):
        return
    selected_rows = torch.unique(selected_indices[valid].to(torch.int64))
    selected_cache = cache.index_select(0, selected_rows)
    if not bool(torch.isfinite(selected_cache).all().item()):
        raise ValueError("valid selected cache rows must contain only finite values")
```

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/dsa_sparse_mla_attention.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
