# Prefill indexer logits (`dsa_mqa_logits_prefill`) | ServingStudio Kernel Library

Score each prefill query against causal index keys and reduce across indexer heads.

In prefill, the DSA indexer scores every visible key for each query: per head, the ReLU of the query-key dot product, weighted by head and summed, then multiplied by the key's scale. The measurement uses one sequence whose last num_queries tokens are the queries, so query i sees keys 0 through num_keys − num_queries + i. Inputs are deterministic FP8 values with FP32 scales.

Category: Attention; group: DSA. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=dsa_mqa_logits_prefill. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `logit[q, k] = scale[k] · Σₕ weight[q, h] · ReLU(query[q, h] · key[k])`
- `W = Σ over two-query tiles ⌈(num_keys − num_queries + 1 + final query index) / 256⌉ · 256`
- `C = 2 · W`
- `TFLOPS = 2 · C · num_heads · head_dim / time`
- `B = num_queries · num_heads · head_dim + num_keys · head_dim + 4 · num_keys + 4 · num_queries · num_heads + 8 · num_queries + 4 · C`
- `GB/s = B / time`

## How it is measured

torch runs the math in FP32 as separate launches and is timed with CUDA events: five warm-up calls, then a loop of back-to-back calls, taking the median of three runs. deepgemm_fp8 runs once for setup, then CUPTI counts only the MQA-logits kernel launches, with the L2 cache flushed before each.

## Caveats

- Only one sequence with clean_logits = false is measured; the torch backend marks invalid positions with NaN.
- Both backends' TFLOPS and the logits term of GB/s use DeepGEMM's schedule of two-query, 256-key tiles, not the torch launches.
- GB/s counts each key once. Every two-query tile reads its key window again, but the one sequence's FP8 keys stay in L2 across tiles, so those re-reads are not HBM traffic.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_queries` | number | tokens | Query tokens in the prefill span. |
| `num_keys` | number | tokens | Cached index keys available to the final query. |
| `num_sequences` | number | requests | Sequences represented by the span. |
| `num_heads` | number | heads | Indexer query heads scored against each key. |
| `head_dim` | number | elements | Elements in each query and key head. |
| `q_dtype` | dtype |  | Element type of indexer queries. |
| `k_dtype` | dtype |  | Element type of cached index keys. |
| `k_scale_dtype` | dtype |  | Element type of per-key quantization scales. |
| `weight_dtype` | dtype |  | Element type of per-query, per-head weights. |
| `output_dtype` | dtype |  | Element type of the score matrix. |
| `span_mode` | label |  | Rule defining the valid key range of each query. |
| `clean_logits` | label |  | Whether invalid score positions are initialized. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `torch`: PyTorch FP32 dot products, ReLU, head reduction and key scaling as separate launches. dtypes `fp8_e4m3`, KV dtypes `fp8_e4m3`, any CUDA GPU.
- `deepgemm_fp8`: DeepGEMM fp8_mqa_logits through vLLM's deep_gemm wrapper: FP8 scoring and head reduction in one kernel. dtypes `fp8_e4m3`, KV dtypes `fp8_e4m3`, SM targets `sm_100f`, `sm_120f`, `sm_90a`, GPUs H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip, H800-SXM5-80GB, H20-SXM-96GB.

## Measured rows

2,057 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `deepgemm_fp8` | `fp8_e4m3` | 810 |
| NVIDIA B200 | `vllm_deepgemm_fp8` | `fp8_e4m3` | 335 |
| NVIDIA H200 | `deepgemm_fp8` | `fp8_e4m3` | 530 |
| NVIDIA H200 | `vllm_deepgemm_fp8` | `fp8_e4m3` | 382 |

## Used by these deployments

- [DeepSeek V4.1 Flash 1017: vLLM, DP attention, EP MoE, DSpark speculative decoding](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dp_attn_ep_moe_dspark.md) (`DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dp_attn_ep_moe_dspark`)
- [DeepSeek V4.1 Flash 1017: vLLM, TP attention, EP MoE, DSpark speculative decoding](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dspark.md) (`DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dspark`)
- [DeepSeek V4.1 Flash: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash/deepseek_v41_vllm.md) (`DeepSeek-V4.1-Flash/deepseek_v41_vllm`)
- [DeepSeek V4.1 Flash: vLLM, TP attention, EP MoE, side streams serialized](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash/deepseek_v41_vllm_serial_streams.md) (`DeepSeek-V4.1-Flash/deepseek_v41_vllm_serial_streams`)
- [GLM-5.2 FP8: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-FP8/glm52_vllm_dsa_moe.md) (`GLM-5.2-FP8/glm52_vllm_dsa_moe`)
- [GLM-5.2 NVFP4: SGLang, TP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, TP and EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, TP and EP, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative`)
- [GLM-5.2 NVFP4: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_pp_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_pp_dsa_moe`)
- [GLM-5.2: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2/glm52_vllm_dsa_moe.md) (`GLM-5.2/glm52_vllm_dsa_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe`)
- [GLM-5.3-Flash: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe`)
- [GLM-5.3-Flash: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe`)
- [GLM-5.3-Flash: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe`)
- [GLM-5.3 NVFP4: vLLM, TP and EP, DFlash speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2.md) (`GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/dsa_mqa_logits_prefill
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/dsa_mqa_logits_prefill/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/dsa_mqa_logits_prefill/configs

## Torch reference

What the kind computes, as Torch code (`profiling/runners/attention/dsa_mqa_logits_prefill_reference.py` in ServingStudio Sim):

```python
"""Torch semantics for the DSA prefill MQA-logits operation."""

from __future__ import annotations

import torch

_SUPPORTED_QK_DTYPES = (
    torch.float8_e4m3fn,
    torch.bfloat16,
    torch.float16,
    torch.float32,
)
_SPAN_DTYPES = (torch.int32, torch.int64)


def dsa_mqa_logits_prefill_reference(
    q: torch.Tensor,
    k: torch.Tensor,
    k_scale: torch.Tensor,
    weights: torch.Tensor,
    k_start: torch.Tensor,
    k_end: torch.Tensor,
    *,
    clean_logits: bool = False,
) -> torch.Tensor:
    """Return FP32 DSA logits for the valid key span of every query.

    Dot products, ReLU, head weighting/reduction, and K scaling are evaluated
    in FP32. Invalid output positions are ``-inf`` when ``clean_logits`` is
    true; otherwise they are NaN sentinels because production does not define
    their contents.

    This models the math and valid-output contract of vLLM v0.23.0 commit
    0fc695fc6d1d82e9a5ac6835ac8e4e1c83703665 and DeepGEMM commit
    891d57b4db1071624b5c8fa0d1e51cb317fa709f,
    ``fp8_fp4_mqa_logits``. Later profiling backends construct the exact
    DeepGEMM layouts and time the production kernel.
    """
    _validate(q, k, k_scale, weights, k_start, k_end, clean_logits)

    q_fp32 = q.float()
    k_fp32 = k.float()
    dot_products = torch.einsum("mhd,nd->mhn", q_fp32, k_fp32)
    reduced = (torch.relu(dot_products) * weights.unsqueeze(-1)).sum(dim=1) * k_scale.unsqueeze(0)

    invalid_value = float("-inf") if clean_logits else float("nan")
    output = torch.full(
        (q.shape[0], k.shape[0]),
        invalid_value,
        dtype=torch.float32,
        device=q.device,
    )
    positions = torch.arange(k.shape[0], device=q.device)
    valid = (positions.unsqueeze(0) >= k_start.unsqueeze(1)) & (
        positions.unsqueeze(0) < k_end.unsqueeze(1)
    )
    output[valid] = reduced[valid]
    return output


def _validate(
    q: object,
    k: object,
    k_scale: object,
    weights: object,
    k_start: object,
    k_end: object,
    clean_logits: object,
) -> None:
    tensors = (
        ("q", q),
        ("k", k),
        ("k_scale", k_scale),
        ("weights", weights),
        ("k_start", k_start),
        ("k_end", k_end),
    )
    for name, tensor in tensors:
        if not isinstance(tensor, torch.Tensor):
            raise TypeError(f"{name} must be a torch.Tensor")
    if not isinstance(clean_logits, bool):
        raise TypeError("clean_logits must be a bool")

    assert isinstance(q, torch.Tensor)
    assert isinstance(k, torch.Tensor)
    assert isinstance(k_scale, torch.Tensor)
    assert isinstance(weights, torch.Tensor)
    assert isinstance(k_start, torch.Tensor)
    assert isinstance(k_end, torch.Tensor)

    expected_ranks = {
        "q": 3,
        "k": 2,
        "k_scale": 1,
        "weights": 2,
        "k_start": 1,
        "k_end": 1,
    }
    for name, tensor in tensors:
        expected_rank = expected_ranks[name]
        if tensor.ndim != expected_rank:
            raise ValueError(f"{name} must be rank {expected_rank}, got rank {tensor.ndim}")
        if any(dimension <= 0 for dimension in tensor.shape):
            raise ValueError(f"{name} dimensions must be positive, got {tuple(tensor.shape)}")

    m, num_heads, head_dim = q.shape
    num_keys, key_dim = k.shape
    if key_dim != head_dim:
        raise ValueError(f"q and k head dimensions must match, got {head_dim} and {key_dim}")
    if k_scale.shape != (num_keys,):
        raise ValueError(f"k_scale must have shape ({num_keys},), got {tuple(k_scale.shape)}")
    if weights.shape != (m, num_heads):
        raise ValueError(f"weights must have shape ({m}, {num_heads}), got {tuple(weights.shape)}")
    for name, spans in (("k_start", k_start), ("k_end", k_end)):
        if spans.shape != (m,):
            raise ValueError(f"{name} must have shape ({m},), got {tuple(spans.shape)}")

    if q.dtype not in _SUPPORTED_QK_DTYPES:
        raise TypeError(
            "q dtype must be torch.float8_e4m3fn, torch.bfloat16, "
            f"torch.float16, or torch.float32, got {q.dtype}"
        )
    if k.dtype != q.dtype:
        raise TypeError(f"k dtype must match q dtype {q.dtype}, got {k.dtype}")
    if k_scale.dtype is not torch.float32:
        raise TypeError(f"k_scale dtype must be torch.float32, got {k_scale.dtype}")
    if weights.dtype is not torch.float32:
        raise TypeError(f"weights dtype must be torch.float32, got {weights.dtype}")
    for name, spans in (("k_start", k_start), ("k_end", k_end)):
        if spans.dtype not in _SPAN_DTYPES:
            raise TypeError(f"{name} dtype must be torch.int32 or torch.int64")
        if spans.layout is not torch.strided or not spans.is_contiguous():
            raise ValueError(f"{name} must be contiguous")

    if q.device.type == "meta":
        raise ValueError("meta tensors are not supported")
    for name, tensor in tensors[1:]:
        if tensor.device != q.device:
            raise ValueError(
                f"{name} must be on the same device as q, got {tensor.device} and {q.device}"
            )

    for name, tensor in tensors[:4]:
        if tensor.layout is not torch.strided:
            raise ValueError(f"{name} must have torch.strided layout")
        if not bool(torch.isfinite(tensor.float()).all().item()):
            raise ValueError(f"{name} must contain only finite values")
    if not bool(torch.all(k_scale > 0).item()):
        raise ValueError("k_scale values must be strictly positive")

    if bool(torch.any(k_start < 0).item()):
        raise ValueError("k_start values must be nonnegative")
    if bool(torch.any(k_end > num_keys).item()):
        raise ValueError(f"k_end values must not exceed the key count {num_keys}")
    if bool(torch.any(k_start > k_end).item()):
        raise ValueError("each k_start value must be less than or equal to k_end")
```

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/dsa_mqa_logits_prefill.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
