# Decode indexer top-k (`dsa_persistent_topk_decode`) | ServingStudio Kernel Library

Select request-local token positions from indexer logits for each DSA decode query.

In decode, the DSA indexer keeps the top_k highest-scoring positions of each query row; sparse MLA attention then reads only those. Each request has next_n rows. With context_mode uniform, a request's rows have valid lengths context_len − next_n + 1 to context_len; with max_ragged, next_n is 1 and request b has max(0, context_len − b) valid logits; uniform_random has uniform lengths. A row no longer than top_k yields only its own positions. The logits are FP32 in rows padded to logits_row_stride; uniform rows all hold the same increasing sequence, uniform_random rows seeded standard-normal scores, max_ragged rows one seeded random template.

Category: Attention; group: DSA. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=dsa_persistent_topk_decode. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `uniform, uniform_random: length[j] = context_len − next_n + 1 + (j mod next_n); max_ragged: length[b] = max(0, context_len − b)`
- `output[j] = top_k positions from logits[j, :length[j]], or natural positions when length[j] ≤ top_k`
- `rows = batch_size · next_n; logical bytes = 4 · Σ_j length[j] + 4 · rows + 4 · rows · top_k`
- `GB/s = logical bytes / time`

## How it is measured

torch is timed with CUDA events: five warm-up calls, then a loop of back-to-back calls, taking the median of three runs. vllm_cuda and vllm_fork_cuda use CUPTI kernel time with the L2 cache flushed before each call and count every launch of the call, the workspace memset and the persistent kernel. Operand construction and the output checks run before timing.

## Caveats

- Time follows the score distribution: the increasing uniform template crowds a short row's valid logits into a narrow value range and the random max_ragged template is shared by every row; uniform_random scores are independent normals, like a live indexer's continuous scores.
- Off SM90 the check before timing accepts a long row that differs from the reference only when the kernel's known threshold-bin buffer overflow explains the difference; such a row's timing is the production kernel's, but its selection is wrong.
- GB/s counts the valid FP32 logits, one int32 length and top_k int32 outputs per row, not padding, scratch space or physical transactions.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `batch_size` | number | requests | Requests decoded together. |
| `context_len` | number | tokens | Longest valid logit prefix per request. |
| `next_n` | number | tokens | Query rows per request in a decode step. |
| `max_model_len` | number | tokens | Visible width of each logit row. |
| `top_k` | number | tokens | Selected request-local positions per query row. |
| `logits_row_stride` | number | elements | Physical spacing between logit rows. |
| `logits_dtype` | dtype |  | Element type of the indexer logits. |
| `index_dtype` | label |  | Element type of the selected positions. |
| `context_mode` | label |  | Pattern of valid prefix lengths across requests; uniform_random also draws the scores at random instead of an increasing sequence. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `torch`: PyTorch's vectorized topk composite, with natural-index output for short rows. dtypes `fp32`, any CUDA GPU.
- `vllm_cuda`: vLLM's persistent_topk: a corrected build of the vLLM v0.23 kernel on SM90, and the vLLM build's op on every other GPU. dtypes `fp32`, any CUDA GPU.
- `vllm_fork_cuda`: persistent_topk as built in the pinned vLLM fork: one persistent kernel picks a histogram, medium or multi-CTA radix path by row length, with a filtered top-k kernel above batch 32. dtypes `fp32`, any CUDA GPU. Source: https://github.com/vllm-project/vllm/blob/main/csrc/libtorch_stable/persistent_topk.cuh
- `vllm_cuda_cooperative`: vLLM's cooperative_topk: one cooperative radix select for up to 64 rows; rejects larger batches. dtypes `fp32`, compute capability 9.0 or newer, GPUs H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip, H800-SXM5-80GB, H20-SXM-96GB. Source: https://github.com/vllm-project/vllm/blob/main/csrc/libtorch_stable/cooperative_topk.cu

## Measured rows

24,369 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `vllm_cuda` | `fp32` | 21,284 |
| NVIDIA B200 | `vllm_cuda_cooperative` | `fp32` | 262 |
| NVIDIA H200 | `vllm_cuda` | `fp32` | 2,634 |
| NVIDIA H200 | `vllm_fork_cuda` | `fp32` | 189 |

## Used by these deployments

- [DeepSeek V4 Flash 0731: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm.md) (`DeepSeek-V4-Flash-0731/deepseek_v4_vllm`)
- [DeepSeek V4 Flash 0731: vLLM, DP attention, EP, parallel streams serialized](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams.md) (`DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams`)
- [GLM-5.2 FP8: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-FP8/glm52_vllm_dsa_moe.md) (`GLM-5.2-FP8/glm52_vllm_dsa_moe`)
- [GLM-5.2 NVFP4: SGLang, TP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, TP and EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, TP and EP, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative`)
- [GLM-5.2 NVFP4: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_pp_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_pp_dsa_moe`)
- [GLM-5.2: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2/glm52_vllm_dsa_moe.md) (`GLM-5.2/glm52_vllm_dsa_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe`)
- [GLM-5.3-Flash: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe`)
- [GLM-5.3-Flash: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe`)
- [GLM-5.3-Flash: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe`)
- [GLM-5.3 NVFP4: vLLM, TP and EP, DFlash speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2.md) (`GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/dsa_persistent_topk_decode
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/dsa_persistent_topk_decode/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/dsa_persistent_topk_decode/configs

## Torch reference

What the kind computes, as Torch code (`profiling/runners/attention/dsa_persistent_topk_decode_reference.py` in ServingStudio Sim):

```python
"""Torch semantics for pinned persistent DSA decode top-k selection."""

from __future__ import annotations

import torch

from profiling.runners.attention.dsa_persistent_topk_decode import VLLM_TOP_K


def dsa_persistent_topk_decode_reference(
    logits: torch.Tensor,
    lengths: torch.Tensor,
    out: torch.Tensor,
    *,
    top_k: int,
    max_seq_len: int,
) -> torch.Tensor:
    """Select valid-prefix, row-local top-k indices into caller-owned storage.

    Rows no longer than ``top_k`` produce their natural local indices followed
    by ``-1``. Longer rows use value-descending order with ascending local index
    as the deterministic tie-break. Pinned CUDA does not contractually specify
    output order or equal-value tie selection, so its eventual correctness gate
    must compare selected sets and values for long rows.

    This models the vLLM v0.23.0 boundary at commit
    0fc695fc6d1d82e9a5ac6835ac8e4e1c83703665: schema
    ``persistent_topk(Tensor logits, Tensor lengths, Tensor! output, Tensor
    workspace, int k, int max_seq_len) -> ()`` in
    ``csrc/libtorch_stable/topk.cu`` and ``persistent_topk.cuh``. GLM-5.2 uses
    K=2048. The reference defines mathematical and valid-output semantics only;
    scratch-workspace reset, persistent/FilteredTopK dispatch, launch sequencing,
    and timing belong to a later production backend once that API is runnable.

    ``max_seq_len`` is validated dispatch metadata and otherwise does not alter
    the result. Invalid logits tails are intentionally never inspected because
    production ``clean_logits=false`` leaves them undefined.
    """
    _validate(
        logits,
        lengths,
        out,
        top_k=top_k,
        max_seq_len=max_seq_len,
    )

    flat_lengths = lengths.view(-1)
    out.fill_(-1)
    for row in range(logits.shape[0]):
        length = int(flat_lengths[row].item())
        if length == 0:
            continue

        if length <= top_k:
            selected = torch.arange(
                length,
                dtype=torch.int32,
                device=logits.device,
            )
        else:
            # Stable sorting preserves ascending local-index order among equal
            # values and touches only the contractual valid prefix.
            selected = torch.argsort(
                logits[row, :length],
                descending=True,
                stable=True,
            )[:top_k].to(torch.int32)
        out[row, : selected.numel()].copy_(selected)

    return out


def _validate(
    logits: object,
    lengths: object,
    out: object,
    *,
    top_k: object,
    max_seq_len: object,
) -> None:
    for name, tensor in (("logits", logits), ("lengths", lengths), ("out", out)):
        if not isinstance(tensor, torch.Tensor):
            raise TypeError(f"{name} must be a torch.Tensor")

    assert isinstance(logits, torch.Tensor)
    assert isinstance(lengths, torch.Tensor)
    assert isinstance(out, torch.Tensor)

    if type(top_k) is not int:
        raise TypeError("top_k must be a Python int")
    if top_k not in VLLM_TOP_K:
        raise ValueError(f"top_k must be one of {sorted(VLLM_TOP_K)}, got {top_k}")
    if type(max_seq_len) is not int:
        raise TypeError("max_seq_len must be a Python int")
    if max_seq_len < 0:
        raise ValueError("max_seq_len must be nonnegative")

    _validate_ranks_and_shapes(logits, lengths, out, top_k)
    _validate_dtypes_and_devices(logits, lengths, out)
    _validate_layouts(logits, lengths, out)

    if max_seq_len > logits.shape[1]:
        raise ValueError(
            f"max_seq_len must not exceed the logits width {logits.shape[1]}, got {max_seq_len}"
        )

    for name, tensor in (("logits", logits), ("lengths", lengths)):
        if torch._C._overlaps(out, tensor):
            raise ValueError(f"out must not alias {name}")

    flat_lengths = lengths.view(-1)
    if bool(torch.any(flat_lengths < 0).item()):
        raise ValueError("lengths must be nonnegative")
    if bool(torch.any(flat_lengths > max_seq_len).item()):
        raise ValueError(f"lengths must not exceed max_seq_len {max_seq_len}")

    # Check only prefixes the production result contracts. Invalid tails can
    # contain arbitrary NaN/Inf and must remain unread.
    for row in range(logits.shape[0]):
        length = int(flat_lengths[row].item())
        if length and not bool(torch.isfinite(logits[row, :length]).all().item()):
            raise ValueError(f"logits valid prefix for row {row} must be finite")


def _validate_ranks_and_shapes(
    logits: torch.Tensor,
    lengths: torch.Tensor,
    out: torch.Tensor,
    top_k: int,
) -> None:
    if logits.ndim != 2:
        raise ValueError(f"logits must be rank 2, got rank {logits.ndim}")
    if any(dimension <= 0 for dimension in logits.shape):
        raise ValueError(f"logits dimensions must be positive, got {tuple(logits.shape)}")
    if lengths.ndim not in (1, 2):
        raise ValueError(f"lengths must be rank 1 or 2, got rank {lengths.ndim}")
    if any(dimension <= 0 for dimension in lengths.shape):
        raise ValueError(f"lengths dimensions must be positive, got {tuple(lengths.shape)}")
    if out.ndim != 2:
        raise ValueError(f"out must be rank 2, got rank {out.ndim}")
    if any(dimension <= 0 for dimension in out.shape):
        raise ValueError(f"out dimensions must be positive, got {tuple(out.shape)}")

    num_rows = logits.shape[0]
    if lengths.numel() != num_rows:
        raise ValueError(f"lengths must contain exactly {num_rows} values, got {lengths.numel()}")
    expected_out_shape = (num_rows, top_k)
    if out.shape != expected_out_shape:
        raise ValueError(f"out shape must be {expected_out_shape}, got {tuple(out.shape)}")


def _validate_dtypes_and_devices(
    logits: torch.Tensor,
    lengths: torch.Tensor,
    out: torch.Tensor,
) -> None:
    if logits.dtype is not torch.float32:
        raise TypeError(f"logits dtype must be torch.float32, got {logits.dtype}")
    if lengths.dtype is not torch.int32:
        raise TypeError(f"lengths dtype must be torch.int32, got {lengths.dtype}")
    if out.dtype is not torch.int32:
        raise TypeError(f"out dtype must be torch.int32, got {out.dtype}")

    if any(tensor.device.type == "meta" for tensor in (logits, lengths, out)):
        raise ValueError("meta tensors are not supported")
    for name, tensor in (("lengths", lengths), ("out", out)):
        if tensor.device != logits.device:
            raise ValueError(
                f"{name} must be on the same device as logits, "
                f"got {tensor.device} and {logits.device}"
            )


def _validate_layouts(
    logits: torch.Tensor,
    lengths: torch.Tensor,
    out: torch.Tensor,
) -> None:
    for name, tensor in (("logits", logits), ("lengths", lengths), ("out", out)):
        if tensor.layout is not torch.strided:
            raise ValueError(f"{name} must have torch.strided layout")
        # Status 1 is definite overlap. Status 2 is indeterminate and remains
        # accepted for otherwise valid padded logits views.
        if int(torch._debug_has_internal_overlap(tensor)) == 1:
            raise ValueError(f"{name} must not have internal overlap")

    if logits.stride(-1) != 1:
        raise ValueError("logits innermost stride must be 1")
    if not lengths.is_contiguous():
        raise ValueError("lengths must be contiguous")
    if not out.is_contiguous():
        raise ValueError("out must be contiguous")
```

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/dsa_persistent_topk_decode.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
