# Logits top-k (`logits_topk`) | ServingStudio Kernel Library

Select the largest top_k scores and their column indices from each score row.

vLLM's vocab-parallel top-k takes the top_k logits from each GPU's vocabulary shard, all-gathers the candidates, and takes top_k again from the gathered block, without gathering the full logits. Both steps are this kernel at different widths: num_columns is the shard's vocabulary in the first and top_k · tp_size in the second. Scores are seeded normal values.

Category: Other. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=logits_topk. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `TFLOPS = num_rows · num_columns / time`
- `GB/s = num_rows · [num_columns · bytes per value + top_k · (bytes per value + 4)] / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. Every launch of the call is counted: selection, the value sort and the index conversion. FlashInfer's selected values are checked against torch.topk before timing.

## Caveats

- TFLOPS treats one score comparison per input element as nominal work; the selection algorithms can inspect elements more than once.
- GB/s counts one read of the scores and writes of top_k values and 4-byte indices per row. FlashInfer converts its int32 indices to int64, so this is not total device traffic.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_rows` | number | rows | Score rows, one per scored position. |
| `num_columns` | number | elements | Scores in each row. |
| `top_k` | number | elements | Scores kept from each row. |
| `dtype` | dtype |  | Element type of the input scores and output values. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `torch`: torch.topk, vLLM's fallback when FlashInfer is unavailable. dtypes `bf16`, `fp16`, `fp32`, any CUDA GPU.
- `flashinfer`: FlashInfer top_k with sorted=True and deterministic=True, the call vLLM's logits processor makes. dtypes `bf16`, `fp16`, `fp32`, any CUDA GPU. Source: https://github.com/flashinfer-ai/flashinfer/blob/main/flashinfer/topk.py

## Measured rows

68 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `flashinfer` | `bf16` | 34 |
| NVIDIA B200 | `torch` | `bf16` | 34 |

## Used by these deployments

- [GLM-5.3 NVFP4: vLLM, TP and EP, DFlash speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2.md) (`GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/logits_topk
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/logits_topk/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/logits_topk/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/logits_topk.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
