# Fused MoE top-k routing (`moe_fused_topk`) | ServingStudio Kernel Library

Select top-k experts from softmax router scores and renormalize their weights.

The step after the MoE router projection: from each token's num_experts logits, take the softmax, keep the top_k experts, and renormalize their probabilities. It returns the weights, the expert IDs and the source index of each routed row. The logits are fixed, tie-free bf16 values rather than model outputs.

Category: MoE; group: Routing and combine. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=moe_fused_topk. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `p = softmax(logits); weights = top_k(p) / sum(top_k(p))`
- `source_index(token, slot) = slot · num_tokens + token`
- `TFLOPS = num_tokens · (5·num_experts − 2 + top_k·(num_experts − 1) − top_k·(top_k − 1)/2 + 2·top_k − 1) / time`
- `GB/s = (2·num_tokens·num_experts + 12·num_tokens·top_k) / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. vllm_cuda counts every kernel of one fused_topk call: a topkGating specialization where vLLM has one for the expert count, otherwise the moeSoftmax and moeTopK pair. torch counts every launch of its sequence. Outputs are checked against the reference first.

## Caveats

- TFLOPS counts comparisons and exponentials as nominal operations.
- GB/s counts one logits read and the three outputs; the torch backend's workspaces are not counted.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_tokens` | number | tokens | Tokens whose router logits are selected. |
| `num_experts` | number | experts | Experts scored for each token. |
| `top_k` | number | experts | Experts selected per token. |
| `dtype` | dtype |  | Element type of the router logits. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `torch`: PyTorch softmax and iterative top-k with preallocated workspaces. dtypes `bf16`, any CUDA GPU.
- `vllm_cuda`: vLLM fused_topk: one topkGating launch for a specialized expert count. dtypes `bf16`, any CUDA GPU. Source: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/fused_moe/router/fused_topk_router.py

## Measured rows

42 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA H200 | `vllm_cuda` | `bf16` | 42 |

## Used by these deployments

- [Qwen3.6 35B-A3B FP8: Single GPU](https://servingstudio.cs.washington.edu/models/Qwen3.6-35B-A3B-FP8/qwen36_local.md) (`Qwen3.6-35B-A3B-FP8/qwen36_local`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/moe_fused_topk
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/moe_fused_topk/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/moe_fused_topk/configs

## Torch reference

What the kind computes, as Torch code (`profiling/runners/moe/moe_fused_topk_reference.py` in ServingStudio Sim):

```python
"""Torch CPU semantics for vLLM's fused MoE top-k router selection."""

from __future__ import annotations

import torch

__all__ = ["moe_fused_topk_reference"]

_INT32_MAX = torch.iinfo(torch.int32).max


def moe_fused_topk_reference(
    logits: torch.Tensor,
    top_k: int,
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
    """Select and renormalize experts from dense BF16 router logits.

    Each row is converted to FP32 and normalized with a stable softmax over all
    experts. Experts are then selected iteratively in descending probability
    order, with exact ties resolved in favor of the lower expert ID. The
    selected probabilities are renormalized to sum to one, matching vLLM's
    ``topkGating`` production path.

    Returns fresh contiguous FP32 weights, int32 expert IDs, and int32 source
    indices. Source index ``(token, slot)`` is ``slot * T + token``.
    """
    num_tokens, num_experts = _validate(logits, top_k)

    logits_fp32 = logits.float()
    row_max = logits_fp32.max(dim=1, keepdim=True).values
    exponentials = torch.exp(logits_fp32 - row_max)
    probabilities = exponentials / exponentials.sum(dim=1, keepdim=True)

    weights = torch.empty((num_tokens, top_k), dtype=torch.float32)
    expert_ids = torch.empty((num_tokens, top_k), dtype=torch.int32)
    source_indices = torch.empty((num_tokens, top_k), dtype=torch.int32)

    for token in range(num_tokens):
        selected: set[int] = set()
        for slot in range(top_k):
            best_expert = -1
            best_probability = -1.0
            for expert in range(num_experts):
                if expert in selected:
                    continue
                probability = float(probabilities[token, expert])
                # Scanning IDs in ascending order and updating only for a
                # strict improvement makes lower IDs win exact ties.
                if probability > best_probability:
                    best_probability = probability
                    best_expert = expert
            selected.add(best_expert)
            weights[token, slot] = best_probability
            expert_ids[token, slot] = best_expert
            source_indices[token, slot] = slot * num_tokens + token

        weights[token].div_(weights[token].sum())

    return weights.contiguous(), expert_ids.contiguous(), source_indices.contiguous()


def _validate(logits: object, top_k: object) -> tuple[int, int]:
    if not isinstance(logits, torch.Tensor):
        raise TypeError("logits must be a torch.Tensor")
    if logits.layout is not torch.strided:
        raise ValueError("logits must have torch.strided layout")
    if logits.device.type == "meta":
        raise ValueError("meta tensors are not supported")
    if logits.device.type != "cpu":
        raise ValueError(f"semantic reference requires CPU logits, got {logits.device.type}")
    if logits.ndim != 2:
        raise ValueError(f"logits must be rank 2, got rank {logits.ndim}")
    if logits.dtype is not torch.bfloat16:
        raise TypeError(f"logits dtype must be torch.bfloat16, got {logits.dtype}")

    num_tokens, num_experts = logits.shape
    if num_tokens <= 0 or num_experts <= 0:
        raise ValueError("logits token and expert dimensions must be positive")
    if type(top_k) is not int:
        raise TypeError("top_k must be an exact integer")
    if top_k <= 0 or top_k > num_experts:
        raise ValueError(f"top_k must be in [1, {num_experts}], got {top_k}")
    if num_tokens * top_k - 1 > _INT32_MAX:
        raise ValueError("source indices exceed the int32 range")
    if not torch.isfinite(logits).all().item():
        raise ValueError("logits must contain only finite values")
    return num_tokens, num_experts
```

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/moe_fused_topk.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
