# MoE routing finalization (`moe_finalize_routing`) | ServingStudio Kernel Library

Unpermute, scale and sum local expert outputs into one row per original token.

With expert parallelism, each GPU computes only the rows routed to its own experts. This TensorRT-LLM kernel, vendored in FlashInfer, gathers those rows back into token order, scales each by its router weight and sums them per token. token_count is all tokens; local_routed_token_count is the rows this GPU computed. The rows are grouped by local expert, as the upstream permutation leaves them.

Category: MoE; group: Routing and combine. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=moe_finalize_routing. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `output[token, h] = Σₖ local_scale[token, k] · expert_row[token, k, h]`
- `TFLOPS = 2·local_routed_token_count·hidden_size / time`
- `GB/s = (2·local_routed_token_count·hidden_size + 2·token_count·hidden_size + 8·local_routed_token_count + 4·token_count·top_k) / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. Only finalizeMoeRoutingKernel launches are counted. The JIT build and one launch run before timing.

## Caveats

- Rows routed elsewhere all point at one expert outside this GPU's range, which the kernel skips; the cross-GPU reduction that follows is a separate step.
- GB/s counts the local rows read and the output written, not the whole expanded-row buffer.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `token_count` | number | tokens | Original tokens in the batch. |
| `hidden_size` | number | elements | Features in each expert output row. |
| `top_k` | number | experts | Experts selected per original token. |
| `num_experts_per_rank` | number | experts | Experts owned by this rank. |
| `local_routed_token_count` | number | tokens | Selected expert rows resident on this rank. |
| `dtype` | dtype |  | Element type of the expert outputs and reduced result. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `flashinfer_trtllm`: FlashInfer's TensorRT-LLM finalizeMoeRoutingKernel through a direct BF16 SM90 launcher. dtypes `bf16`, SM targets `sm_90a`, GPUs H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, H800-SXM5-80GB, H20-SXM-96GB.

## Measured rows

2,137 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `flashinfer_trtllm_sm100` | `bf16` | 583 |
| NVIDIA H200 | `flashinfer_trtllm` | `bf16` | 1,554 |

## Used by these deployments

- [GLM-5.2 FP8: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-FP8/glm52_vllm_dsa_moe.md) (`GLM-5.2-FP8/glm52_vllm_dsa_moe`)
- [GLM-5.2: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2/glm52_vllm_dsa_moe.md) (`GLM-5.2/glm52_vllm_dsa_moe`)
- [Qwen3 235B-A22B FP8: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_vllm_moe_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-FP8/qwen3_vllm_moe_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B Thinking 2507 FP8: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_vllm_moe_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_vllm_moe_dp_attn_ep_ffn`)
- [Qwen3.6 35B-A3B FP8: Single GPU](https://servingstudio.cs.washington.edu/models/Qwen3.6-35B-A3B-FP8/qwen36_local.md) (`Qwen3.6-35B-A3B-FP8/qwen36_local`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/moe_finalize_routing
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/moe_finalize_routing/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/moe_finalize_routing/configs

## Torch reference

What the kind computes, as Torch code (`profiling/runners/moe/flashinfer_trtllm_finalize.py` in ServingStudio Sim):

```python
"""Direct FlashInfer/TensorRT-LLM BF16 MoE finalize-routing runner."""

from __future__ import annotations

import functools
from collections.abc import Callable
from dataclasses import dataclass
from pathlib import Path
from typing import Any

from profiling.db.args import DType
from profiling.profilers.energy import Energy
from profiling.profilers.timer import Timer
from profiling.runners.device import require_cuda_toolkit
from profiling.runners.exceptions import KernelLaunchFailed, ProfilerNotImplemented
from profiling.runners.metrics import ComputeMetrics

_CUDA_MINIMUM = (12, 8)
_KERNEL_NAME = "finalizeMoeRoutingKernel"
_JIT_MODULE_NAME = "vibesim_moe_finalize_routing_bf16_sm90"


@dataclass(frozen=True)
class MoeFinalizeRoutingLaunch:
    run_once: Callable[[], None]
    expanded_permuted_rows: Any
    reduced_unpermuted_output: Any
    final_scales: Any
    unpermuted_row_to_permuted_row: Any
    token_selected_experts: Any
    token_count: int
    hidden_size: int
    top_k: int
    num_experts_per_rank: int
    local_routed_token_count: int


def _validate_args(
    token_count: int,
    hidden_size: int,
    top_k: int,
    num_experts_per_rank: int,
    local_routed_token_count: int,
    dtype: DType | str,
) -> tuple[int, int, int, int, int, DType]:
    token_count = int(token_count)
    hidden_size = int(hidden_size)
    top_k = int(top_k)
    num_experts_per_rank = int(num_experts_per_rank)
    local_routed_token_count = int(local_routed_token_count)
    dtype = DType.from_value(dtype)

    if token_count <= 0:
        raise ValueError(f"token_count must be > 0, got {token_count}")
    if hidden_size <= 0 or hidden_size % 8 != 0:
        raise ValueError(f"hidden_size must be > 0 and divisible by 8, got {hidden_size}")
    if top_k <= 0:
        raise ValueError(f"top_k must be > 0, got {top_k}")
    if num_experts_per_rank <= 0:
        raise ValueError(f"num_experts_per_rank must be > 0, got {num_experts_per_rank}")
    routed_capacity = token_count * top_k
    if local_routed_token_count < 0 or local_routed_token_count > routed_capacity:
        raise ValueError(
            "local_routed_token_count must be in "
            f"[0, token_count*top_k={routed_capacity}], got {local_routed_token_count}"
        )
    if dtype is not DType.BF16:
        raise ValueError(
            f"flashinfer_trtllm moe_finalize_routing requires dtype=bf16, got {dtype.value}"
        )
    return (
        token_count,
        hidden_size,
        top_k,
        num_experts_per_rank,
        local_routed_token_count,
        dtype,
    )


def _balanced_local_slots(routed_capacity: int, local_count: int) -> tuple[bool, ...]:
    """Spread exactly ``local_count`` assignments across the routed slots."""

    if routed_capacity <= 0:
        raise ValueError("routed_capacity must be > 0")
    if local_count < 0 or local_count > routed_capacity:
        raise ValueError("local_count must be within routed_capacity")
    return tuple(
        ((slot_index + 1) * local_count) // routed_capacity
        > (slot_index * local_count) // routed_capacity
        for slot_index in range(routed_capacity)
    )


def _routing_metadata(
    token_count: int,
    top_k: int,
    num_experts_per_rank: int,
    local_routed_token_count: int,
) -> tuple[list[int], list[int], list[float]]:
    routed_capacity = token_count * top_k
    local_slots = _balanced_local_slots(routed_capacity, local_routed_token_count)
    selected_experts: list[int] = []
    local_assignment_index = 0
    for is_local in local_slots:
        if is_local:
            selected_experts.append(local_assignment_index % num_experts_per_rank)
            local_assignment_index += 1
        else:
            # Rank zero owns [0, E); E is the first expert on the synthetic
            # remote rank and is therefore skipped by the production kernel.
            selected_experts.append(num_experts_per_rank)

    # The upstream permutation groups valid rows contiguously by local expert.
    # Preserve that physical locality rather than merely inventing an arbitrary
    # bijection: the finalize kernel follows this map for every local load.
    local_expert_counts = [0] * num_experts_per_rank
    for expert_id in selected_experts:
        if expert_id < num_experts_per_rank:
            local_expert_counts[expert_id] += 1
    local_expert_offsets: list[int] = []
    running_offset = 0
    for expert_count in local_expert_counts:
        local_expert_offsets.append(running_offset)
        running_offset += expert_count
    next_local_row = local_expert_offsets.copy()
    next_remote_row = local_routed_token_count

    unpermute_map = [0] * routed_capacity
    for token_index in range(token_count):
        for top_k_index in range(top_k):
            column_major_slot = token_index + top_k_index * token_count
            row_major_slot = token_index * top_k + top_k_index
            expert_id = selected_experts[row_major_slot]
            if expert_id < num_experts_per_rank:
                unpermute_map[column_major_slot] = next_local_row[expert_id]
                next_local_row[expert_id] += 1
            else:
                # Remote mappings are not dereferenced by this rank, but a full
                # permutation keeps the synthetic metadata structurally legal.
                unpermute_map[column_major_slot] = next_remote_row
                next_remote_row += 1

    # Non-uniform positive scales make the correctness check exercise DEFAULT
    # scaling rather than accidentally accepting the NO_SCALE specialization.
    final_scales = [0.25 + 0.5 * ((slot_index % 7) / 6.0) for slot_index in range(routed_capacity)]
    return selected_experts, unpermute_map, final_scales


@functools.cache
def _load_finalize_module():
    """Build the thin launcher against the selected FlashInfer wheel."""

    try:
        from flashinfer.jit import env as jit_env
        from flashinfer.jit.core import gen_jit_spec, sm90a_nvcc_flags
    except ImportError as exc:
        raise ProfilerNotImplemented(
            "FlashInfer JIT support is required for moe_finalize_routing"
        ) from exc

    binding_source = Path(__file__).resolve().parent / "csrc" / "moe_finalize_routing.cu"
    if not binding_source.is_file():
        raise ProfilerNotImplemented(f"MoE finalize JIT binding is missing: {binding_source}")

    source_root = jit_env.FLASHINFER_CSRC_DIR
    common_source_root = source_root / "nv_internal" / "cpp" / "common"
    include_paths = [
        source_root,
        source_root / "nv_internal",
        source_root / "nv_internal" / "include",
        source_root / "nv_internal" / "tensorrt_llm" / "cutlass_extensions" / "include",
        source_root / "nv_internal" / "tensorrt_llm" / "kernels" / "cutlass_kernels" / "include",
        source_root / "nv_internal" / "tensorrt_llm" / "kernels" / "cutlass_kernels",
    ]
    nvcc_flags = sm90a_nvcc_flags + [
        "-DCOMPILE_HOPPER_TMA_GEMMS",
        "-DCOMPILE_HOPPER_TMA_GROUPED_GEMMS",
        "-DENABLE_BF16",
        "-DENABLE_FP8",
        "-DENABLE_FP8_BLOCK_SCALE",
        "-DUSING_OSS_CUTLASS_MOE_GEMM",
        "-DCUTLASS_ENABLE_GDC_FOR_SM90=1",
    ]
    return gen_jit_spec(
        _JIT_MODULE_NAME,
        [
            binding_source,
            common_source_root / "envUtils.cpp",
            common_source_root / "logger.cpp",
            common_source_root / "stringUtils.cpp",
            common_source_root / "tllmException.cpp",
        ],
        extra_cuda_cflags=nvcc_flags,
        extra_include_paths=include_paths,
    ).build_and_load()


def prepare_moe_finalize_routing_launch(
    torch: Any,
    *,
    token_count: int,
    hidden_size: int,
    top_k: int,
    num_experts_per_rank: int,
    local_routed_token_count: int,
) -> MoeFinalizeRoutingLaunch:
    selected_experts, unpermute_map, scale_values = _routing_metadata(
        token_count,
        top_k,
        num_experts_per_rank,
        local_routed_token_count,
    )
    routed_capacity = token_count * top_k
    expanded_permuted_rows = torch.randn(
        (routed_capacity, hidden_size), dtype=torch.bfloat16, device="cuda"
    )
    reduced_unpermuted_output = torch.empty(
        (token_count, hidden_size), dtype=torch.bfloat16, device="cuda"
    )
    final_scales = torch.tensor(scale_values, dtype=torch.float32, device="cuda")
    unpermuted_row_to_permuted_row = torch.tensor(unpermute_map, dtype=torch.int32, device="cuda")
    token_selected_experts = torch.tensor(selected_experts, dtype=torch.int32, device="cuda")

    def run_once() -> None:
        module = _load_finalize_module()
        module.run_moe_finalize_routing(
            expanded_permuted_rows,
            reduced_unpermuted_output,
            final_scales,
            unpermuted_row_to_permuted_row,
            token_selected_experts,
            token_count,
            hidden_size,
            top_k,
            num_experts_per_rank,
        )

    return MoeFinalizeRoutingLaunch(
        run_once=run_once,
        expanded_permuted_rows=expanded_permuted_rows,
        reduced_unpermuted_output=reduced_unpermuted_output,
        final_scales=final_scales,
        unpermuted_row_to_permuted_row=unpermuted_row_to_permuted_row,
        token_selected_experts=token_selected_experts,
        token_count=token_count,
        hidden_size=hidden_size,
        top_k=top_k,
        num_experts_per_rank=num_experts_per_rank,
        local_routed_token_count=local_routed_token_count,
    )


def torch_reference(launch: MoeFinalizeRoutingLaunch, torch: Any) -> Any:
    """Small-shape semantic reference for the direct launcher."""

    output = torch.zeros(
        (launch.token_count, launch.hidden_size),
        dtype=torch.float32,
        device=launch.expanded_permuted_rows.device,
    )
    expanded = launch.expanded_permuted_rows.float()
    for token_index in range(launch.token_count):
        for top_k_index in range(launch.top_k):
            routed_slot = token_index * launch.top_k + top_k_index
            expert_id = int(launch.token_selected_experts[routed_slot].item())
            if expert_id >= launch.num_experts_per_rank:
                continue
            mapping_slot = token_index + top_k_index * launch.token_count
            permuted_row = int(launch.unpermuted_row_to_permuted_row[mapping_slot].item())
            scale = launch.final_scales[routed_slot]
            output[token_index] += expanded[permuted_row] * scale
    return output.to(torch.bfloat16)


def _logical_bytes(
    token_count: int,
    hidden_size: int,
    top_k: int,
    local_routed_token_count: int,
) -> int:
    return (
        local_routed_token_count * hidden_size * 2
        + token_count * hidden_size * 2
        + local_routed_token_count * 4
        + local_routed_token_count * 4
        + token_count * top_k * 4
    )


def profile_moe_finalize_routing_flashinfer_trtllm(
    token_count: int,
    hidden_size: int,
    top_k: int,
    num_experts_per_rank: int,
    local_routed_token_count: int,
    dtype: DType | str,
) -> ComputeMetrics:
    (
        token_count,
        hidden_size,
        top_k,
        num_experts_per_rank,
        local_routed_token_count,
        _,
    ) = _validate_args(
        token_count,
        hidden_size,
        top_k,
        num_experts_per_rank,
        local_routed_token_count,
        dtype,
    )
    try:
        import torch
    except ImportError as exc:
        raise ProfilerNotImplemented("torch is required for moe_finalize_routing") from exc

    require_cuda_toolkit(torch, _CUDA_MINIMUM, "flashinfer_trtllm moe_finalize_routing")
    try:
        launch = prepare_moe_finalize_routing_launch(
            torch,
            token_count=token_count,
            hidden_size=hidden_size,
            top_k=top_k,
            num_experts_per_rank=num_experts_per_rank,
            local_routed_token_count=local_routed_token_count,
        )
        launch.run_once()
        torch.cuda.synchronize()
        time_ms = Timer.cupti(launch.run_once, kernel_name=_KERNEL_NAME)
        energy_j = Energy.perf(launch.run_once, per_iter_time_ms=time_ms)

        flops = 2 * local_routed_token_count * hidden_size
        tflops = (flops / (time_ms / 1000.0)) / 1e12 if time_ms > 0 else 0.0
        bytes_accessed = _logical_bytes(token_count, hidden_size, top_k, local_routed_token_count)
        bandwidth_gbps = (bytes_accessed / (time_ms / 1000.0)) / 1e9 if time_ms > 0 else 0.0
        return ComputeMetrics(
            time_ms=float(time_ms),
            tflops=float(tflops),
            memory_bandwidth_gbps=float(bandwidth_gbps),
            energy_j=float(energy_j),
        )
    except (RuntimeError, ValueError) as exc:
        raise KernelLaunchFailed(str(exc)) from exc


__all__ = [
    "MoeFinalizeRoutingLaunch",
    "prepare_moe_finalize_routing_launch",
    "profile_moe_finalize_routing_flashinfer_trtllm",
    "torch_reference",
]
```

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/moe_finalize_routing.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
