# BF16 fused MoE (`bf16_fused_moe`) | ServingStudio Kernel Library

Route tokens through BF16 experts and combine their gated MLP outputs.

An MoE layer runs its routed experts as this one FlashInfer call: routing, the gate-up projection, SwiGLU, the down projection and the weighted combine. per_expert_batches gives the tokens routed to every expert across all GPUs, and the first num_local_experts are this GPU's. The router logits are built so that routing selects exactly those counts.

Category: MoE; group: Expert compute. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=bf16_fused_moe. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `local_rows = sum(per_expert_batches[:num_local_experts])`
- `active_experts = count(per_expert_batches[:num_local_experts] > 0)`
- `TFLOPS = 2·local_rows·(hidden_size·2·intermediate_size + intermediate_size·hidden_size) / time`
- `logical_bytes = 2·num_tokens·num_experts + 2·num_experts + 2·local_rows·hidden_size + active_experts·(4·intermediate_size·hidden_size + 2·hidden_size·intermediate_size) + 2·num_tokens·hidden_size`
- `GB/s = logical_bytes / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. Overlapping launches count once, by the time the GPU is busy. Weight preparation and one validated call run before timing.

## Caveats

- GB/s counts logical traffic: routing inputs, the local expert rows, the weights of experts that have rows, and the output.
- The measured call always finalizes the output; vLLM can defer the combine to a later kernel.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_tokens` | number | tokens | Tokens entering the routed MoE layer. |
| `hidden_size` | number | elements | Width of each token's hidden state. |
| `intermediate_size` | number | elements | Width of each expert's intermediate state. |
| `num_experts` | number | experts | Routed experts across all GPUs. |
| `num_local_experts` | number | experts | Routed experts whose weights reside on this GPU. |
| `top_k` | number | experts | Experts selected for each token. |
| `dtype` | dtype |  | Element type of the hidden states and expert weights. |
| `routing_method` | label |  | Router rule, as FlashInfer's RoutingMethodType names it; the runner builds minimax2: sigmoid scores plus a selection bias, weights renormalized. |
| `n_group` | number | groups | Expert groups considered by the router. |
| `topk_group` | number | groups | Expert groups retained before expert selection. |
| `routed_scaling_numerator` | number | parts | Numerator of the routed output scaling factor. |
| `routed_scaling_denominator` | number | parts | Denominator of the routed output scaling factor. |
| `per_expert_batches` | list | tokens | Selected token count for each global expert, in expert order. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `flashinfer_trtllm_sm100`: FlashInfer trtllm_bf16_moe on B200, with vLLM's weight layout: routing and both expert projections in one call. dtypes `bf16`, SM targets `sm_100f`, GPUs B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip. Source: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/fused_moe/experts/trtllm_bf16_moe.py

## Measured rows

6,324 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `flashinfer_trtllm_sm100` | `bf16` | 6,324 |

## Used by these deployments

- [GLM-5.2 NVFP4: vLLM, TP and EP, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative`)
- [Inkling NVFP4: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Inkling-NVFP4/inkling_vllm_nvfp4_rel_attn_moe.md) (`Inkling-NVFP4/inkling_vllm_nvfp4_rel_attn_moe`)
- [Inkling Small NVFP4: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Inkling-Small-NVFP4/inkling_vllm_nvfp4_rel_attn_moe.md) (`Inkling-Small-NVFP4/inkling_vllm_nvfp4_rel_attn_moe`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/bf16_fused_moe
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/bf16_fused_moe/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/bf16_fused_moe/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/bf16_fused_moe.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
