# Block-scaled fused MoE (`nvfp4_fused_moe`) | ServingStudio Kernel Library

Route block-quantized tokens through packed expert weights and produce their MoE outputs.

A routed MoE layer runs as this one FlashInfer call on hidden states already quantized to weight_format (NVFP4, FP8 with 128-wide block scales, or MXFP8 for MXFP4 weights), or on the BF16 hidden states themselves for the weight-only MXFP4/BF16 backend: routing, the gate-up projection, SwiGLU and the down projection. per_expert_batches gives the tokens routed to every expert across all GPUs, and the first num_local_experts are this GPU's. The vLLM backend also combines the expert outputs; the SGLang backend leaves the combine to a later kernel, moe_finalize_fuse_shared. The FP8 block-scale, MXFP4 and MXFP4/BF16 backends also combine. The routed backend takes top-k IDs and weights chosen before the call, as vLLM does under data plus expert parallelism, so routing_method, n_group, topk_group and the scaling factor do not reach the kernel. The MXFP4 backend takes finished top-k ids and weights, so its routing step only groups tokens by expert.

Category: MoE; group: Expert compute. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=nvfp4_fused_moe. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `local_rows = sum(per_expert_batches[:num_local_experts])`
- `active_experts = count(per_expert_batches[:num_local_experts] > 0)`
- `TFLOPS = 2·local_rows·(hidden_size·2·intermediate_size + intermediate_size·hidden_size) / time`
- `output_rows = num_tokens for vLLM; local_rows for SGLang`
- `routing_bytes = b·(num_tokens·num_experts + num_experts); b = 2 for minimax2 (BF16 logits and bias), 4 for deepseek_v3 (FP32); renormalize_naive: routing_bytes = 2·num_tokens·num_experts (BF16 logits, no bias)`
- `routed backend: routing_bytes = 8·num_tokens·top_k`
- `activation_bytes = local_rows·(hidden_size/2 + hidden_size/group_size)`
- `weight_bytes = active_experts·(intermediate_size·hidden_size + intermediate_size·hidden_size/8 + hidden_size·intermediate_size/2 + hidden_size·intermediate_size/group_size)`
- `GB/s = (routing_bytes + activation_bytes + weight_bytes + 12·active_experts + 2·hidden_size·output_rows) / time`
- `FP8 block-scale: routing_bytes = 4·num_tokens·num_experts + 4·num_experts (deepseek_v3), 2·num_tokens·num_experts (renormalize_naive); activation_bytes = local_rows·(hidden_size + 4·hidden_size/128); weight_bytes = active_experts·(3·intermediate_size·hidden_size + 4·3·intermediate_size·hidden_size/128²); GB/s = (routing_bytes + activation_bytes + weight_bytes + 2·hidden_size·num_tokens) / time`
- `MXFP4: routing_bytes = 8·num_tokens·top_k; activation_bytes = local_rows·(hidden_size + hidden_size/32); weight_bytes = active_experts·(3·intermediate_size·hidden_size/2 + 3·intermediate_size·hidden_size/32); GB/s = (routing_bytes + activation_bytes + weight_bytes + 4·num_local_experts + 2·hidden_size·num_tokens) / time`
- `MXFP4/BF16: routing_bytes = 4·(num_tokens·num_experts + num_experts) (FP32 logits and bias, deepseek_v3); activation_bytes = 2·local_rows·hidden_size (BF16, no scale); weight_bytes as MXFP4; GB/s = (routing_bytes + activation_bytes + weight_bytes + 2·hidden_size·num_tokens) / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. Overlapping launches count once, by the time the GPU is busy. The vLLM backend autotunes the call first, as vLLM does; the SGLang backend does not, because SGLang tunes a different signature. Three warm-up calls follow.

## Caveats

- The backends end at different points, so their times cover different work: the vLLM backends include the combine, SGLang does not.
- deepseek_v3 routing is accepted only ungrouped (n_group = topk_group = 1), where forced logits can realize any per-expert histogram; FlashInfer then runs the same routing kernel as minimax2, with the routed scale and FP32 inputs.
- The FP8 block-scale backend's autotuner stops at 8192 tokens, so larger shapes reuse the largest tuned bucket.
- The MXFP4 backend accepts only precomputed_dsv4 routing with group 32, n_group = topk_group = 1 and routed scaling 1/1, with the SwiGLU clamp applied; it too tunes up to 8192 tokens.
- The MXFP4/BF16 backend accepts only ungrouped deepseek_v3 routing over FP32 logits and bias, with group 32 and no SwiGLU clamp, bias, alpha or beta; it too tunes up to 8192 tokens. Its tensor cores run BF16, not the weight format: the Rust config states that with activation_format, and the precision column still shows the weight format.
- Activation quantization is outside the call, and the router logits are synthetic.
- The vLLM NVFP4 backends tune up to max(8192, next power of two ≥ num_tokens); vLLM tunes up to max_num_batched_tokens·dp_size, which picks the same bucket for every shape it serves.
- The NVFP4 backends pass no SwiGLU clamp (gemm1_clamp_limit is None); vLLM passes the model's swiglu_limit when it has one. FP8 block-scale deepseek_v3 rows were measured with SwiGLU clamp 10, renormalize_naive rows without; the clamp is not a key axis.
- renormalize_naive routing takes n_group = topk_group = 0 and no correction bias, and does not apply the routed scale.
- GB/s counts logical traffic for the local rows and the experts that have rows; small side outputs are left out.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_tokens` | number | tokens | Tokens entering the routed MoE layer. |
| `hidden_size` | number | elements | Width of each token's hidden state. |
| `intermediate_size` | number | elements | Width of each expert's intermediate state. |
| `num_experts` | number | experts | Routed experts across all GPUs. |
| `num_local_experts` | number | experts | Routed experts whose weights reside on this GPU. |
| `top_k` | number | experts | Experts selected for each token. |
| `input_dtype` | dtype |  | Element type of the hidden states before any quantization, and of the output. The MXFP4/BF16 backend keeps the activations in it. |
| `weight_format` | dtype |  | Packed expert weight format. It is also the tensor-core precision when the activations are quantized to it: nvfp4_e2m1 for the NVFP4 backends, fp8_e4m3 for the FP8 block-scale backend, mxfp4_e2m1 for the MXFP4 backend (whose activations are MXFP8). The MXFP4/BF16 backend takes mxfp4_e2m1 weights but runs BF16 MMA on input_dtype activations. |
| `group_size` | number | elements | Elements sharing one scale: 16 for NVFP4; 128 for FP8, per 1x128 activation group and 128x128 weight block; 32 for MXFP4. |
| `routing_method` | label |  | Router selection rule. minimax2: sigmoid + bias top-k over BF16 logits. deepseek_v3: DeepSeekV3 sigmoid + bias top-k with the routed scale, over FP32 logits and bias. renormalize_naive: softmax top-k renormalized, over BF16 logits with no bias and no groups (n_group = topk_group = 0). precomputed_dsv4: finished top-k ids and weights. The vLLM NVFP4 backends take minimax2, deepseek_v3 or renormalize_naive, the SGLang backend minimax2, the FP8 block-scale backend deepseek_v3 or renormalize_naive, the MXFP4/BF16 backend deepseek_v3, the MXFP4 backend precomputed_dsv4. |
| `n_group` | number | groups | Expert groups considered by the router. |
| `topk_group` | number | groups | Expert groups retained before expert selection. |
| `routed_scaling_numerator` | number | parts | Numerator of the routed output scaling factor. |
| `routed_scaling_denominator` | number | parts | Denominator of the routed output scaling factor. |
| `per_expert_batches` | list | tokens | Selected token count for each global expert, in expert order. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `flashinfer_trtllm_sm100`: FlashInfer trtllm_fp4_block_scale_moe through vLLM's NVFP4 path, including final expert combination. dtypes `nvfp4_e2m1`, SM targets `sm_100f`, GPUs B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip. Source: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/fused_moe/experts/trtllm_nvfp4_moe.py
- `flashinfer_trtllm_routed_sm100`: FlashInfer trtllm_fp4_block_scale_routed_moe through vLLM's modular NVFP4 path under data plus expert parallelism: top-k IDs and fp32 weights are selected before dispatch, and the call finalizes. dtypes `nvfp4_e2m1`, SM targets `sm_100f`, GPUs B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip. Source: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/fused_moe/experts/trtllm_nvfp4_moe.py
- `flashinfer_trtllm_routed_sm100_fork`: flashinfer_trtllm_routed_sm100's trtllm_fp4_block_scale_routed_moe call, measured with the alignment vLLM checkout's FlashInfer instead of the pinned profiler image's. dtypes `nvfp4_e2m1`, SM targets `sm_100f`, GPUs B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip. Source: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/fused_moe/experts/trtllm_nvfp4_moe.py
- `flashinfer_trtllm_sm100_deferred_finalize`: FlashInfer trtllm_fp4_block_scale_moe through SGLang, leaving expert combination to a later call. dtypes `nvfp4_e2m1`, SM targets `sm_100f`, GPUs B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip. Source: https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/layers/moe/moe_runner/flashinfer_trtllm.py
- `flashinfer_trtllm_fp8_block_sm100`: FlashInfer trtllm_fp8_block_scale_moe through vLLM's FP8 block-scale path, including final expert combination. dtypes `fp8_e4m3`, SM targets `sm_100f`, GPUs B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip. Source: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/fused_moe/experts/trtllm_fp8_moe.py
- `flashinfer_trtllm_sm100_mxfp4`: FlashInfer trtllm_fp4_block_scale_routed_moe (MXFP4 weights, MXFP8 activations) on precomputed top-k, as vLLM's TRT-LLM MXFP4 experts call it, including final expert combination. dtypes `mxfp4_e2m1`, SM targets `sm_100f`, GPUs B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip. Source: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/fused_moe/experts/trtllm_mxfp4_moe.py
- `flashinfer_trtllm_sm100_mxfp4_bf16`: FlashInfer trtllm_fp4_block_scale_moe with MXFP4 weights and BF16 activations (BF16 MMA), routing inside the call, as vLLM's FLASHINFER_TRTLLM_MXFP4_BF16 monolithic experts call it, including final expert combination. dtypes `bf16`, SM targets `sm_100f`, GPUs B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip. Source: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/fused_moe/experts/trtllm_mxfp4_moe.py

## Measured rows

13,358 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `flashinfer_trtllm_fp8_block_sm100` | `fp8_e4m3` | 1,515 |
| NVIDIA B200 | `flashinfer_trtllm_routed_sm100` | `nvfp4_e2m1` | 1,960 |
| NVIDIA B200 | `flashinfer_trtllm_routed_sm100_fork` | `nvfp4_e2m1` | 1,320 |
| NVIDIA B200 | `flashinfer_trtllm_sm100` | `nvfp4_e2m1` | 7,777 |
| NVIDIA B200 | `flashinfer_trtllm_sm100_deferred_finalize` | `nvfp4_e2m1` | 137 |
| NVIDIA B200 | `flashinfer_trtllm_sm100_mxfp4` | `mxfp4_e2m1` | 357 |
| NVIDIA B200 | `flashinfer_trtllm_sm100_mxfp4_bf16` | `mxfp4_e2m1` | 292 |

## Used by these deployments

- [DeepSeek V4.1 Flash 1017: vLLM, DP attention, EP MoE, DSpark speculative decoding](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dp_attn_ep_moe_dspark.md) (`DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dp_attn_ep_moe_dspark`)
- [DeepSeek V4.1 Flash 1017: vLLM, TP attention, EP MoE, DSpark speculative decoding](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dspark.md) (`DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dspark`)
- [DeepSeek V4.1 Flash: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash/deepseek_v41_vllm.md) (`DeepSeek-V4.1-Flash/deepseek_v41_vllm`)
- [DeepSeek V4.1 Flash: vLLM, TP attention, EP MoE, side streams serialized](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash/deepseek_v41_vllm_serial_streams.md) (`DeepSeek-V4.1-Flash/deepseek_v41_vllm_serial_streams`)
- [GLM-5.2 NVFP4: SGLang, TP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, TP and EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, TP and EP, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative`)
- [GLM-5.2 NVFP4: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_pp_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_pp_dsa_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe`)
- [GLM-5.3-Flash: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe`)
- [GLM-5.3-Flash: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe`)
- [GLM-5.3-Flash: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe`)
- [GLM-5.3 NVFP4: vLLM, TP and EP, DFlash speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2.md) (`GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2`)
- [Inkling NVFP4: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Inkling-NVFP4/inkling_vllm_nvfp4_rel_attn_moe.md) (`Inkling-NVFP4/inkling_vllm_nvfp4_rel_attn_moe`)
- [Inkling Small NVFP4: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Inkling-Small-NVFP4/inkling_vllm_nvfp4_rel_attn_moe.md) (`Inkling-Small-NVFP4/inkling_vllm_nvfp4_rel_attn_moe`)
- [MiMo V2.6 Flash RL: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/MiMo-V2.6-Flash-RL/mimo_v2_vllm_fp8_swa_moe.md) (`MiMo-V2.6-Flash-RL/mimo_v2_vllm_fp8_swa_moe`)
- [Qwen3.8 Flash Next NVFP4: vLLM, one GPU, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative.md) (`Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/nvfp4_fused_moe
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/nvfp4_fused_moe/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/nvfp4_fused_moe/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/nvfp4_fused_moe.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
