# vLLM fused MoE GEMM (`vllm_fused_moe`) | ServingStudio Kernel Library

Gather selected token rows and multiply them by FP8 expert weights in one Triton launch.

A routed MoE layer runs its expert projections as two launches of vLLM's Triton fused_moe_kernel with FP8 block scales. The gate-up launch (w13) reads one activation row per token; the down launch (w2) reads one row per token-expert pair and multiplies by the router weight. Each measured launch uses the tile sizes from vLLM's tuned tables and the expert alignment vLLM would choose for the given per-expert counts.

Category: MoE; group: Expert compute. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=vllm_fused_moe. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `rows = sum(per_group_batches)`
- `active_experts = count(per_group_batches > 0)`
- `TFLOPS = 2·rows·n·k / time`
- `logical_bytes = rows·k + 4·rows·⌈k/block_size⌉ + active_experts·n·k + 4·active_experts·⌈n/block_size⌉·⌈k/block_size⌉ + 2·rows·n`
- `GB/s = logical_bytes / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. Only the Triton launch is timed; expert alignment and one compiling call run before it.

## Caveats

- TFLOPS and GB/s count routed rows and active experts without the padded tile work performed by the kernel.
- Routing, alignment, activation quantization and final expert reduction are outside the timed launch.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `n` | number | elements | Output features of this expert projection. |
| `k` | number | elements | Input features of this expert projection. |
| `dtype` | dtype |  | Element type of the FP8 activation and weight. |
| `num_local_experts` | number | experts | Experts whose weights reside on this GPU. |
| `num_tokens` | number | tokens | Tokens entering the routed MoE layer. |
| `experts_per_token` | number | experts | Experts selected for each token. |
| `launch_role` | label |  | Projection role: w13 for gate/up or w2 for down. |
| `block_size` | number | elements | FP8 activation and weight scale granularity. |
| `per_group_batches` | list | tokens | Selected token count for each local expert, in expert order. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_triton`: vLLM invoke_fused_moe_triton_kernel with FP8 block scales and its tuned expert alignment. dtypes `fp8_e4m3`, compute capability 8.9 or newer, GPUs H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip, H800-SXM5-80GB, H20-SXM-96GB, L40, L40S, L4. Source: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/fused_moe/fused_moe.py

## Measured rows

194 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA H200 | `vllm_triton` | `fp8_e4m3` | 194 |

## Used by these deployments

- [Qwen3.6 35B-A3B FP8: Single GPU](https://servingstudio.cs.washington.edu/models/Qwen3.6-35B-A3B-FP8/qwen36_local.md) (`Qwen3.6-35B-A3B-FP8/qwen36_local`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/vllm_fused_moe
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/vllm_fused_moe/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/vllm_fused_moe/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/vllm_fused_moe.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
