# MXFP4 Marlin MoE GEMM (`mxfp4_marlin_moe_gemm`) | ServingStudio Kernel Library

Multiply routed BF16 activations by packed MXFP4 expert weights in one Marlin launch.

Routed experts run both expert projections as this Marlin GEMM on MXFP4 weights. The first projection (FC1) reads each token for its six expert selections and does not apply router weights; the second (FC2) reads one row per selection and multiplies by the router weight. The local rows are aligned into block_size_m-row expert blocks before timing.

Category: MoE; group: Expert compute. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=mxfp4_marlin_moe_gemm. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `local_rows = sum(per_group_batches)`
- `active_experts = count(per_group_batches > 0)`
- `TFLOPS = 2·local_rows·n·k / time`
- `logical_bytes = 2·local_rows·k + active_experts·n·k/2 + active_experts·n·⌊k/32⌋ + 2·local_rows·n + (4·local_rows if mul_topk_weights else 0)`
- `GB/s = logical_bytes / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. Five warm-up calls run first; route alignment, weight packing and an output check are not timed.

## Caveats

- TFLOPS counts only routed rows, not work on padded expert blocks. GB/s counts logical BF16 activations and outputs, packed weights and scales, and router weights only when applied.
- Marlin's thread tiles need n % 64 = 0 and k % 128 = 0, or n % 128 = 0 and k % 64 = 0.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `m` | number | tokens | Activation rows entering this expert projection. |
| `n` | number | elements | Output features of this expert projection. |
| `k` | number | elements | Input features of this expert projection. |
| `dtype` | dtype |  | Element type of the activation and output. |
| `input_top_k` | number | experts | Expert selections per activation row in this launch. |
| `block_size_m` | number | tokens | Routed token rows in each aligned expert block. |
| `mul_topk_weights` | label |  | Whether the projection output is multiplied by router weights. |
| `per_group_batches` | list | tokens | Selected token count for each local expert, in expert order. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_marlin`: vLLM moe_wna16_marlin_gemm with packed MXFP4 weights and aligned expert routes. dtypes `bf16`, compute capability 8.0 or newer, GPUs A100-SXM4-40GB, A100-SXM4-80GB, A100-PCIe-40GB, A100-PCIe-80GB, H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip, H800-SXM5-80GB, H20-SXM-96GB, A40, L40, L40S, L4. Source: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/fused_moe/experts/marlin_moe.py

## Measured rows

313 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA H200 | `vllm_marlin` | `bf16` | 313 |

## Used by these deployments

- [DeepSeek V4 Flash 0731: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm.md) (`DeepSeek-V4-Flash-0731/deepseek_v4_vllm`)
- [DeepSeek V4 Flash 0731: vLLM, DP attention, EP, parallel streams serialized](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams.md) (`DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/mxfp4_marlin_moe_gemm
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/mxfp4_marlin_moe_gemm/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/mxfp4_marlin_moe_gemm/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/mxfp4_marlin_moe_gemm.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
