# Grouped GEMM (`grouped_gemm`) | ServingStudio Kernel Library

Multiply routed token batches by their respective MoE expert weights.

A MoE layer's expert projections run as one grouped GEMM: each local expert multiplies its own token rows by its own weight. per_group_batches lists the rows per expert, zeros included, so one total can be spread evenly or skewed across experts. The torch backend runs one torch._grouped_mm over the stacked rows. DeepGEMM pads each nonempty expert's rows to its alignment and runs in fp8 with bf16 output.

Category: MoE; group: Expert compute. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=grouped_gemm. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `C_g[m_g, n] = A_g[m_g, k] · B_g[k, n], with m_g = per_group_batches[g]`
- `TFLOPS = 2·(Σ m_g)·n·k / time`
- `torch GB/s = [(Σ m_g)·(k + n) + E_active·k·n]·bytes per element / time`
- `deepgemm GB/s = [(Σ aligned(m_g))·(k + 2·n) + E_active·k·n] / time`
- `E_active = number of experts with m_g > 0; aligned(0) = 0`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. DeepGEMM runs once before timing; the fp8 casts, padding and expert-index construction are not timed.

## Caveats

- GB/s counts only the weights of experts that have tokens.
- For deepgemm, GB/s counts the padded rows and TFLOPS only the real ones.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `n` | number | elements | Output features in each expert weight. |
| `k` | number | elements | Input features in each expert weight. |
| `dtype` | dtype |  | Element type of the multiply; fp8_e4m3 writes bf16 output. |
| `num_local_experts` | number | experts | Experts assigned to this GPU. |
| `per_group_batches` | list | tokens | Token rows assigned to each local expert, in expert order. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `torch`: torch._grouped_mm with cumulative token offsets and one weight matrix per expert. dtypes `bf16`, `fp16`, any CUDA GPU.
- `deepgemm`: DeepGEMM m_grouped_fp8_gemm_nt_contiguous with per-token activation scales, per-block weight scales, and bf16 output. dtypes `fp8_e4m3`, any CUDA GPU.

## Measured rows

9,652 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA H200 | `deepgemm` | `fp8_e4m3` | 3,806 |
| NVIDIA H200 | `torch` | `bf16` | 5,846 |

## Used by these deployments

- [GLM-5.2: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2/glm52_vllm_dsa_moe.md) (`GLM-5.2/glm52_vllm_dsa_moe`)
- [Qwen3 235B-A22B FP8: FFN side of an attention–FFN split](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_fp8_ffn_moe.md) (`Qwen3-235B-A22B-FP8/qwen3_fp8_ffn_moe`)
- [Qwen3 235B-A22B FP8: DP attention, EP MoE, DeepGEMM experts](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_moe_fp8_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-FP8/qwen3_moe_fp8_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B Thinking 2507 FP8: DP attention, EP MoE, DeepGEMM experts](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_moe_fp8_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_moe_fp8_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B: FFN side of an attention–FFN split](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B/qwen3_ffn_moe.md) (`Qwen3-235B-A22B/qwen3_ffn_moe`)
- [Qwen3 235B-A22B: DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B/qwen3_moe_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B/qwen3_moe_dp_attn_ep_ffn`)
- [Qwen3 Coder 480B-A35B: FFN side of an attention–FFN split](https://servingstudio.cs.washington.edu/models/Qwen3-Coder-480B-A35B-Instruct/qwen3_fp8_ffn_moe.md) (`Qwen3-Coder-480B-A35B-Instruct/qwen3_fp8_ffn_moe`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/grouped_gemm
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/grouped_gemm/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/grouped_gemm/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/grouped_gemm.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
