# FP8 block-scale grouped GEMM (`fp8_blockscale_grouped_gemm`) | ServingStudio Kernel Library

Multiply routed fp8 token rows by block-scaled expert weights and write bf16 output.

TensorRT-LLM's fused MoE, as vendored by FlashInfer, runs the routed experts' gate-up projection as this grouped GEMM. per_group_batches lists the local rows per expert. num_input_tokens and experts_per_token matter too: the kernel recipe is chosen from the global input-token count, and buffers are sized for num_input_tokens · experts_per_token routed rows before expert parallelism decides which rows are local.

Category: MoE; group: Expert compute. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=fp8_blockscale_grouped_gemm. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `C_g[m_g, n] = A_g[m_g, k] · B_g[k, n], with m_g = per_group_batches[g]`
- `M = Σ m_g; E_active = number of experts with m_g > 0`
- `TFLOPS = 2·M·n·k / time`
- `GB/s = [M·k + 4·M·(k/128) + E_active·n·k + 4·E_active·(n/128)·(k/128) + 2·M·n] / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. One untimed launch and a check of the recipe's kernel names run first. Only the GroupedWithOffset GEMM kernel for the shape is counted.

## Caveats

- The fp8 activation and weight buffers are uninitialized and every scale is 1.
- GB/s counts the real local rows and the weights of experts that have rows, not the allocated capacity.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `n` | number | elements | Output features in each expert weight. |
| `k` | number | elements | Input features in each expert weight. |
| `dtype` | dtype |  | Element type of activations and weights; output is bf16. |
| `num_local_experts` | number | experts | Experts assigned to this GPU. |
| `num_input_tokens` | number | tokens | Input tokens before top-k expert selection. |
| `experts_per_token` | number | experts | Experts selected for each input token. |
| `per_group_batches` | list | tokens | Routed token rows assigned to each local expert, in expert order. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `flashinfer_trtllm`: The TensorRT-LLM grouped_gemm_dispatch vendored in FlashInfer, called through a local binding without the rest of the MoE. dtypes `fp8_e4m3`, SM targets `sm_90a`, GPUs H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, H800-SXM5-80GB, H20-SXM-96GB.

## Measured rows

3,678 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA H200 | `flashinfer_trtllm` | `fp8_e4m3` | 3,678 |

## Used by these deployments

- [GLM-5.2 FP8: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-FP8/glm52_vllm_dsa_moe.md) (`GLM-5.2-FP8/glm52_vllm_dsa_moe`)
- [Qwen3 235B-A22B FP8: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_vllm_moe_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-FP8/qwen3_vllm_moe_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B Thinking 2507 FP8: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_vllm_moe_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_vllm_moe_dp_attn_ep_ffn`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/fp8_blockscale_grouped_gemm
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/fp8_blockscale_grouped_gemm/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/fp8_blockscale_grouped_gemm/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/fp8_blockscale_grouped_gemm.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
