# Dense GEMM (`single_gemm`) | ServingStudio Kernel Library

One matrix multiply: an activation block of m tokens times a weight matrix.

Every linear layer in a transformer is a dense GEMM: QKV and output projections, dense MLPs, the LM head. m is the number of tokens in the batch; n and k come from the weight shape. Backends that share this table differ in how the weight is laid out and which library dispatches the multiply, and the simulator picks the fastest measured backend per shape.

Category: GEMM. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=single_gemm. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `C[m, n] = A[m, k] · B[k, n]`
- `TFLOPS = 2·m·n·k / time`
- `GB/s = (m·k + k·n + m·n) · bytes per element / time`
- `deepgemm, flashinfer_fp8_tensor (fp8 in, bf16 out): GB/s = (m·k + k·n + 2·m·n) / time`
- `flashinfer_cutedsl (nvfp4 in, bf16 out): GB/s = ((m·k + k·n)·(1/2 + 1/16) + 2·m·n) / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. The torch and SGLang backends count overlapping launches once, by the time the GPU is busy, because Blackwell can split one logical GEMM into several; deepgemm, flashinfer_fp8_tensor, flashinfer_cutedsl and flashinfer_mxfp8 sum their launches. flashinfer_cutedsl times only the GEMM: the activation quantization before it is the nvfp4_quant kind. flashinfer_mxfp8 includes its activation quantization and autotunes the call once before timing; flashinfer_mxfp8_prequantized passes a QuantizedActivation and times the GEMM only. flashinfer_fp8_tensor times only the GEMM on already-quantized fp8 input, after autotuning FlashInfer's hybrid token buckets as vLLM's warmup does.

## Caveats

- Inputs are random normal tensors, so data-dependent effects such as sparsity are not measured.
- deepgemm scales fp8_e4m3 operands per 128-element block; flashinfer_fp8_tensor scales each operand by one FP32 scalar.
- torch and torch_linear compute the same product. They are separate rows because the weight layout changes which cuBLAS kernel runs.
- flashinfer_mxfp8 takes bf16 activations and includes their MXFP8 quantization in the time; its GB/s adds the bf16 read and the quantized write of the activation. flashinfer_mxfp8_prequantized is the same call on an activation its producer already quantized, so it times the GEMM alone.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `m` | number | tokens | Rows of the activation: tokens in the batch. |
| `n` | number | elements | Output features of the weight. |
| `k` | number | elements | Input features, the reduction dimension. |
| `dtype` | dtype |  | Element type of A and B. fp8_e4m3, nvfp4_e2m1 and mxfp8_e4m3 write bf16 output. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `torch`: torch.mm on a contiguous right-hand side. dtypes `bf16`, `fp16`, any CUDA GPU. Source: https://pytorch.org/docs/stable/generated/torch.mm.html
- `torch_linear_vllm`: The same F.linear call, run in vLLM's pinned environment. dtypes `bf16`, any CUDA GPU. Source: https://github.com/vllm-project/vllm
- `torch_linear`: F.linear with the (n, k) weight layout used by vLLM and Transformers. dtypes `bf16`, `fp16`, any CUDA GPU. Source: https://pytorch.org/docs/stable/generated/torch.nn.functional.linear.html
- `sglang_bf16_auto`: SGLang's BF16 dispatch with bf16_gemm_backend='auto': cutedsl_bf16_gemm when use_cutedsl_bf16_gemm(m, n, k) accepts the shape, otherwise F.linear. SM100 only. dtypes `bf16`, SM targets `sm_100f`, GPUs B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip. Source: https://github.com/sgl-project/sglang
- `sglang_fused_a_auto`: SGLang's full fused-A dispatch: dsv3_fused_a_gemm when 1 ≤ m ≤ 16, n % 16 == 0 and k % 256 == 0; otherwise cutedsl_bf16_gemm when use_cutedsl_bf16_gemm(m, n, k) accepts the shape; otherwise F.linear. SM100 only. dtypes `bf16`, SM targets `sm_100f`, GPUs B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip. Source: https://github.com/sgl-project/sglang
- `deepgemm`: DeepGEMM fp8_gemm_nt: fp8 A with one scale per token per 128 k-elements, fp8 B with one scale per 128×128 block, bf16 output. On Blackwell the scales are UE8M0, packed into DeepGEMM's layout before timing, as vLLM passes them. dtypes `fp8_e4m3`, any CUDA GPU. Source: https://github.com/deepseek-ai/DeepGEMM
- `flashinfer_fp8_tensor`: vLLM's FlashInferFP8ScaledMMLinearKernel after its activation quantization: flashinfer bmm_fp8(backend='auto') on fp8 A and the column-major fp8 weight, one FP32 scale each, bf16 output, tactic autotuned per hybrid token bucket. SM10x only. dtypes `fp8_e4m3`, SM targets `sm_100f`, GPUs B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip. Source: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/kernels/linear/scaled_mm/flashinfer.py
- `flashinfer_cutedsl`: vLLM's FlashInferCuteDslNvFp4LinearKernel after its activation quantization: flashinfer mm_fp4(backend='cute-dsl') on packed FP4 A and B with swizzled E4M3 scales per 16 k-elements, bf16 output, heuristic tactic (vLLM skips fp4_gemm autotuning). SM10x only. dtypes `nvfp4_e2m1`, SM targets `sm_100f`, GPUs B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip. Source: https://github.com/flashinfer-ai/flashinfer
- `flashinfer_mxfp8`: vLLM's FlashInferCutedslMxfp8LinearKernel: FlashInfer MXFP8 quantization of the bf16 activation (UE8M0 per 32, swizzled) and the CuTe-DSL block-scaled GEMM mm_mxfp8, bf16 output. dtypes `mxfp8_e4m3`, SM targets `sm_100f`, GPUs B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip. Source: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/kernels/linear/mxfp8/flashinfer.py
- `flashinfer_mxfp8_prequantized`: vLLM's FlashInferCutedslMxfp8LinearKernel on an already-quantized MXFP8 activation (QuantizedActivation): the CuTe-DSL block-scaled GEMM mm_mxfp8 alone, bf16 output. dtypes `mxfp8_e4m3`, SM targets `sm_100f`, GPUs B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip. Source: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/kernels/linear/mxfp8/flashinfer.py

## Measured rows

16,451 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `deepgemm` | `fp8_e4m3` | 1,020 |
| NVIDIA B200 | `flashinfer_cutedsl` | `nvfp4_e2m1` | 612 |
| NVIDIA B200 | `flashinfer_fp8_tensor` | `fp8_e4m3` | 459 |
| NVIDIA B200 | `flashinfer_mxfp8` | `mxfp8_e4m3` | 754 |
| NVIDIA B200 | `flashinfer_mxfp8_prequantized` | `mxfp8_e4m3` | 290 |
| NVIDIA B200 | `sglang_bf16_auto` | `bf16` | 816 |
| NVIDIA B200 | `sglang_fused_a_auto` | `bf16` | 204 |
| NVIDIA B200 | `torch_linear` | `bf16` | 1,904 |
| NVIDIA B200 | `torch_linear_vllm` | `bf16` | 4,012 |
| NVIDIA H200 | `deepgemm` | `fp8_e4m3` | 2,312 |
| NVIDIA H200 | `torch` | `bf16` | 1,619 |
| NVIDIA H200 | `torch` | `fp16` | 1 |
| NVIDIA H200 | `torch_linear` | `bf16` | 2,448 |

## Used by these deployments

- [DeepSeek V4 Flash 0731: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm.md) (`DeepSeek-V4-Flash-0731/deepseek_v4_vllm`)
- [DeepSeek V4 Flash 0731: vLLM, DP attention, EP, parallel streams serialized](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams.md) (`DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams`)
- [DeepSeek V4.1 Flash 1017: vLLM, DP attention, EP MoE, DSpark speculative decoding](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dp_attn_ep_moe_dspark.md) (`DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dp_attn_ep_moe_dspark`)
- [DeepSeek V4.1 Flash 1017: vLLM, TP attention, EP MoE, DSpark speculative decoding](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dspark.md) (`DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dspark`)
- [DeepSeek V4.1 Flash: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash/deepseek_v41_vllm.md) (`DeepSeek-V4.1-Flash/deepseek_v41_vllm`)
- [DeepSeek V4.1 Flash: vLLM, TP attention, EP MoE, side streams serialized](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash/deepseek_v41_vllm_serial_streams.md) (`DeepSeek-V4.1-Flash/deepseek_v41_vllm_serial_streams`)
- [GLM-5.2 FP8: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-FP8/glm52_vllm_dsa_moe.md) (`GLM-5.2-FP8/glm52_vllm_dsa_moe`)
- [GLM-5.2 NVFP4: SGLang, TP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, TP and EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, TP and EP, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative`)
- [GLM-5.2 NVFP4: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_pp_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_pp_dsa_moe`)
- [GLM-5.2: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2/glm52_vllm_dsa_moe.md) (`GLM-5.2/glm52_vllm_dsa_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe`)
- [GLM-5.3-Flash: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe`)
- [GLM-5.3-Flash: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe`)
- [GLM-5.3-Flash: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe`)
- [GLM-5.3 NVFP4: vLLM, TP and EP, DFlash speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2.md) (`GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2`)
- [Inkling NVFP4: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Inkling-NVFP4/inkling_vllm_nvfp4_rel_attn_moe.md) (`Inkling-NVFP4/inkling_vllm_nvfp4_rel_attn_moe`)
- [Inkling Small NVFP4: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Inkling-Small-NVFP4/inkling_vllm_nvfp4_rel_attn_moe.md) (`Inkling-Small-NVFP4/inkling_vllm_nvfp4_rel_attn_moe`)
- [Llama 3.1 8B: Single GPU](https://servingstudio.cs.washington.edu/models/Llama-3.1-8B/llama3_dense.md) (`Llama-3.1-8B/llama3_dense`)
- [Llama 3.1 8B: TP](https://servingstudio.cs.washington.edu/models/Llama-3.1-8B/llama3_dense_tp.md) (`Llama-3.1-8B/llama3_dense_tp`)
- [Llama 3.1 8B: DP attention, TP FFN](https://servingstudio.cs.washington.edu/models/Llama-3.1-8B/llama3_dp_attn_tp_ffn.md) (`Llama-3.1-8B/llama3_dp_attn_tp_ffn`)
- [MiMo V2.6 Flash RL: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/MiMo-V2.6-Flash-RL/mimo_v2_vllm_fp8_swa_moe.md) (`MiMo-V2.6-Flash-RL/mimo_v2_vllm_fp8_swa_moe`)
- [Qwen3 235B-A22B FP8: FFN side of an attention–FFN split](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_fp8_ffn_moe.md) (`Qwen3-235B-A22B-FP8/qwen3_fp8_ffn_moe`)
- [Qwen3 235B-A22B FP8: DP attention, EP MoE, DeepGEMM experts](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_moe_fp8_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-FP8/qwen3_moe_fp8_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B FP8: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_vllm_moe_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-FP8/qwen3_vllm_moe_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B Thinking 2507 FP8: DP attention, EP MoE, DeepGEMM experts](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_moe_fp8_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_moe_fp8_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B Thinking 2507 FP8: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_vllm_moe_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_vllm_moe_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B: FFN side of an attention–FFN split](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B/qwen3_ffn_moe.md) (`Qwen3-235B-A22B/qwen3_ffn_moe`)
- [Qwen3 235B-A22B: DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B/qwen3_moe_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B/qwen3_moe_dp_attn_ep_ffn`)
- [Qwen3 Coder 480B-A35B: FFN side of an attention–FFN split](https://servingstudio.cs.washington.edu/models/Qwen3-Coder-480B-A35B-Instruct/qwen3_fp8_ffn_moe.md) (`Qwen3-Coder-480B-A35B-Instruct/qwen3_fp8_ffn_moe`)
- [Qwen3.6 35B-A3B FP8: Single GPU](https://servingstudio.cs.washington.edu/models/Qwen3.6-35B-A3B-FP8/qwen36_local.md) (`Qwen3.6-35B-A3B-FP8/qwen36_local`)
- [Qwen3.8 27B NVFP4: vLLM, one GPU](https://servingstudio.cs.washington.edu/models/Qwen3.8-27B-NVFP4/qwen35_vllm_nvfp4_gdn_dense.md) (`Qwen3.8-27B-NVFP4/qwen35_vllm_nvfp4_gdn_dense`)
- [Qwen3.8 Flash Next NVFP4: vLLM, one GPU, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative.md) (`Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/single_gemm
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/single_gemm/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/single_gemm/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/single_gemm.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
