# FP32-output GEMM (`gemm_fp32_output`) | ServingStudio Kernel Library

Multiply bf16 or fp32 activations and weights while writing an fp32 result.

Some call sites need an fp32 result. vLLM computes attention KV-compressor projections and MoE router logits from bf16 inputs with torch.mm and out_dtype=float32, and the DSA indexer's head weights from fp32 inputs with a plain torch.mm; both are measured by torch_cublas. SGLang's MoE router computes router logits in fp32, measured by sglang_router_auto. The alignment vLLM fork's _linear_with_fp32_out, which picks cute-DSL ll_bf16 or cuBLAS by m, is measured by vllm_ll_bf16_auto. m is the tokens; n and k are the output and input features.

Category: GEMM. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=gemm_fp32_output. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `C[m, n] = A[m, k] · B[n, k]ᵀ, with fp32 output`
- `TFLOPS = 2·m·n·k / time`
- `GB/s = [bytes(input_dtype)·(m·k + n·k) + 4·m·n] / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. torch_cublas counts every launch of the call, one kernel or a split-K pair. sglang_router_auto and vllm_ll_bf16_auto count overlapping launches once, by the time the GPU is busy.

## Caveats

- The fp32 form runs at float32 matmul precision highest; the cast of the activation to fp32 is a separate launch, not timed here.
- Outside its dedicated shapes, sglang_router_auto calls linear_bf16_fp32, whose kernel (cuBLAS, HPC-Ops or DeepGEMM) is chosen by SGLANG_OPT_BF16_FP32_GEMM_ALGO in the profiling environment.
- vllm_ll_bf16_auto switches kernels at m = 64 (cute-DSL ll_bf16 at or below, cuBLAS above); within ll_bf16 it runs the dot-product kernel when m ≤ 4 or k \< 2048 and split-K otherwise, so its time is not smooth across those points.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `m` | number | tokens | Token rows in the activation. |
| `n` | number | elements | Output features in the weight. |
| `k` | number | elements | Input features, the reduction dimension. |
| `input_dtype` | dtype |  | Element type of the activation and weight; output is fp32. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `torch_cublas`: torch.mm with out_dtype=torch.float32 on bf16 inputs, as vLLM calls it for KV-compressor projections and router logits, or plain torch.mm on fp32 inputs. dtypes `bf16`, `fp32`, any CUDA GPU. Source: https://github.com/vllm-project/vllm/blob/main/vllm/models/deepseek_v4/attention.py
- `sglang_router_auto`: SGLang's router dispatch (MoEGate.forward): tiny_gemm_bf16 when m is at most tiny_router_gemm_max_tokens (16 on SM90+ for a BF16 weight can_use_tiny_gemm accepts, else never); otherwise linear_bf16_fp32. dtypes `bf16`, any CUDA GPU. Source: https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/deepseek_v2.py
- `vllm_ll_bf16_auto`: The alignment fork's _linear_with_fp32_out dispatch: cute-DSL ll_bf16_gemm when m ≤ 64 on SM90+ (dot-product kernel at m ≤ 4 or k \< 2048, else split-K), otherwise torch.mm with out_dtype=float32. dtypes `bf16`, any CUDA GPU.

## Measured rows

1,048 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `sglang_router_auto` | `bf16` | 68 |
| NVIDIA B200 | `torch_cublas` | `bf16` | 476 |
| NVIDIA B200 | `torch_cublas` | `fp32` | 68 |
| NVIDIA B200 | `vllm_ll_bf16_auto` | `bf16` | 164 |
| NVIDIA H200 | `torch_cublas` | `bf16` | 272 |

## Used by these deployments

- [DeepSeek V4 Flash 0731: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm.md) (`DeepSeek-V4-Flash-0731/deepseek_v4_vllm`)
- [DeepSeek V4 Flash 0731: vLLM, DP attention, EP, parallel streams serialized](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams.md) (`DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams`)
- [DeepSeek V4.1 Flash 1017: vLLM, DP attention, EP MoE, DSpark speculative decoding](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dp_attn_ep_moe_dspark.md) (`DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dp_attn_ep_moe_dspark`)
- [DeepSeek V4.1 Flash 1017: vLLM, TP attention, EP MoE, DSpark speculative decoding](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dspark.md) (`DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dspark`)
- [DeepSeek V4.1 Flash: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash/deepseek_v41_vllm.md) (`DeepSeek-V4.1-Flash/deepseek_v41_vllm`)
- [DeepSeek V4.1 Flash: vLLM, TP attention, EP MoE, side streams serialized](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash/deepseek_v41_vllm_serial_streams.md) (`DeepSeek-V4.1-Flash/deepseek_v41_vllm_serial_streams`)
- [GLM-5.2 NVFP4: SGLang, TP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe`)
- [GLM-5.3-Flash: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe`)
- [GLM-5.3-Flash: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe`)
- [GLM-5.3-Flash: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe`)
- [Inkling NVFP4: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Inkling-NVFP4/inkling_vllm_nvfp4_rel_attn_moe.md) (`Inkling-NVFP4/inkling_vllm_nvfp4_rel_attn_moe`)
- [Inkling Small NVFP4: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Inkling-Small-NVFP4/inkling_vllm_nvfp4_rel_attn_moe.md) (`Inkling-Small-NVFP4/inkling_vllm_nvfp4_rel_attn_moe`)
- [MiMo V2.6 Flash RL: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/MiMo-V2.6-Flash-RL/mimo_v2_vllm_fp8_swa_moe.md) (`MiMo-V2.6-Flash-RL/mimo_v2_vllm_fp8_swa_moe`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/gemm_fp32_output
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/gemm_fp32_output/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/gemm_fp32_output/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/gemm_fp32_output.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
