# Batched GEMM (`batched_gemm`) | ServingStudio Kernel Library

Multiply one activation and weight matrix pair per attention head (MLA) or per head group (grouped output projection).

vLLM's MLA runs two per-head batched multiplies. Query absorption multiplies each head's query by W_UK, mapping it into the compressed KV space; value expansion multiplies each head's attention output by W_UV, mapping it back to the value head. num_batches is the heads on this GPU and m the tokens. Each backend builds the same strided views of the packed weight, activation and output that vLLM passes to torch.bmm. A grouped low-rank attention output projection is the same shape of work: each group of 8 heads' 4,096-wide attention output is multiplied by that group's weight, as one DeepGEMM einsum on MXFP8 operands.

Category: GEMM. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=batched_gemm. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `C[b, m, n] = A[b, m, k] · B[b, k, n]`
- `TFLOPS = 2·num_batches·m·n·k / time`
- `GB/s = num_batches·(m·k + k·n + m·n)·bytes per element / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. Each MLA backend times only its torch.bmm call; operand construction and any head-padding copy are outside the timed call. The grouped output projection times its one DeepGEMM launch after three warm-up calls.

## Caveats

- GB/s counts the logical operand elements, not the gaps in the packed storage the strided views skip.
- Each backend fixes k and n to its layout: k = 192 or 256, n = 512 for query absorption and k = 512, n = 256 for value expansion.
- torch_mla_v_up reads an attention output padded to 64 heads, so it takes at most 64 heads.
- The grouped output projection's activation is the FP8 attention output with UE8M0 scales, stored in 8 padded group slots of which the call reads the first num_batches. Its GB/s counts one byte per input element plus one scale byte per 32, and two bytes per output element.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_batches` | number | heads | Attention heads (or, for the grouped output projection, head groups) multiplied independently on this GPU. |
| `m` | number | tokens | Token rows per attention head. |
| `n` | number | elements | Output features per attention head. |
| `k` | number | elements | Input features per attention head. |
| `dtype` | dtype |  | Element type of both inputs: bf16 for the MLA backends, with bf16 output; mxfp8_e4m3 for the grouped output projection, with bf16 output. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `torch_mla_q_absorb`: torch.bmm for query absorption, on vLLM's packed W_UK view. dtypes `bf16`, any CUDA GPU. Source: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/attention/mla_attention.py
- `torch_mla_v_up`: torch.bmm for value expansion, on vLLM's packed W_UV view and strided output. dtypes `bf16`, any CUDA GPU. Source: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/attention/mla_attention.py
- `torch_mla_q_absorb_no_rope`: torch.bmm for query absorption when the query head has no RoPE part, on vLLM's packed W_UK view. dtypes `bf16`, any CUDA GPU. Source: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/attention/mla_attention.py
- `torch_mla_v_up_unpadded`: torch.bmm for value expansion from an attention output with no head padding, on vLLM's packed W_UV view. dtypes `bf16`, any CUDA GPU. Source: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/attention/mla_attention.py
- `deepgemm_mxfp8_einsum_grouped_o_proj`: DeepGEMM fp8_einsum("bhr,hdr-\>bhd") with a (1, 1, 32) MXFP8 recipe, as vLLM's DeepGemmMxfp8BmmLinearKernel applies a grouped output projection to the fused attention's FP8 output. dtypes `mxfp8_e4m3`, SM targets `sm_100f`, GPUs B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip. Source: https://github.com/deepseek-ai/DeepGEMM

## Measured rows

1,360 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `deepgemm_mxfp8_einsum_grouped_o_proj` | `mxfp8_e4m3` | 272 |
| NVIDIA B200 | `torch_mla_q_absorb` | `bf16` | 204 |
| NVIDIA B200 | `torch_mla_q_absorb_no_rope` | `bf16` | 204 |
| NVIDIA B200 | `torch_mla_v_up` | `bf16` | 204 |
| NVIDIA B200 | `torch_mla_v_up_unpadded` | `bf16` | 204 |
| NVIDIA H200 | `torch_mla_q_absorb` | `bf16` | 136 |
| NVIDIA H200 | `torch_mla_v_up` | `bf16` | 136 |

## Used by these deployments

- [DeepSeek V4.1 Flash 1017: vLLM, DP attention, EP MoE, DSpark speculative decoding](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dp_attn_ep_moe_dspark.md) (`DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dp_attn_ep_moe_dspark`)
- [DeepSeek V4.1 Flash 1017: vLLM, TP attention, EP MoE, DSpark speculative decoding](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dspark.md) (`DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dspark`)
- [DeepSeek V4.1 Flash: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash/deepseek_v41_vllm.md) (`DeepSeek-V4.1-Flash/deepseek_v41_vllm`)
- [DeepSeek V4.1 Flash: vLLM, TP attention, EP MoE, side streams serialized](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash/deepseek_v41_vllm_serial_streams.md) (`DeepSeek-V4.1-Flash/deepseek_v41_vllm_serial_streams`)
- [GLM-5.2 FP8: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-FP8/glm52_vllm_dsa_moe.md) (`GLM-5.2-FP8/glm52_vllm_dsa_moe`)
- [GLM-5.2 NVFP4: SGLang, TP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, TP and EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, TP and EP, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative`)
- [GLM-5.2 NVFP4: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_pp_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_pp_dsa_moe`)
- [GLM-5.2: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2/glm52_vllm_dsa_moe.md) (`GLM-5.2/glm52_vllm_dsa_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe`)
- [GLM-5.3-Flash: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe`)
- [GLM-5.3-Flash: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe`)
- [GLM-5.3-Flash: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe`)
- [GLM-5.3 NVFP4: vLLM, TP and EP, DFlash speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2.md) (`GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/batched_gemm
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/batched_gemm/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/batched_gemm/configs

## Torch reference

What the kind computes, as Torch code (`profiling/runners/gemm/batched_gemm_reference.py` in ServingStudio Sim):

```python
"""Layout-agnostic Torch semantic reference for batched matrix multiplication."""

from __future__ import annotations

import torch

_SUPPORTED_DTYPES = (torch.bfloat16, torch.float16, torch.float32)


def batched_gemm_reference(
    lhs: torch.Tensor,
    rhs: torch.Tensor,
    out: torch.Tensor | None = None,
) -> torch.Tensor:
    """Compute ``[B, M, K] @ [B, K, N]`` with FP32 accumulation.

    The optional output may be an arbitrary non-overlapping strided view. When
    supplied, it is updated and returned without mutating either input.
    """
    # Source boundary: vLLM v0.23.0 commit 0fc695fc6d1d82e9a5ac6835ac8e4e1c83703665
    # uses BMM for MLA Q absorption (H,M,192)x(H,192,512) and V-up
    # (H,M,512)x(H,512,256). Profiling backends construct those GLM layouts.
    _validate_inputs(lhs, rhs)
    expected_shape = (lhs.shape[0], lhs.shape[1], rhs.shape[2])
    if out is not None:
        _validate_out(out, lhs, rhs, expected_shape)

    result_fp32 = torch.einsum(
        "bmk,bkn->bmn",
        lhs.to(dtype=torch.float32),
        rhs.to(dtype=torch.float32),
    )
    result = result_fp32.to(dtype=lhs.dtype)
    if out is None:
        return result

    try:
        out.copy_(result)
    except RuntimeError as exc:
        raise ValueError("out must be writable") from exc
    return out


def _validate_inputs(lhs: object, rhs: object) -> None:
    if not isinstance(lhs, torch.Tensor):
        raise TypeError("lhs must be a torch.Tensor")
    if not isinstance(rhs, torch.Tensor):
        raise TypeError("rhs must be a torch.Tensor")

    if lhs.ndim != 3:
        raise ValueError(f"lhs must be rank 3, got rank {lhs.ndim}")
    if rhs.ndim != 3:
        raise ValueError(f"rhs must be rank 3, got rank {rhs.ndim}")
    if any(dim <= 0 for dim in lhs.shape):
        raise ValueError(f"lhs dimensions must be positive, got {tuple(lhs.shape)}")
    if any(dim <= 0 for dim in rhs.shape):
        raise ValueError(f"rhs dimensions must be positive, got {tuple(rhs.shape)}")

    if lhs.shape[0] != rhs.shape[0]:
        raise ValueError(
            f"lhs and rhs must have the same batch dimension, got {lhs.shape[0]} "
            f"and {rhs.shape[0]}"
        )
    if lhs.shape[2] != rhs.shape[1]:
        raise ValueError(
            f"lhs and rhs inner K dimensions must match, got {lhs.shape[2]} "
            f"and {rhs.shape[1]}"
        )

    if lhs.dtype not in _SUPPORTED_DTYPES:
        raise TypeError(
            "lhs dtype must be torch.bfloat16, torch.float16, or torch.float32, "
            f"got {lhs.dtype}"
        )
    if rhs.dtype != lhs.dtype:
        raise TypeError(
            f"lhs and rhs must have the same dtype, got {lhs.dtype} and {rhs.dtype}"
        )
    if rhs.device != lhs.device:
        raise ValueError(
            f"lhs and rhs must be on the same device, got {lhs.device} and {rhs.device}"
        )
    if lhs.layout is not torch.strided or rhs.layout is not torch.strided:
        raise ValueError("lhs and rhs must have torch.strided layout")


def _validate_out(
    out: object,
    lhs: torch.Tensor,
    rhs: torch.Tensor,
    expected_shape: tuple[int, int, int],
) -> None:
    if not isinstance(out, torch.Tensor):
        raise TypeError("out must be a torch.Tensor")
    if out.ndim != 3:
        raise ValueError(f"out must be rank 3, got rank {out.ndim}")
    if tuple(out.shape) != expected_shape:
        raise ValueError(f"out must have shape {expected_shape}, got {tuple(out.shape)}")
    if out.dtype != lhs.dtype:
        raise TypeError(f"out dtype must match input dtype {lhs.dtype}, got {out.dtype}")
    if out.device != lhs.device:
        raise ValueError(
            f"out must be on the same device as inputs, got {out.device} and {lhs.device}"
        )
    if out.layout is not torch.strided:
        raise ValueError("out must have torch.strided layout")

    # Status 1 is definite overlap; status 2 means Torch could not prove either
    # outcome and must remain accepted for valid arbitrary-strided views.
    if int(torch._debug_has_internal_overlap(out)) == 1:
        raise ValueError("out must not have internal overlap")
    if torch._C._overlaps(out, lhs) or torch._C._overlaps(out, rhs):
        raise ValueError("out must not alias lhs or rhs")
```

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/batched_gemm.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
