# vLLM MLA query RoPE (`vllm_mla_rope`) | ServingStudio Kernel Library

Rotate the MLA query's positional columns and materialize the full query tensor.

In vLLM's MLA attention, RoPE changes only the rope_dim columns of each query head, but the Inductor-compiled kernel reads and writes the whole query, including the qk_nope_head_dim columns it leaves unchanged, because vLLM writes the rotated slice back and then passes the whole query to attention. The measurement compiles the same block for one fixed shape, with sequential positions, and checks it against an eager run before timing.

Category: Attention; group: MLA. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=vllm_mla_rope. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `output query = [q_nope, RoPE(q_rope)]`
- `GB/s = 2 · num_tokens · num_heads · (qk_nope_head_dim + rope_dim) · input bytes / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches with the L2 cache flushed before each. The triton_poi_fused launch is counted when Inductor emits it; for shapes where Inductor declines the fusion, the union of all GPU busy intervals is taken instead.

## Caveats

- GB/s counts one full read and one full write of the query; the rotary-table gather and the second read of the rotary columns are excluded.
- At decode-scale token counts the measurement is slower than serving, where vLLM compiles a variant per CUDA-graph size; trust the prefill range.
- Once a worker process has compiled more shapes than dynamo's cache limit, later shapes run a slower dynamic kernel, so rows filled in large batches can read up to 5.5x slower.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_tokens` | number | tokens | Query token rows transformed together. |
| `num_heads` | number | heads | Query heads on this GPU. |
| `qk_nope_head_dim` | number | elements | Non-rotary columns in each query head. |
| `rope_dim` | number | elements | Rotary columns in each query head. |
| `max_position` | number | positions | Rows in the rotary lookup table. |
| `is_neox_style` | label |  | Whether the rotary dimensions use NeoX pairing. |
| `input_dtype` | dtype |  | Element type of the query and key inputs. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_inductor`: vLLM's native rotary embedding compiled by Inductor into one full-query Triton kernel. dtypes `bf16`, compute capability 8.0 or newer, GPUs A100-SXM4-40GB, A100-SXM4-80GB, A100-PCIe-40GB, A100-PCIe-80GB, H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip, H800-SXM5-80GB, H20-SXM-96GB, A40, L40, L40S, L4.

## Measured rows

272 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `vllm_inductor` | `bf16` | 204 |
| NVIDIA H200 | `vllm_inductor` | `bf16` | 68 |

## Used by these deployments

- [GLM-5.2 FP8: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-FP8/glm52_vllm_dsa_moe.md) (`GLM-5.2-FP8/glm52_vllm_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, TP and EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, TP and EP, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative`)
- [GLM-5.2 NVFP4: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_pp_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_pp_dsa_moe`)
- [GLM-5.2: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2/glm52_vllm_dsa_moe.md) (`GLM-5.2/glm52_vllm_dsa_moe`)
- [GLM-5.3 NVFP4: vLLM, TP and EP, DFlash speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2.md) (`GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/vllm_mla_rope
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/vllm_mla_rope/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/vllm_mla_rope/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/vllm_mla_rope.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
