# MLA RoPE and FP8 quantization (`mla_rope_quantize_fp8`) | ServingStudio Kernel Library

Rotate MLA queries and keys, quantize them to FP8, and assemble each query.

On SGLang's MLA path, one FlashInfer call applies RoPE to the rope_dim columns of each query head and of the shared key, and quantizes both rotary and non-rotary parts to FP8. The query's non-rotary part is already absorbed into the latent space, so it is kv_lora_rank wide. Query parts go into slices of one FP8 tensor; key parts go to separate outputs. Positions are random rows of the max_position-row lookup table.

Category: Attention; group: MLA. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=mla_rope_quantize_fp8. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `query output = FP8([q_nope, RoPE(q_rope)]); key outputs = FP8(k_nope), FP8(RoPE(k_rope))`
- `TFLOPS = num_tokens · (num_heads + 1) · rope_dim · 3 / time`
- `GB/s = num_tokens · ((num_heads + 1) · (kv_lora_rank + rope_dim) · (input bytes + output bytes) + rope_dim · 4 + 8) / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. One call and a synchronization run first, then five warm-up calls; every launch of the FlashInfer call is counted.

## Caveats

- The cosine and sine table holds random values, not real RoPE angles; the access pattern is the same.
- TFLOPS counts three operations per rotary element as a nominal rate.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_tokens` | number | tokens | Query and key token rows to transform. |
| `num_heads` | number | heads | Query heads on this GPU. |
| `kv_lora_rank` | number | elements | Width of the non-rotary latent key and query part. |
| `rope_dim` | number | elements | Width of the rotary query and key part. |
| `max_position` | number | positions | Rows in the cosine and sine lookup table. |
| `is_neox_style` | label |  | Whether the rotary dimensions use NeoX pairing. |
| `input_dtype` | dtype |  | Element type of the input query and key parts. |
| `quant_dtype` | dtype |  | Element type of the transformed outputs. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `flashinfer`: FlashInfer mla_rope_quantize_fp8 with pre-split MLA inputs and FP8 E4M3 outputs. dtypes `bf16`, KV dtypes `fp8_e4m3`, any CUDA GPU.

## Measured rows

137 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `flashinfer` | `bf16` | 137 |

## Used by these deployments

- [GLM-5.2 NVFP4: SGLang, TP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/mla_rope_quantize_fp8
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/mla_rope_quantize_fp8/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/mla_rope_quantize_fp8/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/mla_rope_quantize_fp8.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
