# Query norm, RoPE and KV insert (`qnorm_rope_kv_insert`) | ServingStudio Kernel Library

Normalize and rotate queries while rotating and inserting mixed FP8/bf16 KV rows into a packed MLA cache.

Between the attention projections and attention, one CUDA call prepares both sides. Each query head is RMS-normalized, RoPE is applied to its trailing rope_dim elements, and it is written into a head-padded output. Each KV row gets RoPE, its first 448 elements are quantized to FP8 while the rotary part stays BF16, and only the last num_insert_tokens rows are written to the packed cache. Inputs are seeded BF16 values with consecutive positions and cache slots.

Category: Attention; group: MLA. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=qnorm_rope_kv_insert. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `q_out = RoPE(q / √(mean(q²) + rms_eps))`
- `KV cache receives 448 FP8 bytes, 128 bf16 bytes and 8 scale bytes per inserted row`
- `GB/s = (2 · num_tokens · (num_heads · head_dim + head_dim + padded_heads · head_dim) + 8 · (num_tokens + num_insert_tokens) + 4 · num_tokens · rope_dim + 584 · num_insert_tokens) / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. Three warm-up calls run first; the call rewrites the same preallocated cache each time.

## Caveats

- GB/s counts logical input, output, position, RoPE-table and inserted-cache bytes, not physical memory transactions.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_tokens` | number | tokens | Query and KV rows processed together. |
| `num_insert_tokens` | number | tokens | Rows inserted into the packed KV cache. |
| `num_heads` | number | heads | Query heads per token. |
| `padded_heads` | number | heads | Query heads in the padded output. |
| `head_dim` | number | elements | Elements per query head and KV row. |
| `rope_dim` | number | elements | Trailing elements rotated by RoPE. |
| `block_size` | number | tokens | Token capacity of each packed cache block. |
| `rms_eps` | number | unitless | Constant added to each query mean square before RMS normalization. |
| `input_dtype` | dtype |  | Element type of the query and KV inputs. |
| `cache_dtype` | label |  | Format of the packed MLA cache. |
| `cache_layout` | label |  | Ordering of data and scales within each cache block. |
| `scale_format` | label |  | Format of the cache quantization scales. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_cuda`: vLLM's fused query-norm/RoPE and KV RoPE-quantize-insert CUDA call writes mixed FP8/bf16 MLA cache blocks. dtypes `bf16`, KV dtypes `fp8_e4m3`, compute capability 8.0 or newer, GPUs A100-SXM4-40GB, A100-SXM4-80GB, A100-PCIe-40GB, A100-PCIe-80GB, H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip, H800-SXM5-80GB, H20-SXM-96GB, A40, L40, L40S, L4. Source: https://github.com/vllm-project/vllm/blob/main/csrc/libtorch_stable/fused_qnorm_rope_kv_insert_kernel.cu

## Measured rows

339 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA H200 | `vllm_cuda` | `bf16` | 339 |

## Used by these deployments

- [DeepSeek V4 Flash 0731: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm.md) (`DeepSeek-V4-Flash-0731/deepseek_v4_vllm`)
- [DeepSeek V4 Flash 0731: vLLM, DP attention, EP, parallel streams serialized](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams.md) (`DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/qnorm_rope_kv_insert
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/qnorm_rope_kv_insert/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/qnorm_rope_kv_insert/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/qnorm_rope_kv_insert.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
