# Query and KV RMSNorm (`q_kv_rms_norm`) | ServingStudio Kernel Library

RMS-normalize query-side and KV-side projections with separate weights.

After the attention input projections, the query-side rows (q_dim wide) and the KV-side rows (kv_dim wide) are RMS-normalized with their own weights in one call, before the queries and cache entries are built. Inputs and weights are seeded random BF16 values.

Category: Normalization; group: RMSNorm. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=q_kv_rms_norm. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `yq = q / √(mean(q²) + rms_eps) · q_weight`
- `ykv = kv / √(mean(kv²) + rms_eps) · kv_weight`
- `GB/s = 2 · (2 · num_tokens · (q_dim + kv_dim) + q_dim + kv_dim) / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. Three warm-up calls run first; every launch of the call is counted.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_tokens` | number | tokens | Tokens normalized together. |
| `q_dim` | number | elements | Width of each query-side input row. |
| `kv_dim` | number | elements | Width of each KV-side input row. |
| `rms_eps` | number | unitless | Constant added to each mean square before RMS normalization. |
| `dtype` | dtype |  | Element type of the inputs and weights. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_triton`: vLLM's fused_q_kv_rmsnorm Triton call normalizes both projections together. dtypes `bf16`, any CUDA GPU. Source: https://github.com/vllm-project/vllm/blob/main/vllm/models/common/ops/fused_qk_rmsnorm.py

## Measured rows

136 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `vllm_triton` | `bf16` | 68 |
| NVIDIA H200 | `vllm_triton` | `bf16` | 68 |

## Used by these deployments

- [DeepSeek V4 Flash 0731: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm.md) (`DeepSeek-V4-Flash-0731/deepseek_v4_vllm`)
- [DeepSeek V4 Flash 0731: vLLM, DP attention, EP, parallel streams serialized](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams.md) (`DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams`)
- [GLM-5.3-Flash NVFP4: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe`)
- [GLM-5.3-Flash: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe`)
- [GLM-5.3-Flash: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe`)
- [GLM-5.3-Flash: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/q_kv_rms_norm
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/q_kv_rms_norm/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/q_kv_rms_norm/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/q_kv_rms_norm.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
