# RMSNorm (`rms_norm`) | ServingStudio Kernel Library

Normalize each token's hidden vector by its root mean square and scale it by a weight.

RMSNorm on its own, without the residual add that residual_rms_norm fuses in. Models use it wherever a tensor is normalized alone, for example before the attention projections. The input has m token rows of hidden features; input and weight are random normal. ε is 1e-6 for flashinfer and vllm_gemma_native, 1e-5 for vllm_cuda.

Category: Normalization; group: RMSNorm. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=rms_norm. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `y = x / √(mean(x²) + ε) · weight, per token`
- `flashinfer: TFLOPS = 5·m·hidden / time; GB/s = 2·m·hidden·bytes per element / time`
- `vllm_cuda: TFLOPS = (4·m·hidden + 2·m) / time; GB/s = (2·m·hidden + hidden)·bytes per element / time`
- `vllm_gemma_native: y = x / √(mean(x²) + ε) · (1 + weight); same TFLOPS and GB/s counts as vllm_cuda`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. flashinfer and vllm_cuda count only their norm kernel (RMSNormKernel, rms_norm_kernel); vllm_gemma_native counts every eager launch of the call. vllm_cuda and vllm_gemma_native check one output against a PyTorch RMSNorm before timing.

## Caveats

- vllm_gemma_native is launch-bound below the 32-row grid start: at hidden 2560 the cache over-predicts 1 to 24 rows by up to 1.27x (about 5 µs a call) on B200.
- The backends count work differently: flashinfer's GB/s leaves out the weight read and its TFLOPS assumes 5 operations per element; vllm_cuda counts the weight read and 4 operations per element plus 2 per row.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `m` | number | tokens | Tokens in the batch. |
| `hidden` | number | elements | Features normalized per token. |
| `dtype` | dtype |  | Element type of the input and weight. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `flashinfer`: FlashInfer's norm.rmsnorm. dtypes `bf16`, `fp16`, any CUDA GPU. Source: https://github.com/flashinfer-ai/flashinfer/blob/main/flashinfer/norm/__init__.py
- `vllm_cuda`: vLLM's rms_norm CUDA op, as RMSNorm.forward_cuda launches it without a residual. dtypes `bf16`, any CUDA GPU. Source: https://github.com/vllm-project/vllm/blob/main/csrc/libtorch_stable/layernorm_kernels.cu
- `vllm_gemma_native`: vLLM's GemmaRMSNorm.forward_native run eagerly: the vllm.ir rms_norm native implementation with an fp32 1 + weight. dtypes `bf16`, any CUDA GPU. Source: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/layernorm.py

## Measured rows

756 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `flashinfer` | `bf16` | 252 |
| NVIDIA B200 | `vllm_cuda` | `bf16` | 63 |
| NVIDIA B200 | `vllm_gemma_native` | `bf16` | 126 |
| NVIDIA H200 | `flashinfer` | `bf16` | 315 |

## Used by these deployments

- [GLM-5.2 FP8: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-FP8/glm52_vllm_dsa_moe.md) (`GLM-5.2-FP8/glm52_vllm_dsa_moe`)
- [GLM-5.2 NVFP4: SGLang, TP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, TP and EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, TP and EP, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative`)
- [GLM-5.2 NVFP4: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_pp_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_pp_dsa_moe`)
- [GLM-5.2: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2/glm52_vllm_dsa_moe.md) (`GLM-5.2/glm52_vllm_dsa_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe`)
- [GLM-5.3-Flash: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe`)
- [GLM-5.3-Flash: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe`)
- [GLM-5.3-Flash: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe`)
- [GLM-5.3 NVFP4: vLLM, TP and EP, DFlash speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2.md) (`GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2`)
- [Llama 3.1 8B: Single GPU](https://servingstudio.cs.washington.edu/models/Llama-3.1-8B/llama3_dense.md) (`Llama-3.1-8B/llama3_dense`)
- [Llama 3.1 8B: TP](https://servingstudio.cs.washington.edu/models/Llama-3.1-8B/llama3_dense_tp.md) (`Llama-3.1-8B/llama3_dense_tp`)
- [Llama 3.1 8B: DP attention, TP FFN](https://servingstudio.cs.washington.edu/models/Llama-3.1-8B/llama3_dp_attn_tp_ffn.md) (`Llama-3.1-8B/llama3_dp_attn_tp_ffn`)
- [Qwen3 235B-A22B FP8: FFN side of an attention–FFN split](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_fp8_ffn_moe.md) (`Qwen3-235B-A22B-FP8/qwen3_fp8_ffn_moe`)
- [Qwen3 235B-A22B FP8: DP attention, EP MoE, DeepGEMM experts](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_moe_fp8_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-FP8/qwen3_moe_fp8_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B FP8: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_vllm_moe_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-FP8/qwen3_vllm_moe_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B Thinking 2507 FP8: DP attention, EP MoE, DeepGEMM experts](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_moe_fp8_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_moe_fp8_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B Thinking 2507 FP8: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_vllm_moe_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_vllm_moe_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B: FFN side of an attention–FFN split](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B/qwen3_ffn_moe.md) (`Qwen3-235B-A22B/qwen3_ffn_moe`)
- [Qwen3 235B-A22B: DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B/qwen3_moe_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B/qwen3_moe_dp_attn_ep_ffn`)
- [Qwen3 Coder 480B-A35B: FFN side of an attention–FFN split](https://servingstudio.cs.washington.edu/models/Qwen3-Coder-480B-A35B-Instruct/qwen3_fp8_ffn_moe.md) (`Qwen3-Coder-480B-A35B-Instruct/qwen3_fp8_ffn_moe`)
- [Qwen3.6 35B-A3B FP8: Single GPU](https://servingstudio.cs.washington.edu/models/Qwen3.6-35B-A3B-FP8/qwen36_local.md) (`Qwen3.6-35B-A3B-FP8/qwen36_local`)
- [Qwen3.8 Flash Next NVFP4: vLLM, one GPU, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative.md) (`Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/rms_norm
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/rms_norm/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/rms_norm/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/rms_norm.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
