# Residual add + RMSNorm (`residual_rms_norm`) | ServingStudio Kernel Library

Add the residual stream, then normalize: the fused step between transformer blocks.

Before each attention and MLP block, the model adds the previous block's output to the residual stream and RMS-normalizes the sum. vLLM fuses both into one kernel that writes the normalized output over the input and the new residual over the old one. The torch backend runs the same math as separate launches.

Category: Normalization; group: RMSNorm. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=residual_rms_norm. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `s = x + residual`
- `y = s / √(mean(s²) + ε) · weight, with ε = 1e-5`
- `returns (y, s)`
- `GB/s = (4·m·hidden + hidden) · bytes per element / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. vllm_cuda counts only its fused_add_rms_norm_kernel launch; torch counts every launch of the reference.

## Caveats

- The reference, which the torch backend times, computes the sum and the RMS in fp32 and rounds to the input dtype before the weight multiply.
- GB/s counts logical traffic: x, residual and weight read, y and s written. The torch backend's fp32 temporaries are not counted.
- vllm_cuda works in place, so it is timed on zero inputs with a weight of ones: repeated calls then cannot overflow fp16, and the launch and memory access are unchanged.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `m` | number | tokens | Tokens in the batch. |
| `hidden` | number | elements | Hidden size of the model. |
| `dtype` | dtype |  | Element type of x, residual and weight. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `torch`: The PyTorch reference, timed as its separate launches. dtypes `bf16`, `fp16`, any CUDA GPU.
- `vllm_cuda`: vLLM's fused_add_rms_norm CUDA kernel: one in-place launch. dtypes `bf16`, `fp16`, any CUDA GPU. Source: https://github.com/vllm-project/vllm/blob/main/csrc/libtorch_stable/layernorm_kernels.cu

## Measured rows

272 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `vllm_cuda` | `bf16` | 68 |
| NVIDIA H200 | `vllm_cuda` | `bf16` | 204 |

## Used by these deployments

- [GLM-5.2 FP8: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-FP8/glm52_vllm_dsa_moe.md) (`GLM-5.2-FP8/glm52_vllm_dsa_moe`)
- [GLM-5.2 NVFP4: SGLang, TP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, TP and EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, TP and EP, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative`)
- [GLM-5.2 NVFP4: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_pp_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_pp_dsa_moe`)
- [GLM-5.2: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2/glm52_vllm_dsa_moe.md) (`GLM-5.2/glm52_vllm_dsa_moe`)
- [GLM-5.3 NVFP4: vLLM, TP and EP, DFlash speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2.md) (`GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2`)
- [Qwen3.6 35B-A3B FP8: Single GPU](https://servingstudio.cs.washington.edu/models/Qwen3.6-35B-A3B-FP8/qwen36_local.md) (`Qwen3.6-35B-A3B-FP8/qwen36_local`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/residual_rms_norm
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/residual_rms_norm/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/residual_rms_norm/configs

## Torch reference

What the kind computes, as Torch code (`profiling/runners/norm/residual_rms_norm_reference.py` in ServingStudio Sim):

```python
"""Torch semantic reference for residual-add RMSNorm.

The residual sum and RMS reduction are evaluated in FP32. The function returns
both the normalized output and the input-dtype residual sum without mutating
the caller's tensors.
"""

from __future__ import annotations

import math

import torch

_SUPPORTED_DTYPES = (torch.bfloat16, torch.float16, torch.float32)


def residual_rms_norm_reference(
    x: torch.Tensor,
    residual: torch.Tensor,
    weight: torch.Tensor,
    eps: float = 1e-5,
) -> tuple[torch.Tensor, torch.Tensor]:
    """Return weighted RMSNorm output and residual sum for two 2-D inputs."""
    # Source semantics: vLLM RMSNorm.forward_static/fused_add_rms_norm.
    if not isinstance(x, torch.Tensor):
        raise TypeError("x must be a torch.Tensor")
    if not isinstance(residual, torch.Tensor):
        raise TypeError("residual must be a torch.Tensor")
    if not isinstance(weight, torch.Tensor):
        raise TypeError("weight must be a torch.Tensor")

    if x.ndim != 2:
        raise ValueError(f"x must be 2-D, got rank {x.ndim}")
    if residual.ndim != 2:
        raise ValueError(f"residual must be 2-D, got rank {residual.ndim}")
    if x.shape != residual.shape:
        raise ValueError(
            f"x and residual must have the same shape, got {x.shape} and {residual.shape}"
        )

    m, hidden = x.shape
    if m <= 0 or hidden <= 0:
        raise ValueError(f"x and residual dimensions must be positive, got {(m, hidden)}")

    if x.dtype not in _SUPPORTED_DTYPES:
        raise TypeError(
            f"x dtype must be one of torch.bfloat16, torch.float16, or torch.float32, got {x.dtype}"
        )
    if residual.dtype != x.dtype:
        raise TypeError(
            f"x and residual must have the same dtype, got {x.dtype} and {residual.dtype}"
        )
    if weight.ndim != 1 or weight.shape != (hidden,):
        raise ValueError(f"weight must have shape ({hidden},), got {tuple(weight.shape)}")
    if weight.dtype != x.dtype:
        raise TypeError(f"weight dtype must match input dtype {x.dtype}, got {weight.dtype}")
    if residual.device != x.device or weight.device != x.device:
        raise ValueError("x, residual, and weight must be on the same device")

    if isinstance(eps, bool):
        raise TypeError("eps must be a positive finite number")
    try:
        eps_value = float(eps)
    except (TypeError, ValueError) as exc:
        raise TypeError("eps must be a positive finite number") from exc
    if not math.isfinite(eps_value) or eps_value <= 0:
        raise ValueError(f"eps must be positive and finite, got {eps}")

    summed_fp32 = x.float() + residual.float()
    residual_out = summed_fp32.to(x.dtype)
    variance_fp32 = summed_fp32.square().mean(dim=-1, keepdim=True)
    normalized_fp32 = summed_fp32 * torch.rsqrt(variance_fp32 + eps_value)
    normalized_out = normalized_fp32.to(x.dtype) * weight
    return normalized_out, residual_out
```

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/residual_rms_norm.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
