# Summed RMSNorm with fused outputs (`multi_add_rms_norm`) | ServingStudio Kernel Library

Sum one to three residual-stream tensors, RMS-normalize, and write any of the sum, the normalized rows and their FP8 quantization in one kernel.

Under torch.compile vLLM's native fused_add_rms_norm becomes part of one Inductor reduction kernel together with its neighbours. Inductor does not store a residual sum that only feeds the next layer's norm: that kernel reads the previous addends again and re-adds them. A static FP8 input quantization of the following linear layer fuses in as a third output plane. The output set is part of the kernel: each plane is a full write.

Category: Normalization; group: RMSNorm. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=multi_add_rms_norm. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `s = x₁ + … + x_k, accumulated in fp32`
- `n = s / √(mean(s²) + ε) · weight`
- `outputs: s (store_sum), n (store_normed), clamp(n / scale, ±448) as E4M3 (store_fp8)`
- `GB/s = (k·m·h·b + h·b + m·h·(b·store_sum + b·store_normed + store_fp8)) / time, with m = num_tokens, h = hidden_dim, b = bytes per element`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. The block is compiled once with a symbolic token count, and its first call, at 8192 tokens or at the row's token count when larger, picks the launch config the row then uses, as vLLM's profile run at max_num_batched_tokens does. Every launch of the compiled block is counted.

## Caveats

- GB/s counts each input once. The kernel reads the addends twice, once for the mean square and once for the outputs; the second read mostly hits L2.
- The weight is a plain multiplier; a (1 + w) weight adds one addition per element in registers and leaves the traffic unchanged.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_tokens` | number | tokens | Rows normalized together. |
| `hidden_dim` | number | elements | Width of each row. |
| `num_addends` | number | tensors | Row-aligned tensors summed before the norm: 1 is a plain RMSNorm, 2 a residual add, 3 a residual add that also recomputes the previous one. |
| `store_sum` | label |  | Whether the sum is written back as the new residual. |
| `store_normed` | label |  | Whether the normalized rows are written in dtype. |
| `store_fp8` | label |  | Whether the normalized rows are written as static per-tensor FP8 E4M3. |
| `dtype` | dtype |  | Element type of the addends, the weight and the stored rows. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_inductor`: vLLM's native fused_add_rms_norm and QuantFP8 compiled by Inductor into one Triton reduction kernel. dtypes `bf16`, compute capability 8.0 or newer, GPUs A100-SXM4-40GB, A100-SXM4-80GB, A100-PCIe-40GB, A100-PCIe-80GB, H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip, H800-SXM5-80GB, H20-SXM-96GB, A40, L40, L40S, L4. Source: https://github.com/vllm-project/vllm/blob/main/vllm/ir/ops/layernorm.py

## Measured rows

340 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `vllm_inductor` | `bf16` | 340 |

## Used by these deployments

- [Qwen3.8 27B NVFP4: vLLM, one GPU](https://servingstudio.cs.washington.edu/models/Qwen3.8-27B-NVFP4/qwen35_vllm_nvfp4_gdn_dense.md) (`Qwen3.8-27B-NVFP4/qwen35_vllm_nvfp4_gdn_dense`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/multi_add_rms_norm
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/multi_add_rms_norm/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/multi_add_rms_norm/configs

## Torch reference

What the kind computes, as Torch code (`profiling/runners/norm/multi_add_rms_norm_reference.py` in ServingStudio Sim):

```python
"""Independent value reference for the summed RMSNorm with fused outputs."""

from typing import Any

FP8_E4M3_MAX = 448.0


def multi_add_rms_norm_reference(
    torch: Any,
    addends: list[Any],
    weight: Any,
    scale: Any,
    eps: float,
) -> tuple[Any, Any, Any]:
    """Return (sum, normalized, fp8) for row-aligned addends; all three planes, always."""
    total = addends[0].float()
    for addend in addends[1:]:
        total = total + addend.float()
    normalized = total * torch.rsqrt(total.square().mean(-1, keepdim=True) + eps) * weight.float()
    fp8 = (normalized / scale.float()).clamp(-FP8_E4M3_MAX, FP8_E4M3_MAX).to(torch.float8_e4m3fn)
    dtype = addends[0].dtype
    return total.to(dtype), normalized.to(dtype), fp8


__all__ = ["multi_add_rms_norm_reference"]
```

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/multi_add_rms_norm.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
