# All-reduce + residual add + RMSNorm (`all_reduce_residual_rms_norm`) | ServingStudio Kernel Library

Sum a tensor across GPUs, add the residual and RMS-normalize the result in a fused call.

vLLM fuses the all-reduce at the end of a tensor-parallel block with the residual add and RMSNorm that follow into one FlashInfer kernel, over a TRT-LLM IPC workspace or, by default on B200, an NVLink-multicast (MNNVL) workspace, up to a per-GPU fusion-size limit, above which it runs them unfused. The input and residual are random normal \[num_tokens, hidden_dim\] tensors; the norm weight is all ones and ε is 1e-6.

Category: Communication. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=all_reduce_residual_rms_norm. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `s = Σᵣ inputᵣ + residual`
- `y = s / √(mean(s²) + 1e-6) · weight`
- `MNNVL: one-shot when num_tokens·hidden_dim·num_gpus·bytes per element ≤ 1 MiB, otherwise two-shot`
- `algbw = num_tokens·hidden_dim·bytes per element / time`
- `busbw = algbw · 2(num_gpus − 1) / num_gpus`

## How it is measured

Wall-clock time of CUDA graph replays, since vLLM runs this call inside a graph. Each graph holds ten fused calls. After 50 direct calls and 5 replays, the timed replays run from a barrier to a device synchronize, and the slowest rank's mean per call is kept. flashinfer_trtllm times 100 calls; flashinfer_mnnvl times 1000, after checking the residual and normalized outputs against an FP32 reference.

## Caveats

- algbw and busbw divide the all-reduce input size by the time of the whole fused call, so they do not isolate link traffic from the add and the norm.
- Host time around graph replays includes the replay launch cost, not only kernel time.
- The fabric argument labels the row; the run uses whatever links connect the reserved GPUs.
- flashinfer_mnnvl is measured only where vLLM routes to it: fabric nvlink, strategy auto, a TP size vLLM has a FlashInfer budget for, and a tensor within that budget.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_gpus` | number | GPUs | GPUs in the tensor-parallel group. |
| `num_tokens` | number | tokens | Token rows in each GPU's input tensor. |
| `hidden_dim` | number | elements | Features in each input row. |
| `dtype` | dtype |  | Element type of the input and residual tensors. |
| `fabric` | label |  | Interconnect label used to identify the measurement; the run uses the reserved GPUs' links. |
| `strategy` | label |  | All-reduce strategy: auto, oneshot or twoshot. |
| `launch_with_pdl` | label |  | Whether programmatic dependent launch is enabled. |
| `trigger_completion_at_end` | label |  | Whether PDL completion is signaled at the end of the device kernel. |
| `fp32_acc` | label |  | Whether the all-reduce uses fp32 accumulation. |

## Metrics

- Time (`time_ms`, ms)
- Algorithm bandwidth (`algbw_gbps`, GB/s)
- Bus bandwidth (`busbw_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `flashinfer_trtllm`: FlashInfer's TRT-LLM allreduce_fusion call with the kARResidualRMSNorm pattern. dtypes `bf16`, any CUDA GPU. Source: https://github.com/flashinfer-ai/flashinfer/blob/main/flashinfer/comm/allreduce.py
- `flashinfer_mnnvl`: FlashInfer's allreduce_fusion with the kARResidualRMSNorm pattern over an NVLink-multicast workspace, as vLLM calls it on B200; FlashInfer picks one-shot or two-shot by size. dtypes `bf16`, SM targets `sm_100a`, GPUs B200-SXM-180GB, GB200-Superchip. Source: https://github.com/flashinfer-ai/flashinfer/blob/main/flashinfer/comm/trtllm_mnnvl_ar.py

## Measured rows

143 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `flashinfer_mnnvl` | `bf16` | 41 |
| NVIDIA B200 | `flashinfer_trtllm` | `bf16` | 93 |
| NVIDIA H200 | `flashinfer_trtllm` | `bf16` | 9 |

## Used by these deployments

- [GLM-5.2 NVFP4: SGLang, TP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, TP and EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, TP and EP, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative`)
- [GLM-5.3 NVFP4: vLLM, TP and EP, DFlash speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2.md) (`GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2`)
- [MiMo V2.6 Flash RL: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/MiMo-V2.6-Flash-RL/mimo_v2_vllm_fp8_swa_moe.md) (`MiMo-V2.6-Flash-RL/mimo_v2_vllm_fp8_swa_moe`)
- [Qwen3 235B-A22B FP8: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_vllm_moe_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-FP8/qwen3_vllm_moe_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B Thinking 2507 FP8: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_vllm_moe_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_vllm_moe_dp_attn_ep_ffn`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/all_reduce_residual_rms_norm
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/all_reduce_residual_rms_norm/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/all_reduce_residual_rms_norm/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/all_reduce_residual_rms_norm.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
