# All-reduce (`all_reduce`) | ServingStudio Kernel Library

Sum one buffer across the GPUs of a tensor-parallel group.

Tensor-parallel attention and MLP blocks end with an all-reduce of the row-parallel projection's output, so every GPU holds the full sum. message_size_bytes is the full buffer each GPU contributes, not a shard. Measured on the GPUs of one node, over whatever links connect them.

Category: Communication. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=all_reduce. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `algbw = message_size_bytes / time`
- `busbw = algbw · 2(N − 1) / N, for N = num_gpus`

## How it is measured

Wall time between two CUDA events around repeated calls. Each size runs 50 warm-up all-reduces, a barrier, then 100 timed calls back to back in the same live process group; the time is rank 0's mean per call. Energy is not measured for collectives.

## Caveats

- busbw is the number to compare with the link: it scales algbw by the 2(N − 1)/N traffic of a ring all-reduce. The NVLink figure in the GPU catalog counts both directions.
- Cost is keyed by bytes, not dtype: rows are measured at bf16, and the simulator reads an fp8 payload as fewer bytes on the same curve.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_gpus` | number | GPUs | GPUs in the group, one rank each. |
| `message_size_bytes` | number | bytes | Full buffer each GPU contributes and receives, not a per-GPU shard. |
| `dtype` | dtype |  | Element type of the buffer. The simulator profiles bf16 only. |
| `fabric` | label |  | Interconnect label, such as nvlink. It keys the row; the run uses whatever links the reserved GPUs have. |

## Metrics

- Time (`time_ms`, ms)
- Algorithm bandwidth (`algbw_gbps`, GB/s)
- Bus bandwidth (`busbw_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `nccl`: torch.distributed.all_reduce with sum, on the NCCL backend. any CUDA GPU. Source: https://github.com/NVIDIA/nccl
- `nvshmem`: nvshmem4py reduce with sum over TEAM_WORLD, on symmetric-memory buffers, so every GPU receives the sum. any CUDA GPU. Source: https://github.com/NVIDIA/nvshmem

## Measured rows

171 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `nccl` |  | 38 |
| NVIDIA B200 | `nvshmem` |  | 38 |
| NVIDIA H200 | `nccl` |  | 57 |
| NVIDIA H200 | `nvshmem` |  | 38 |

## Used by these deployments

- [DeepSeek V4.1 Flash 1017: vLLM, TP attention, EP MoE, DSpark speculative decoding](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dspark.md) (`DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dspark`)
- [DeepSeek V4.1 Flash: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash/deepseek_v41_vllm.md) (`DeepSeek-V4.1-Flash/deepseek_v41_vllm`)
- [DeepSeek V4.1 Flash: vLLM, TP attention, EP MoE, side streams serialized](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash/deepseek_v41_vllm_serial_streams.md) (`DeepSeek-V4.1-Flash/deepseek_v41_vllm_serial_streams`)
- [GLM-5.2 NVFP4: SGLang, TP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, TP and EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, TP and EP, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative`)
- [GLM-5.3-Flash NVFP4: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe`)
- [GLM-5.3-Flash: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe`)
- [GLM-5.3 NVFP4: vLLM, TP and EP, DFlash speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2.md) (`GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2`)
- [Inkling NVFP4: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Inkling-NVFP4/inkling_vllm_nvfp4_rel_attn_moe.md) (`Inkling-NVFP4/inkling_vllm_nvfp4_rel_attn_moe`)
- [Inkling Small NVFP4: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Inkling-Small-NVFP4/inkling_vllm_nvfp4_rel_attn_moe.md) (`Inkling-Small-NVFP4/inkling_vllm_nvfp4_rel_attn_moe`)
- [Llama 3.1 8B: TP](https://servingstudio.cs.washington.edu/models/Llama-3.1-8B/llama3_dense_tp.md) (`Llama-3.1-8B/llama3_dense_tp`)
- [Llama 3.1 8B: DP attention, TP FFN](https://servingstudio.cs.washington.edu/models/Llama-3.1-8B/llama3_dp_attn_tp_ffn.md) (`Llama-3.1-8B/llama3_dp_attn_tp_ffn`)
- [MiMo V2.6 Flash RL: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/MiMo-V2.6-Flash-RL/mimo_v2_vllm_fp8_swa_moe.md) (`MiMo-V2.6-Flash-RL/mimo_v2_vllm_fp8_swa_moe`)
- [Qwen3 235B-A22B FP8: FFN side of an attention–FFN split](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_fp8_ffn_moe.md) (`Qwen3-235B-A22B-FP8/qwen3_fp8_ffn_moe`)
- [Qwen3 235B-A22B FP8: DP attention, EP MoE, DeepGEMM experts](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_moe_fp8_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-FP8/qwen3_moe_fp8_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B FP8: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_vllm_moe_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-FP8/qwen3_vllm_moe_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B Thinking 2507 FP8: DP attention, EP MoE, DeepGEMM experts](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_moe_fp8_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_moe_fp8_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B Thinking 2507 FP8: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_vllm_moe_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_vllm_moe_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B: FFN side of an attention–FFN split](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B/qwen3_ffn_moe.md) (`Qwen3-235B-A22B/qwen3_ffn_moe`)
- [Qwen3 235B-A22B: DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B/qwen3_moe_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B/qwen3_moe_dp_attn_ep_ffn`)
- [Qwen3 Coder 480B-A35B: FFN side of an attention–FFN split](https://servingstudio.cs.washington.edu/models/Qwen3-Coder-480B-A35B-Instruct/qwen3_fp8_ffn_moe.md) (`Qwen3-Coder-480B-A35B-Instruct/qwen3_fp8_ffn_moe`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/all_reduce
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/all_reduce/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/all_reduce/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/all_reduce.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
