# All-reduce, FlashInfer kernels (`all_reduce_fusion`) | ServingStudio Kernel Library

Sum a \[num_tokens, hidden_dim\] tensor across the GPUs of a tensor-parallel group with FlashInfer's all-reduce.

vLLM can end a tensor-parallel block with FlashInfer's all-reduce instead of NCCL, over a TRT-LLM IPC workspace or, by default on B200, an NVLink-multicast (MNNVL) workspace. Its cost depends on the \[num_tokens, hidden_dim\] shape rather than on the bytes alone, and vLLM changes its launch at 16 tokens, so it is measured by shape, unlike the byte-keyed all_reduce. Each GPU contributes a random normal tensor.

Category: Communication. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=all_reduce_fusion. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `output = Σᵣ inputᵣ, on each GPU`
- `algbw = num_tokens·hidden_dim·bytes per element / time`
- `busbw = algbw · 2(num_gpus − 1) / num_gpus`
- `MNNVL: one-shot when num_tokens·hidden_dim·num_gpus·bytes per element ≤ 1 MiB, otherwise two-shot`

## How it is measured

Wall-clock time of CUDA graph replays, since vLLM runs this collective inside a graph. After 50 direct calls and 50 more all-reduces in graph replays, the timed replays run from a barrier to a device synchronize, and the slowest rank's mean per all-reduce is kept. flashinfer_trtllm puts one all-reduce in each graph and times 100; flashinfer_mnnvl puts ten in each graph and times 1000, after checking one output against the FP32 sum of every rank's input.

## Caveats

- PDL is on, and completion is signaled at kernel end only above 16 tokens, as vLLM launches it.
- Host time around graph replays includes the replay launch cost, not only kernel time.
- The fabric argument labels the row; the run uses whatever links connect the reserved GPUs.
- flashinfer_mnnvl is measured only where vLLM routes to it: fabric nvlink, a TP size vLLM has a FlashInfer all-reduce budget for, and a tensor within that budget; vLLM sends larger tensors to another all-reduce.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_gpus` | number | GPUs | GPUs in the tensor-parallel group. |
| `num_tokens` | number | tokens | Token rows in each GPU's input tensor. |
| `hidden_dim` | number | elements | Features in each input row. |
| `dtype` | dtype |  | Element type of the input tensor. |
| `fabric` | label |  | Interconnect label used to identify the measurement; the run uses the reserved GPUs' links. |

## Metrics

- Time (`time_ms`, ms)
- Algorithm bandwidth (`algbw_gbps`, GB/s)
- Bus bandwidth (`busbw_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `flashinfer_trtllm`: FlashInfer's TRT-LLM allreduce_fusion call with the kAllReduce pattern and PDL enabled. dtypes `bf16`, any CUDA GPU. Source: https://github.com/flashinfer-ai/flashinfer/blob/main/flashinfer/comm/allreduce.py
- `flashinfer_mnnvl`: FlashInfer's allreduce_fusion over an NVLink-multicast workspace, as vLLM calls it on B200; FlashInfer picks one-shot or two-shot by size. dtypes `bf16`, SM targets `sm_100a`, GPUs B200-SXM-180GB, GB200-Superchip. Source: https://github.com/flashinfer-ai/flashinfer/blob/main/flashinfer/comm/trtllm_mnnvl_ar.py

## Measured rows

150 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `flashinfer_mnnvl` | `bf16` | 86 |
| NVIDIA B200 | `flashinfer_trtllm` | `bf16` | 64 |

## Used by these deployments

- [DeepSeek V4.1 Flash 1017: vLLM, TP attention, EP MoE, DSpark speculative decoding](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dspark.md) (`DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dspark`)
- [DeepSeek V4.1 Flash: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash/deepseek_v41_vllm.md) (`DeepSeek-V4.1-Flash/deepseek_v41_vllm`)
- [DeepSeek V4.1 Flash: vLLM, TP attention, EP MoE, side streams serialized](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash/deepseek_v41_vllm_serial_streams.md) (`DeepSeek-V4.1-Flash/deepseek_v41_vllm_serial_streams`)
- [GLM-5.2 NVFP4: SGLang, TP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, TP and EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, TP and EP, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative`)
- [GLM-5.3-Flash NVFP4: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe`)
- [GLM-5.3-Flash: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe`)
- [GLM-5.3 NVFP4: vLLM, TP and EP, DFlash speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2.md) (`GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/all_reduce_fusion
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/all_reduce_fusion/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/all_reduce_fusion/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/all_reduce_fusion.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
