# MoE all-gather of hidden states and router logits (`moe_ep_all_gather`) | ServingStudio Kernel Library

Gather hidden states and router logits from every data-parallel GPU.

Under vLLM's default all-to-all backend, allgather_reducescatter, a MoE layer with data and expert parallelism starts by gathering every GPU's hidden states and router logits, so each GPU can route all tokens. GPU r contributes per_rank_tokens\[r\] tokens and receives both full tensors. With block-FP8 experts the GPUs first quantize their tokens, so the hidden states travel as fp8_e4m3 together with one fp32 scale per 128 elements. The gathers run in one NCCL group: all_gather when the counts are equal, all_gatherv otherwise.

Category: Communication. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=moe_ep_all_gather. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `token bytes = 2·hidden_size + 4·num_experts (bf16), hidden_size + 4·hidden_size/128 + 4·num_experts (fp8_e4m3)`
- `total bytes = sum(per_rank_tokens) · token bytes`
- `algbw = (total bytes / num_gpus) / time`
- `busbw = (total bytes − min(per_rank_tokens)·token bytes) / time`

## How it is measured

CUDA-event time around 100 repeated groups of PyNCCL calls after 50 warm-up groups and a barrier. Each GPU times its own stream; the largest mean per-call time across ranks is kept. Input tensors are built before timing.

## Caveats

- Measured with NCCL_NVLS_ENABLE=0: NVLS multicast is unavailable on the profiling host.
- Only the tensors named above are gathered; a production call that adds other extra tensors moves more bytes.
- The gathers are timed together, so their individual costs are not reported.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_gpus` | number | GPUs | GPUs in the data-parallel group. |
| `per_rank_tokens` | list | tokens | Token counts for each GPU, in rank order. |
| `hidden_size` | number | elements | Elements in each token's hidden state. |
| `num_experts` | number | experts | Router logits per token, one per expert. |
| `hidden_dtype` | dtype |  | Element type of the gathered hidden states: bf16, or fp8_e4m3 with one fp32 scale per 128 elements gathered alongside. |
| `router_dtype` | dtype |  | Element type of the router logits; fp32 is measured. |
| `fabric` | label |  | Interconnect label; this backend requires nvlink. |

## Metrics

- Time (`time_ms`, ms)
- Algorithm bandwidth (`algbw_gbps`, GB/s)
- Bus bandwidth (`busbw_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_pynccl`: vLLM's PyNCCL wrapper: all gathers in one NCCL group, with all_gatherv for unequal token counts. dtypes `bf16`, `fp8_e4m3`, any CUDA GPU. Source: https://github.com/vllm-project/vllm/blob/main/vllm/distributed/device_communicators/pynccl.py

## Measured rows

62 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `vllm_pynccl` | `fp8_e4m3` | 46 |
| NVIDIA H200 | `vllm_pynccl` | `bf16` | 16 |

## Used by these deployments

- [DeepSeek V4 Flash 0731: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm.md) (`DeepSeek-V4-Flash-0731/deepseek_v4_vllm`)
- [DeepSeek V4 Flash 0731: vLLM, DP attention, EP, parallel streams serialized](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams.md) (`DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams`)
- [GLM-5.3-Flash: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/moe_ep_all_gather
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/moe_ep_all_gather/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/moe_ep_all_gather/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/moe_ep_all_gather.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
