# MoE all-gather of quantized activations and top-k routing (`moe_ep_quantized_all_gather`) | ServingStudio Kernel Library

Gather NVFP4 or MXFP8 activations, scales and top-k routing from every data-parallel GPU.

With data and expert parallelism, vLLM's allgather_reducescatter backend can quantize activations and select experts before dispatch. Each GPU then gathers four tensors from every GPU: the quantized activations, either packed NVFP4 (hidden_size/2 bytes per token, one fp8 e4m3 block scale per 16 elements) or MXFP8 (hidden_size bytes, one ue8m0 scale per 32 elements), the fp32 top-k weights, the int32 top-k expert IDs and the block scales. MXFP8 is DeepSeek-V4.1's TRT-LLM MXFP4 experts' dispatch. GPU r contributes per_rank_tokens\[r\] tokens. The four gathers run in one NCCL group: all_gather when the counts are equal, all_gatherv otherwise.

Category: Communication. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=moe_ep_quantized_all_gather. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `token bytes = hidden_size/2 + hidden_size/16 + 8·top_k (NVFP4)`
- `token bytes = hidden_size + hidden_size/32 + 8·top_k (MXFP8)`
- `total bytes = sum(per_rank_tokens) · token bytes`
- `algbw = (total bytes / num_gpus) / time`
- `busbw = (total bytes − min(per_rank_tokens) · token bytes) / time`

## How it is measured

CUDA-event time around 100 repeated groups of four PyNCCL calls after 50 warm-up groups and a barrier. Each GPU times its own stream; the largest mean per-call time across ranks is kept. Input tensors are built before timing.

## Caveats

- Measured with NCCL_NVLS_ENABLE=0: NVLS multicast is unavailable on the profiling host.
- The scales are gathered in the linear layout; the swizzle happens after the gather and is not part of this measurement.
- The four gathers are timed together, so their individual costs are not reported.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_gpus` | number | GPUs | GPUs in the data-parallel group. |
| `per_rank_tokens` | list | tokens | Token counts for each GPU, in rank order. |
| `hidden_size` | number | elements | Elements in each token's hidden state. |
| `top_k` | number | experts | Experts selected per token. |
| `activation_dtype` | dtype |  | Element type of the gathered activations: nvfp4_e2m1 or mxfp8_e4m3. |
| `fabric` | label |  | Interconnect label; this backend requires nvlink. |

## Metrics

- Time (`time_ms`, ms)
- Algorithm bandwidth (`algbw_gbps`, GB/s)
- Bus bandwidth (`busbw_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_pynccl`: vLLM's PyNCCL wrapper: the four gathers in one NCCL group, with all_gatherv for unequal token counts. dtypes `mxfp8_e4m3`, `nvfp4_e2m1`, any CUDA GPU. Source: https://github.com/vllm-project/vllm/blob/main/vllm/distributed/device_communicators/all2all.py

## Measured rows

130 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `vllm_pynccl` | `mxfp8_e4m3` | 38 |
| NVIDIA B200 | `vllm_pynccl` | `nvfp4_e2m1` | 92 |

## Used by these deployments

- [DeepSeek V4.1 Flash 1017: vLLM, DP attention, EP MoE, DSpark speculative decoding](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dp_attn_ep_moe_dspark.md) (`DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dp_attn_ep_moe_dspark`)
- [GLM-5.2 NVFP4: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/moe_ep_quantized_all_gather
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/moe_ep_quantized_all_gather/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/moe_ep_quantized_all_gather/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/moe_ep_quantized_all_gather.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
