# MoE all-to-all transfer (`moe_alltoall`) | ServingStudio Kernel Library

Move token rows to expert GPUs or return expert outputs to their source GPUs.

In expert-parallel MoE, dispatch sends each token's row to the GPUs that own its selected experts, and combine sends the expert outputs back. vLLM runs both legs as this one transfer under --all2all-backend=flashinfer_nvlink_two_sided. The transfer ends when its slowest GPU ends, so the two row counts are those of the busiest sender and the busiest receiver. The legs differ in how they read their input: dispatch reads a token row once for every GPU it goes to, and combine reads each expert-output row once.

Category: Communication. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=moe_alltoall. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `payload bytes = max_send_rows · hidden_bytes`
- `algbw = payload bytes / time`
- `busbw = algbw`

## How it is measured

Mean time per call from CUDA events around 20 replays of a CUDA graph that captures 4 to 100 moe_comm calls, fewer for larger transfers. Ten eager calls and two replays warm up first. Rank 0's time is kept.

## Caveats

- Combine's zero fill and top-k sum, which vLLM runs around the transfer, are not included; the simulator prices them as separate elementwise kernels.
- The transfer indices are constructed from the requested row counts, rather than sampled from a router's expert choices.
- Bandwidth counts rank 0's sent payload, not traffic across all GPUs.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `ep_size` | number | GPUs | GPUs in the expert-parallel group. |
| `max_send_rows` | number | rows | Rows sent by the busiest GPU. |
| `max_recv_rows` | number | rows | Rows received by the busiest GPU. |
| `top_k` | number | experts | Expert slots selected per token. |
| `slot_count` | number | experts | Expert slots across the group. |
| `hidden_bytes` | number | bytes | Bytes in one transferred row. |
| `direction` | label |  | Transfer direction: dispatch or combine. |
| `fabric` | label |  | Interconnect label used to identify the measurement. |

## Metrics

- Time (`time_ms`, ms)
- Algorithm bandwidth (`algbw_gbps`, GB/s)
- Bus bandwidth (`busbw_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `flashinfer_mnnvl`: FlashInfer's MNNVL moe_comm: every GPU sends and receives indexed rows through a workspace in fabric memory. any CUDA GPU. Source: https://github.com/flashinfer-ai/flashinfer/blob/main/flashinfer/comm/trtllm_alltoall.py

## Measured rows

470 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA H200 | `flashinfer_mnnvl` |  | 470 |

## Used by these deployments

- [GLM-5.2 FP8: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-FP8/glm52_vllm_dsa_moe.md) (`GLM-5.2-FP8/glm52_vllm_dsa_moe`)
- [GLM-5.2: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2/glm52_vllm_dsa_moe.md) (`GLM-5.2/glm52_vllm_dsa_moe`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/moe_alltoall
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/moe_alltoall/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/moe_alltoall/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/moe_alltoall.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
