# MoE expert sum (`moe_sum`) | ServingStudio Kernel Library

Sum the routed expert output rows for each token.

Routed experts leave top_k output rows per token; this step sums them into one hidden_dim-wide row. vLLM's moe_sum has dedicated kernels for small top_k and a general kernel for the rest; older builds fell back to torch's sum over the expert dimension at top_k = 6. The expert rows are random bf16 values.

Category: MoE; group: Routing and combine. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=moe_sum. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `output[token, h] = Σₖ expert_output[token, k, h]`
- `TFLOPS = num_tokens · hidden_dim · (top_k − 1) / time`
- `GB/s = 2·num_tokens·hidden_dim·(top_k + 1) / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. Five warm-up calls run first. Every launch of the call is counted.

## Caveats

- At top_k = 6 the profiling vLLM build fell back to torch's reduce_kernel, so those rows time that kernel; newer builds use a dedicated vectorized kernel.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_tokens` | number | tokens | Tokens whose expert outputs are reduced. |
| `top_k` | number | experts | Expert outputs summed for each token. |
| `hidden_dim` | number | elements | Features in each expert output. |
| `dtype` | dtype |  | Element type of the expert outputs. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_cuda`: vLLM's _custom_ops.moe_sum, timed as one call at any top_k. dtypes `bf16`, any CUDA GPU. Source: https://github.com/vllm-project/vllm/blob/main/vllm/_custom_ops.py

## Measured rows

68 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA H200 | `vllm_cuda` | `bf16` | 68 |

## Used by these deployments

- [DeepSeek V4 Flash 0731: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm.md) (`DeepSeek-V4-Flash-0731/deepseek_v4_vllm`)
- [DeepSeek V4 Flash 0731: vLLM, DP attention, EP, parallel streams serialized](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams.md) (`DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/moe_sum
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/moe_sum/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/moe_sum/configs

## Torch reference

What the kind computes, as Torch code (`profiling/runners/moe/moe_sum_reference.py` in ServingStudio Sim):

```python
"""Independent value reference for DeepSeek's routed-expert reduction."""

from typing import Any


def moe_sum_reference(torch: Any, input_tensor: Any) -> Any:
    if input_tensor.ndim != 3:
        raise ValueError("input_tensor must have shape [tokens, top_k, hidden_dim]")
    if input_tensor.dtype is not torch.bfloat16:
        raise TypeError("input_tensor must use bfloat16")
    if not input_tensor.is_contiguous() or any(size <= 0 for size in input_tensor.shape):
        raise ValueError("input_tensor must be non-empty contiguous storage")
    return input_tensor.float().sum(dim=1).to(torch.bfloat16)


__all__ = ["moe_sum_reference"]
```

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/moe_sum.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
