# Qwen3 235B-A22B Thinking 2507 FP8: DP attention, EP MoE, DeepGEMM experts | ServingStudio Models

Deployment `Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_moe_fp8_dp_attn_ep_ffn` of Qwen3 235B-A22B Thinking 2507 FP8 (`Qwen/Qwen3-235B-A22B-Thinking-2507-FP8`, family Qwen). Arch `qwen3_moe_fp8_dp_attn_ep_ffn`; Whole iteration; GPU NVIDIA H200. Interactive page, with its cost tree and live prediction: https://servingstudio.cs.washington.edu/models.html?preset=Qwen3-235B-A22B-Thinking-2507-FP8%2Fqwen3_moe_fp8_dp_attn_ep_ffn. All models: https://servingstudio.cs.washington.edu/models.md.

## Configurations

The preset sweeps `workload`; each configuration (member) is one value of each.

| Parameters | GPUs per replica | Case form | Groups | max_model_len | Can be timed |
| --- | --- | --- | --- | --- | --- |
| workload ctx8k_out1k | 4 | `iter` | 1 | 40,960 | yes |
| workload uniform | 4 | `iter` | 1 | 40,960 | yes |

## Timing a batch

`POST https://servingstudio.cs.washington.edu/api/public/v1/predict` with the preset id, one configuration's `params` and 1 to 64 `cases`. A case is `{"groups": [...]}`, one object per group (the table's Groups column), each with the fields of the configuration's case form. The answer gives each case's time per section and per node of its cost tree. The servingstudio-timing-predict skill walks through it: https://servingstudio.cs.washington.edu/skills/servingstudio-timing-predict.md.

```json
{"preset":"Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_moe_fp8_dp_attn_ep_ffn","params":{"workload":"ctx8k_out1k"},"cases":["…"]}
```

Fields of a group for case form `iter`:

| Field | Type | Required | Meaning |
| --- | --- | --- | --- |
| `prefill_chunk_pairs` | int_pair_list | no | Prefill requests, one `[prefix_len, append_len]` each: the tokens already in the KV cache and the tokens this iteration computes (`prefix_len` 0 for a fresh prompt). |
| `decode_kv_lens` | int_list | no | Decode requests, one KV length each: the context its new token attends to. Give this or `decode_count`, not both. |
| `decode_count` | int | no | Shorthand for `decode_kv_lens`: this many decode requests, each `average_decode_length` long. |
| `average_decode_length` | int | no | The KV length of each `decode_count` request; required with it. |

## Kernels of configuration workload ctx8k_out1k

Each section's leaf slots, in order: the kernel kind each calls and the backends measured for it. Other configurations share the shape but not every size.

### Section `iter`

| Slot | Kernel | Backends |
| --- | --- | --- |
| `unified.embedding` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.attn_block.input_norm` | [rms_norm](https://servingstudio.cs.washington.edu/kernels/rms_norm.md) | `flashinfer` |
| `unified.attn_block.qkv_proj.input_quant` | [fp8_per_token_group_quant](https://servingstudio.cs.washington.edu/kernels/fp8_per_token_group_quant.md) | `vllm_cuda` |
| `unified.attn_block.qkv_proj.gemm` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `deepgemm` |
| `unified.attn_block.attn.kv_cache_append` | [kv_cache_append](https://servingstudio.cs.washington.edu/kernels/kv_cache_append.md) | `vllm_cuda` |
| `unified.attn_block.attn.prefill` | [flashinfer_attn_prefill](https://servingstudio.cs.washington.edu/kernels/flashinfer_attn_prefill.md) | `fa3` |
| `unified.attn_block.attn.decode` | [flashinfer_attn_decode](https://servingstudio.cs.washington.edu/kernels/flashinfer_attn_decode.md) | `fa2` |
| `unified.attn_block.o_proj.input_quant` | [fp8_per_token_group_quant](https://servingstudio.cs.washington.edu/kernels/fp8_per_token_group_quant.md) | `vllm_cuda` |
| `unified.attn_block.o_proj.gemm` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `deepgemm` |
| `unified.attn_block.tp_allreduce` | [all_reduce](https://servingstudio.cs.washington.edu/kernels/all_reduce.md) | `nvshmem` |
| `unified.moe_router.post_attn_norm` | [rms_norm](https://servingstudio.cs.washington.edu/kernels/rms_norm.md) | `flashinfer` |
| `unified.moe_router.router_gemm.input_quant` | [fp8_per_token_group_quant](https://servingstudio.cs.washington.edu/kernels/fp8_per_token_group_quant.md) | `vllm_cuda` |
| `unified.moe_router.router_gemm.gemm` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `deepgemm` |
| `unified.moe_dispatch.dispatch_inter` | [p2p_inter](https://servingstudio.cs.washington.edu/kernels/p2p_inter.md) | `nvshmem` |
| `unified.moe_dispatch.dispatch_intra` | [p2p_intra](https://servingstudio.cs.washington.edu/kernels/p2p_intra.md) | `nvshmem` |
| `unified.moe_expert_compute.gate_up` | [grouped_gemm](https://servingstudio.cs.washington.edu/kernels/grouped_gemm.md) | `deepgemm` |
| `unified.moe_expert_compute.activation` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.moe_expert_compute.down` | [grouped_gemm](https://servingstudio.cs.washington.edu/kernels/grouped_gemm.md) | `deepgemm` |
| `unified.moe_expert_compute.gate_up` | [grouped_gemm](https://servingstudio.cs.washington.edu/kernels/grouped_gemm.md) | `deepgemm` |
| `unified.moe_expert_compute.activation` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.moe_expert_compute.down` | [grouped_gemm](https://servingstudio.cs.washington.edu/kernels/grouped_gemm.md) | `deepgemm` |
| `unified.moe_expert_compute.gate_up` | [grouped_gemm](https://servingstudio.cs.washington.edu/kernels/grouped_gemm.md) | `deepgemm` |
| `unified.moe_expert_compute.activation` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.moe_expert_compute.down` | [grouped_gemm](https://servingstudio.cs.washington.edu/kernels/grouped_gemm.md) | `deepgemm` |
| `unified.moe_expert_compute.gate_up` | [grouped_gemm](https://servingstudio.cs.washington.edu/kernels/grouped_gemm.md) | `deepgemm` |
| `unified.moe_expert_compute.activation` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.moe_expert_compute.down` | [grouped_gemm](https://servingstudio.cs.washington.edu/kernels/grouped_gemm.md) | `deepgemm` |
| `unified.moe_local_reduce` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.moe_combine.combine_intra_reduce` | [p2p_intra](https://servingstudio.cs.washington.edu/kernels/p2p_intra.md) | `nvshmem` |
| `unified.moe_combine.combine_inter_reduce` | [p2p_inter](https://servingstudio.cs.washington.edu/kernels/p2p_inter.md) | `nvshmem` |
| `unified.moe_combine.combine_inter_bcast` | [p2p_inter](https://servingstudio.cs.washington.edu/kernels/p2p_inter.md) | `nvshmem` |
| `unified.moe_combine.combine_intra_fanout` | [p2p_intra](https://servingstudio.cs.washington.edu/kernels/p2p_intra.md) | `nvshmem` |
| `unified.final_norm` | [rms_norm](https://servingstudio.cs.washington.edu/kernels/rms_norm.md) | `flashinfer` |
| `unified.lm_head.input_quant` | [fp8_per_token_group_quant](https://servingstudio.cs.washington.edu/kernels/fp8_per_token_group_quant.md) | `vllm_cuda` |
| `unified.lm_head.gemm` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `deepgemm` |

## Simulations that run it

- [`Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_moe_fp8_dp_attn_ep_ffn_hp_unified`](https://servingstudio.cs.washington.edu/simulate.html?preset=Qwen3-235B-A22B-Thinking-2507-FP8%2Fqwen3_moe_fp8_dp_attn_ep_ffn_hp_unified): unified deployment, 2 configurations

## API

- This deployment in the model list: https://servingstudio.cs.washington.edu/api/public/v1/models
- The kernels of one configuration: https://servingstudio.cs.washington.edu/api/public/v1/models/Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_moe_fp8_dp_attn_ep_ffn/tree?workload=ctx8k_out1k

---

This page as Markdown: https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_moe_fp8_dp_attn_ep_ffn.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
