# Qwen3.6 35B-A3B FP8: Single GPU | ServingStudio Models

Deployment `Qwen3.6-35B-A3B-FP8/qwen36_local` of Qwen3.6 35B-A3B FP8 (`Qwen/Qwen3.6-35B-A3B-FP8`, family Qwen). Arch `qwen36_local`; Whole iteration; GPU NVIDIA H200. Interactive page, with its cost tree and live prediction: https://servingstudio.cs.washington.edu/models.html?preset=Qwen3.6-35B-A3B-FP8%2Fqwen36_local. All models: https://servingstudio.cs.washington.edu/models.md.

## Configurations

The preset sweeps `workload`; each configuration (member) is one value of each.

| Parameters | GPUs per replica | Case form | Groups | max_model_len | Can be timed |
| --- | --- | --- | --- | --- | --- |
| workload ctx8k_out1k | 1 | `iter` | 1 | 262,144 | yes |

## Timing a batch

`POST https://servingstudio.cs.washington.edu/api/public/v1/predict` with the preset id, one configuration's `params` and 1 to 64 `cases`. A case is `{"groups": [...]}`, one object per group (the table's Groups column), each with the fields of the configuration's case form. The answer gives each case's time per section and per node of its cost tree. The servingstudio-timing-predict skill walks through it: https://servingstudio.cs.washington.edu/skills/servingstudio-timing-predict.md.

```json
{"preset":"Qwen3.6-35B-A3B-FP8/qwen36_local","params":{"workload":"ctx8k_out1k"},"cases":["…"]}
```

Fields of a group for case form `iter`:

| Field | Type | Required | Meaning |
| --- | --- | --- | --- |
| `prefill_chunk_pairs` | int_pair_list | no | Prefill requests, one `[prefix_len, append_len]` each: the tokens already in the KV cache and the tokens this iteration computes (`prefix_len` 0 for a fresh prompt). |
| `decode_kv_lens` | int_list | no | Decode requests, one KV length each: the context its new token attends to. Give this or `decode_count`, not both. |
| `decode_count` | int | no | Shorthand for `decode_kv_lens`: this many decode requests, each `average_decode_length` long. |
| `average_decode_length` | int | no | The KV length of each `decode_count` request; required with it. |

## Kernels of configuration workload ctx8k_out1k

Each section's leaf slots, in order: the kernel kind each calls and the backends measured for it. Other configurations share the shape but not every size.

### Section `iter`

| Slot | Kernel | Backends |
| --- | --- | --- |
| `unified.embedding` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.gdn.input_add_rms_norm` | [residual_rms_norm](https://servingstudio.cs.washington.edu/kernels/residual_rms_norm.md) | `vllm_cuda` |
| `unified.gdn.qkvz.input_quant` | [fp8_per_token_group_quant](https://servingstudio.cs.washington.edu/kernels/fp8_per_token_group_quant.md) | `vllm_cuda` |
| `unified.gdn.qkvz.gemm` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `deepgemm` |
| `unified.gdn.ba` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear` |
| `unified.gdn.split_b` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.gdn.split_a` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.gdn.core_output_zero` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.gdn.state_gather` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.gdn.state_zero` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.gdn.prefill.causal_conv` | [gdn_causal_conv_prefill](https://servingstudio.cs.washington.edu/kernels/gdn_causal_conv_prefill.md) | `vllm_triton` |
| `unified.gdn.prefill.post_conv` | [gdn_prefill_post_conv](https://servingstudio.cs.washington.edu/kernels/gdn_prefill_post_conv.md) | `vllm_triton` |
| `unified.gdn.prefill.chunk_delta_rule` | [gdn_chunk_delta_rule](https://servingstudio.cs.washington.edu/kernels/gdn_chunk_delta_rule.md) | `flashinfer` |
| `unified.gdn.state_scatter` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.gdn.decode.causal_conv` | [gdn_causal_conv_decode](https://servingstudio.cs.washington.edu/kernels/gdn_causal_conv_decode.md) | `vllm_triton` |
| `unified.gdn.decode.recurrent` | [gdn_recurrent_decode](https://servingstudio.cs.washington.edu/kernels/gdn_recurrent_decode.md) | `vllm_triton` |
| `unified.gdn.core_output_copy` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.gdn.gated_norm` | [gdn_gated_rms_norm](https://servingstudio.cs.washington.edu/kernels/gdn_gated_rms_norm.md) | `vllm_triton` |
| `unified.gdn.out_proj.input_quant` | [fp8_per_token_group_quant](https://servingstudio.cs.washington.edu/kernels/fp8_per_token_group_quant.md) | `vllm_cuda` |
| `unified.gdn.out_proj.gemm` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `deepgemm` |
| `unified.gdn.post_attention_add_rms_norm` | [residual_rms_norm](https://servingstudio.cs.washington.edu/kernels/residual_rms_norm.md) | `vllm_cuda` |
| `unified.router.router.gemm` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear` |
| `unified.router.topk` | [moe_fused_topk](https://servingstudio.cs.washington.edu/kernels/moe_fused_topk.md) | `vllm_cuda` |
| `unified.router.align` | [moe_align_block_size](https://servingstudio.cs.washington.edu/kernels/moe_align_block_size.md) | `vllm_cuda` |
| `unified.routed_expert.gate_up.input_quant` | [fp8_per_token_group_quant](https://servingstudio.cs.washington.edu/kernels/fp8_per_token_group_quant.md) | `vllm_cuda` |
| `unified.routed_expert.gate_up.gemm` | [vllm_fused_moe](https://servingstudio.cs.washington.edu/kernels/vllm_fused_moe.md) | `vllm_triton` |
| `unified.routed_expert.activation` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.routed_expert.down.input_quant` | [fp8_per_token_group_quant](https://servingstudio.cs.washington.edu/kernels/fp8_per_token_group_quant.md) | `vllm_cuda` |
| `unified.routed_expert.down.gemm` | [vllm_fused_moe](https://servingstudio.cs.washington.edu/kernels/vllm_fused_moe.md) | `vllm_triton` |
| `unified.shared_expert.gate_up.input_quant` | [fp8_per_token_group_quant](https://servingstudio.cs.washington.edu/kernels/fp8_per_token_group_quant.md) | `vllm_cuda` |
| `unified.shared_expert.gate_up.gemm` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `deepgemm` |
| `unified.shared_expert.silu_and_mul` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.shared_expert.down.input_quant` | [fp8_per_token_group_quant](https://servingstudio.cs.washington.edu/kernels/fp8_per_token_group_quant.md) | `vllm_cuda` |
| `unified.shared_expert.down.gemm` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `deepgemm` |
| `unified.shared_expert.shared_gate` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear` |
| `unified.shared_expert.shared_gate_sigmoid` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `torch` |
| `unified.shared_expert.apply_shared_gate` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `torch` |
| `unified.finalize.finalize` | [moe_finalize_routing](https://servingstudio.cs.washington.edu/kernels/moe_finalize_routing.md) | `flashinfer_trtllm` |
| `unified.finalize.shared_routed_add` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.gated_gqa.input_add_rms_norm` | [residual_rms_norm](https://servingstudio.cs.washington.edu/kernels/residual_rms_norm.md) | `vllm_cuda` |
| `unified.gated_gqa.qkv_gate.input_quant` | [fp8_per_token_group_quant](https://servingstudio.cs.washington.edu/kernels/fp8_per_token_group_quant.md) | `vllm_cuda` |
| `unified.gated_gqa.qkv_gate.gemm` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `deepgemm` |
| `unified.gated_gqa.q_norm` | [rms_norm](https://servingstudio.cs.washington.edu/kernels/rms_norm.md) | `flashinfer` |
| `unified.gated_gqa.k_norm` | [rms_norm](https://servingstudio.cs.washington.edu/kernels/rms_norm.md) | `flashinfer` |
| `unified.gated_gqa.partial_rope` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.gated_gqa.attention.kv_cache_append` | [kv_cache_append](https://servingstudio.cs.washington.edu/kernels/kv_cache_append.md) | `vllm_cuda` |
| `unified.gated_gqa.attention.prefill` | [flashinfer_attn_prefill](https://servingstudio.cs.washington.edu/kernels/flashinfer_attn_prefill.md) | `fa2`, `fa3` |
| `unified.gated_gqa.attention.decode` | [flashinfer_attn_decode](https://servingstudio.cs.washington.edu/kernels/flashinfer_attn_decode.md) | `fa2`, `fa3` |
| `unified.gated_gqa.output_gate` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.gated_gqa.out_proj.input_quant` | [fp8_per_token_group_quant](https://servingstudio.cs.washington.edu/kernels/fp8_per_token_group_quant.md) | `vllm_cuda` |
| `unified.gated_gqa.out_proj.gemm` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `deepgemm` |
| `unified.gated_gqa.post_attention_add_rms_norm` | [residual_rms_norm](https://servingstudio.cs.washington.edu/kernels/residual_rms_norm.md) | `vllm_cuda` |
| `unified.router.router.gemm` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear` |
| `unified.router.topk` | [moe_fused_topk](https://servingstudio.cs.washington.edu/kernels/moe_fused_topk.md) | `vllm_cuda` |
| `unified.router.align` | [moe_align_block_size](https://servingstudio.cs.washington.edu/kernels/moe_align_block_size.md) | `vllm_cuda` |
| `unified.routed_expert.gate_up.input_quant` | [fp8_per_token_group_quant](https://servingstudio.cs.washington.edu/kernels/fp8_per_token_group_quant.md) | `vllm_cuda` |
| `unified.routed_expert.gate_up.gemm` | [vllm_fused_moe](https://servingstudio.cs.washington.edu/kernels/vllm_fused_moe.md) | `vllm_triton` |
| `unified.routed_expert.activation` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.routed_expert.down.input_quant` | [fp8_per_token_group_quant](https://servingstudio.cs.washington.edu/kernels/fp8_per_token_group_quant.md) | `vllm_cuda` |
| `unified.routed_expert.down.gemm` | [vllm_fused_moe](https://servingstudio.cs.washington.edu/kernels/vllm_fused_moe.md) | `vllm_triton` |
| `unified.shared_expert.gate_up.input_quant` | [fp8_per_token_group_quant](https://servingstudio.cs.washington.edu/kernels/fp8_per_token_group_quant.md) | `vllm_cuda` |
| `unified.shared_expert.gate_up.gemm` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `deepgemm` |
| `unified.shared_expert.silu_and_mul` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.shared_expert.down.input_quant` | [fp8_per_token_group_quant](https://servingstudio.cs.washington.edu/kernels/fp8_per_token_group_quant.md) | `vllm_cuda` |
| `unified.shared_expert.down.gemm` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `deepgemm` |
| `unified.shared_expert.shared_gate` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear` |
| `unified.shared_expert.shared_gate_sigmoid` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `torch` |
| `unified.shared_expert.apply_shared_gate` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `torch` |
| `unified.finalize.finalize` | [moe_finalize_routing](https://servingstudio.cs.washington.edu/kernels/moe_finalize_routing.md) | `flashinfer_trtllm` |
| `unified.finalize.shared_routed_add` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.head.final_add_rms_norm` | [residual_rms_norm](https://servingstudio.cs.washington.edu/kernels/residual_rms_norm.md) | `vllm_cuda` |
| `unified.head.lm_head` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear` |

## Simulations that run it

- [`Qwen3.6-35B-A3B-FP8/qwen36_local_barebone`](https://servingstudio.cs.washington.edu/simulate.html?preset=Qwen3.6-35B-A3B-FP8%2Fqwen36_local_barebone): unified deployment, 4 configurations

## API

- This deployment in the model list: https://servingstudio.cs.washington.edu/api/public/v1/models
- The kernels of one configuration: https://servingstudio.cs.washington.edu/api/public/v1/models/Qwen3.6-35B-A3B-FP8/qwen36_local/tree?workload=ctx8k_out1k

---

This page as Markdown: https://servingstudio.cs.washington.edu/models/Qwen3.6-35B-A3B-FP8/qwen36_local.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
