# MiMo V2.6 Flash RL: vLLM, TP attention, EP MoE | ServingStudio Models

Deployment `MiMo-V2.6-Flash-RL/mimo_v2_vllm_fp8_swa_moe` of MiMo V2.6 Flash RL (`XiaomiMiMo/MiMo-V2.6-Flash-RL`, family MiMo). Arch `mimo_v2_vllm_fp8_swa_moe`; Whole iteration; GPU NVIDIA B200. Interactive page, with its cost tree and live prediction: https://servingstudio.cs.washington.edu/models.html?preset=MiMo-V2.6-Flash-RL%2Fmimo_v2_vllm_fp8_swa_moe. All models: https://servingstudio.cs.washington.edu/models.md.

## Configurations

The preset sweeps `workload`; each configuration (member) is one value of each.

| Parameters | GPUs per replica | Case form | Groups | max_model_len | Can be timed |
| --- | --- | --- | --- | --- | --- |
| workload diverse_100 | 4 | `iter` | 1 | 131,072 | yes |

## Timing a batch

`POST https://servingstudio.cs.washington.edu/api/public/v1/predict` with the preset id, one configuration's `params` and 1 to 64 `cases`. A case is `{"groups": [...]}`, one object per group (the table's Groups column), each with the fields of the configuration's case form. The answer gives each case's time per section and per node of its cost tree. The servingstudio-timing-predict skill walks through it: https://servingstudio.cs.washington.edu/skills/servingstudio-timing-predict.md.

```json
{"preset":"MiMo-V2.6-Flash-RL/mimo_v2_vllm_fp8_swa_moe","params":{"workload":"diverse_100"},"cases":["…"]}
```

Fields of a group for case form `iter`:

| Field | Type | Required | Meaning |
| --- | --- | --- | --- |
| `prefill_chunk_pairs` | int_pair_list | no | Prefill requests, one `[prefix_len, append_len]` each: the tokens already in the KV cache and the tokens this iteration computes (`prefix_len` 0 for a fresh prompt). |
| `decode_kv_lens` | int_list | no | Decode requests, one KV length each: the context its new token attends to. Give this or `decode_count`, not both. |
| `decode_count` | int | no | Shorthand for `decode_kv_lens`: this many decode requests, each `average_decode_length` long. |
| `average_decode_length` | int | no | The KV length of each `decode_count` request; required with it. |

## Kernels of configuration workload diverse_100

Each section's leaf slots, in order: the kernel kind each calls and the backends measured for it. Other configurations share the shape but not every size.

### Section `iter`

| Slot | Kernel | Backends |
| --- | --- | --- |
| `unified.embed` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.embed.ar_norm` | [all_reduce_residual_rms_norm](https://servingstudio.cs.washington.edu/kernels/all_reduce_residual_rms_norm.md) | `flashinfer_mnnvl` |
| `unified.embed.ar_norm_all_reduce` | [all_reduce](https://servingstudio.cs.washington.edu/kernels/all_reduce.md) | `nccl` |
| `unified.embed.ar_norm_residual_norm` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.global_dense.attn.qkv_proj.input_quant` | [fp8_per_token_group_quant](https://servingstudio.cs.washington.edu/kernels/fp8_per_token_group_quant.md) | `vllm_cuda` |
| `unified.global_dense.attn.qkv_proj.gemm` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `deepgemm` |
| `unified.global_dense.attn.rope` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.global_dense.attn.kv_append` | [kv_cache_append](https://servingstudio.cs.washington.edu/kernels/kv_cache_append.md) | `vllm_triton_diffkv` |
| `unified.global_dense.attn.attention` | [fa4_varlen_attention](https://servingstudio.cs.washington.edu/kernels/fa4_varlen_attention.md) | `vllm_fa4` |
| `unified.global_dense.attn.o_proj` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.global_dense.attn.ar_norm` | [all_reduce_residual_rms_norm](https://servingstudio.cs.washington.edu/kernels/all_reduce_residual_rms_norm.md) | `flashinfer_mnnvl` |
| `unified.global_dense.attn.ar_norm_all_reduce` | [all_reduce](https://servingstudio.cs.washington.edu/kernels/all_reduce.md) | `nccl` |
| `unified.global_dense.attn.ar_norm_residual_norm` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.global_dense.ffn.gate_up_input_quant` | [fp8_per_token_group_quant](https://servingstudio.cs.washington.edu/kernels/fp8_per_token_group_quant.md) | `vllm_cuda` |
| `unified.global_dense.ffn.gate_up` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `deepgemm` |
| `unified.global_dense.ffn.act_and_mul` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.global_dense.ffn.down_input_quant` | [fp8_per_token_group_quant](https://servingstudio.cs.washington.edu/kernels/fp8_per_token_group_quant.md) | `vllm_cuda` |
| `unified.global_dense.ffn.down` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `deepgemm` |
| `unified.global_dense.ffn.ar_norm` | [all_reduce_residual_rms_norm](https://servingstudio.cs.washington.edu/kernels/all_reduce_residual_rms_norm.md) | `flashinfer_mnnvl` |
| `unified.global_dense.ffn.ar_norm_all_reduce` | [all_reduce](https://servingstudio.cs.washington.edu/kernels/all_reduce.md) | `nccl` |
| `unified.global_dense.ffn.ar_norm_residual_norm` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.swa_moe.attn.qkv_proj.input_quant` | [fp8_per_token_group_quant](https://servingstudio.cs.washington.edu/kernels/fp8_per_token_group_quant.md) | `vllm_cuda` |
| `unified.swa_moe.attn.qkv_proj.gemm` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `deepgemm` |
| `unified.swa_moe.attn.rope` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.swa_moe.attn.kv_append` | [kv_cache_append](https://servingstudio.cs.washington.edu/kernels/kv_cache_append.md) | `vllm_triton_diffkv` |
| `unified.swa_moe.attn.attention` | [fa4_varlen_attention](https://servingstudio.cs.washington.edu/kernels/fa4_varlen_attention.md) | `vllm_fa4` |
| `unified.swa_moe.attn.o_proj` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.swa_moe.attn.ar_norm` | [all_reduce_residual_rms_norm](https://servingstudio.cs.washington.edu/kernels/all_reduce_residual_rms_norm.md) | `flashinfer_mnnvl` |
| `unified.swa_moe.attn.ar_norm_all_reduce` | [all_reduce](https://servingstudio.cs.washington.edu/kernels/all_reduce.md) | `nccl` |
| `unified.swa_moe.attn.ar_norm_residual_norm` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.swa_moe.moe.router` | [gemm_fp32_output](https://servingstudio.cs.washington.edu/kernels/gemm_fp32_output.md) | `torch_cublas` |
| `unified.swa_moe.moe.routed_rank0.fused_moe` | [nvfp4_fused_moe](https://servingstudio.cs.washington.edu/kernels/nvfp4_fused_moe.md) | `flashinfer_trtllm_sm100_mxfp4_bf16` |
| `unified.swa_moe.moe.routed_rank1.fused_moe` | [nvfp4_fused_moe](https://servingstudio.cs.washington.edu/kernels/nvfp4_fused_moe.md) | `flashinfer_trtllm_sm100_mxfp4_bf16` |
| `unified.swa_moe.moe.routed_rank2.fused_moe` | [nvfp4_fused_moe](https://servingstudio.cs.washington.edu/kernels/nvfp4_fused_moe.md) | `flashinfer_trtllm_sm100_mxfp4_bf16` |
| `unified.swa_moe.moe.routed_rank3.fused_moe` | [nvfp4_fused_moe](https://servingstudio.cs.washington.edu/kernels/nvfp4_fused_moe.md) | `flashinfer_trtllm_sm100_mxfp4_bf16` |
| `unified.swa_moe.moe.ar_norm` | [all_reduce_residual_rms_norm](https://servingstudio.cs.washington.edu/kernels/all_reduce_residual_rms_norm.md) | `flashinfer_mnnvl` |
| `unified.swa_moe.moe.ar_norm_all_reduce` | [all_reduce](https://servingstudio.cs.washington.edu/kernels/all_reduce.md) | `nccl` |
| `unified.swa_moe.moe.ar_norm_residual_norm` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.global_moe.attn.qkv_proj.input_quant` | [fp8_per_token_group_quant](https://servingstudio.cs.washington.edu/kernels/fp8_per_token_group_quant.md) | `vllm_cuda` |
| `unified.global_moe.attn.qkv_proj.gemm` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `deepgemm` |
| `unified.global_moe.attn.rope` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.global_moe.attn.kv_append` | [kv_cache_append](https://servingstudio.cs.washington.edu/kernels/kv_cache_append.md) | `vllm_triton_diffkv` |
| `unified.global_moe.attn.attention` | [fa4_varlen_attention](https://servingstudio.cs.washington.edu/kernels/fa4_varlen_attention.md) | `vllm_fa4` |
| `unified.global_moe.attn.o_proj` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.global_moe.attn.ar_norm` | [all_reduce_residual_rms_norm](https://servingstudio.cs.washington.edu/kernels/all_reduce_residual_rms_norm.md) | `flashinfer_mnnvl` |
| `unified.global_moe.attn.ar_norm_all_reduce` | [all_reduce](https://servingstudio.cs.washington.edu/kernels/all_reduce.md) | `nccl` |
| `unified.global_moe.attn.ar_norm_residual_norm` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.global_moe.moe.router` | [gemm_fp32_output](https://servingstudio.cs.washington.edu/kernels/gemm_fp32_output.md) | `torch_cublas` |
| `unified.global_moe.moe.routed_rank0.fused_moe` | [nvfp4_fused_moe](https://servingstudio.cs.washington.edu/kernels/nvfp4_fused_moe.md) | `flashinfer_trtllm_sm100_mxfp4_bf16` |
| `unified.global_moe.moe.routed_rank1.fused_moe` | [nvfp4_fused_moe](https://servingstudio.cs.washington.edu/kernels/nvfp4_fused_moe.md) | `flashinfer_trtllm_sm100_mxfp4_bf16` |
| `unified.global_moe.moe.routed_rank2.fused_moe` | [nvfp4_fused_moe](https://servingstudio.cs.washington.edu/kernels/nvfp4_fused_moe.md) | `flashinfer_trtllm_sm100_mxfp4_bf16` |
| `unified.global_moe.moe.routed_rank3.fused_moe` | [nvfp4_fused_moe](https://servingstudio.cs.washington.edu/kernels/nvfp4_fused_moe.md) | `flashinfer_trtllm_sm100_mxfp4_bf16` |
| `unified.global_moe.moe.ar_norm` | [all_reduce_residual_rms_norm](https://servingstudio.cs.washington.edu/kernels/all_reduce_residual_rms_norm.md) | `flashinfer_mnnvl` |
| `unified.global_moe.moe.ar_norm_all_reduce` | [all_reduce](https://servingstudio.cs.washington.edu/kernels/all_reduce.md) | `nccl` |
| `unified.global_moe.moe.ar_norm_residual_norm` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.logits_select` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.lm_head` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.logits_all_gather` | [all_reduce](https://servingstudio.cs.washington.edu/kernels/all_reduce.md) | `nccl` |

## Simulations that run it

- [`MiMo-V2.6-Flash-RL/mimo_v2_vllm_fp8_swa_moe_chunked_prefill`](https://servingstudio.cs.washington.edu/simulate.html?preset=MiMo-V2.6-Flash-RL%2Fmimo_v2_vllm_fp8_swa_moe_chunked_prefill): unified deployment, 2 configurations

## API

- This deployment in the model list: https://servingstudio.cs.washington.edu/api/public/v1/models
- The kernels of one configuration: https://servingstudio.cs.washington.edu/api/public/v1/models/MiMo-V2.6-Flash-RL/mimo_v2_vllm_fp8_swa_moe/tree?workload=diverse_100

---

This page as Markdown: https://servingstudio.cs.washington.edu/models/MiMo-V2.6-Flash-RL/mimo_v2_vllm_fp8_swa_moe.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
