# Simulate | ServingStudio

Send traffic to a deployment and read the latency and throughput its clients would see.

Interactive page: https://servingstudio.cs.washington.edu/simulate.html. The page reads its lists from ServingStudio Sim's public API at https://servingstudio.cs.washington.edu/api/public/v1 (OpenAPI: https://servingstudio.cs.washington.edu/api/public/v1/openapi.json). An agent can use it through the [servingstudio-simulate](https://servingstudio.cs.washington.edu/skills/servingstudio-simulate.md) skill.

This text was generated when the site was built, from Sim commit `db977adc881595c5b48aa070bddc7cd6f33db130`; the API has the current lists.

A simulation replays a stream of requests through one deployment and reports time to first token (TTFT), time per output token (TPOT), end-to-end latency and throughput. 32 deployments can be simulated. Start one with `POST /api/public/v1/simulate` and read it from `/api/public/v1/simulations/{simulation_id}`. At most 2 simulations run at once and 32 wait; a run may use up to 2,000 requests and is stopped after 10 minutes.

## Deployments

### Llama 3.1 8B

- [`Llama-3.1-8B/llama3_dense_tp_barebone`](https://servingstudio.cs.washington.edu/simulate.html?preset=Llama-3.1-8B%2Fllama3_dense_tp_barebone): unified deployment; main: [TP](https://servingstudio.cs.washington.edu/models/Llama-3.1-8B/llama3_dense_tp.md) on NVIDIA H200, worker `barebone`; sweeps `tp_size` 1, 2, 4, 8; `replicas` 1, 2, 4, 8; 10 configurations; captures: `diverse_100` (100 requests, mean 20,617 prompt and 1,352 output tokens), `ctx8k_out1k/glm52_fp8` (256 requests, mean 8,192 prompt and 1,024 output tokens), `balanced_c32` (256 requests, mean 1,856 prompt and 192 output tokens), `c32_long` (64 requests, mean 1,856 prompt and 192 output tokens), `ctx8k_out1k/qwen36_35b_a3b_fp8` (256 requests, mean 8,192 prompt and 1,024 output tokens), `quadrant_c48` (192 requests, mean 5,512 prompt and 1,600 output tokens), `shape_grid_128` (128 requests, mean 4,080 prompt and 424 output tokens)
- [`Llama-3.1-8B/llama3_dense_tp_pd`](https://servingstudio.cs.washington.edu/simulate.html?preset=Llama-3.1-8B%2Fllama3_dense_tp_pd): pd deployment; prefill: [TP](https://servingstudio.cs.washington.edu/models/Llama-3.1-8B/llama3_dense_tp.md) on NVIDIA H200, worker `pd_prefill`, decode: [TP](https://servingstudio.cs.washington.edu/models/Llama-3.1-8B/llama3_dense_tp.md) on NVIDIA H200, worker `pd_decode`; sweeps `prefill_tp` 1, 2, 4, 8; `prefill_replicas` 1, 2, 4; `decode_tp` 1, 2, 4, 8; `decode_replicas` 1, 2, 4, 8, 16, 32; 162 configurations; captures: `diverse_100` (100 requests, mean 20,617 prompt and 1,352 output tokens), `ctx8k_out1k/glm52_fp8` (256 requests, mean 8,192 prompt and 1,024 output tokens), `balanced_c32` (256 requests, mean 1,856 prompt and 192 output tokens), `c32_long` (64 requests, mean 1,856 prompt and 192 output tokens), `ctx8k_out1k/qwen36_35b_a3b_fp8` (256 requests, mean 8,192 prompt and 1,024 output tokens), `quadrant_c48` (192 requests, mean 5,512 prompt and 1,600 output tokens), `shape_grid_128` (128 requests, mean 4,080 prompt and 424 output tokens)

### Qwen3 235B-A22B

- [`Qwen3-235B-A22B/qwen3_moe_dp_attn_ep_ffn_hp_unified`](https://servingstudio.cs.washington.edu/simulate.html?preset=Qwen3-235B-A22B%2Fqwen3_moe_dp_attn_ep_ffn_hp_unified): unified deployment; main: [DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B/qwen3_moe_dp_attn_ep_ffn.md) on NVIDIA H200, worker `hp_unified`; sweeps `ep_size` 8, 32; `replicas` 1, 2; 4 configurations; captures: `ctx8k_out1k` (256 requests, mean 8,192 prompt and 1,024 output tokens)

### Qwen3 235B-A22B FP8

- [`Qwen3-235B-A22B-FP8/qwen3_moe_fp8_dp_attn_ep_ffn_hp_unified`](https://servingstudio.cs.washington.edu/simulate.html?preset=Qwen3-235B-A22B-FP8%2Fqwen3_moe_fp8_dp_attn_ep_ffn_hp_unified): unified deployment; main: [DP attention, EP MoE, DeepGEMM experts](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_moe_fp8_dp_attn_ep_ffn.md) on NVIDIA H200, worker `hp_unified`; sweeps `replicas` 1, 2; 2 configurations; captures: `ctx8k_out1k` (256 requests, mean 8,192 prompt and 1,024 output tokens)
- [`Qwen3-235B-A22B-FP8/qwen3_vllm_moe_dp_attn_ep_ffn_hp_unified`](https://servingstudio.cs.washington.edu/simulate.html?preset=Qwen3-235B-A22B-FP8%2Fqwen3_vllm_moe_dp_attn_ep_ffn_hp_unified): unified deployment; main: [vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_vllm_moe_dp_attn_ep_ffn.md) on NVIDIA H200, worker `hp_unified`; sweeps `replicas` 1, 2; 2 configurations; captures: `ctx8k_out1k` (256 requests, mean 8,192 prompt and 1,024 output tokens)

### Qwen3 235B-A22B Thinking 2507 FP8

- [`Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_moe_fp8_dp_attn_ep_ffn_hp_unified`](https://servingstudio.cs.washington.edu/simulate.html?preset=Qwen3-235B-A22B-Thinking-2507-FP8%2Fqwen3_moe_fp8_dp_attn_ep_ffn_hp_unified): unified deployment; main: [DP attention, EP MoE, DeepGEMM experts](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_moe_fp8_dp_attn_ep_ffn.md) on NVIDIA H200, worker `hp_unified`; sweeps `replicas` 1, 2; 2 configurations; captures: `ctx8k_out1k` (256 requests, mean 8,192 prompt and 1,024 output tokens)
- [`Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_vllm_moe_dp_attn_ep_ffn_hp_unified`](https://servingstudio.cs.washington.edu/simulate.html?preset=Qwen3-235B-A22B-Thinking-2507-FP8%2Fqwen3_vllm_moe_dp_attn_ep_ffn_hp_unified): unified deployment; main: [vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_vllm_moe_dp_attn_ep_ffn.md) on NVIDIA H200, worker `hp_unified`; sweeps `replicas` 1, 2; 2 configurations; captures: `ctx8k_out1k` (256 requests, mean 8,192 prompt and 1,024 output tokens)

### Qwen3.6 35B-A3B FP8

- [`Qwen3.6-35B-A3B-FP8/qwen36_local_barebone`](https://servingstudio.cs.washington.edu/simulate.html?preset=Qwen3.6-35B-A3B-FP8%2Fqwen36_local_barebone): unified deployment; main: [Single GPU](https://servingstudio.cs.washington.edu/models/Qwen3.6-35B-A3B-FP8/qwen36_local.md) on NVIDIA H200, worker `barebone`; sweeps `replicas` 1, 2, 4, 8; 4 configurations; captures: `ctx8k_out1k` (256 requests, mean 8,192 prompt and 1,024 output tokens)

### Qwen3.8 27B NVFP4

- [`Qwen3.8-27B-NVFP4/qwen35_vllm_nvfp4_gdn_dense_chunked_prefill`](https://servingstudio.cs.washington.edu/simulate.html?preset=Qwen3.8-27B-NVFP4%2Fqwen35_vllm_nvfp4_gdn_dense_chunked_prefill): unified deployment; main: [vLLM, one GPU](https://servingstudio.cs.washington.edu/models/Qwen3.8-27B-NVFP4/qwen35_vllm_nvfp4_gdn_dense.md) on NVIDIA B200, worker `chunked_prefill`; sweeps `replicas` 1, 2; 2 configurations; captures: `diverse_100` (100 requests, mean 20,617 prompt and 1,352 output tokens), `ctx8k_out1k/glm52_fp8` (256 requests, mean 8,192 prompt and 1,024 output tokens), `balanced_c32` (256 requests, mean 1,856 prompt and 192 output tokens), `c32_long` (64 requests, mean 1,856 prompt and 192 output tokens), `ctx8k_out1k/qwen36_35b_a3b_fp8` (256 requests, mean 8,192 prompt and 1,024 output tokens), `quadrant_c48` (192 requests, mean 5,512 prompt and 1,600 output tokens), `shape_grid_128` (128 requests, mean 4,080 prompt and 424 output tokens)

### Qwen3.8 Flash Next NVFP4

- [`Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative`](https://servingstudio.cs.washington.edu/simulate.html?preset=Qwen3.8-Flash-Next-NVFP4%2Fqwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative): unified deployment; main: [vLLM, one GPU, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative.md) on NVIDIA B200, worker `speculative`; sweeps `replicas` 1, 2; 2 configurations; captures: `diverse_100` (100 requests, mean 20,617 prompt and 1,352 output tokens)

### DeepSeek V4 Flash 0731

- [`DeepSeek-V4-Flash-0731/deepseek_v4_vllm_chunked_prefill`](https://servingstudio.cs.washington.edu/simulate.html?preset=DeepSeek-V4-Flash-0731%2Fdeepseek_v4_vllm_chunked_prefill): unified deployment; main: [vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm.md) on NVIDIA H200, worker `chunked_prefill`; sweeps `replicas` 1, 2; 2 configurations; captures: `shape_grid_128` (128 requests, mean 4,080 prompt and 424 output tokens)
- [`DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams_chunked_prefill`](https://servingstudio.cs.washington.edu/simulate.html?preset=DeepSeek-V4-Flash-0731%2Fdeepseek_v4_vllm_serial_streams_chunked_prefill): unified deployment; main: [vLLM, DP attention, EP, parallel streams serialized](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams.md) on NVIDIA H200, worker `chunked_prefill`; sweeps `replicas` 1, 2; 2 configurations; captures: `shape_grid_128` (128 requests, mean 4,080 prompt and 424 output tokens)

### GLM-5.2 FP8

- [`GLM-5.2-FP8/glm52_vllm_dsa_moe_hp_unified`](https://servingstudio.cs.washington.edu/simulate.html?preset=GLM-5.2-FP8%2Fglm52_vllm_dsa_moe_hp_unified): unified deployment; main: [vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-FP8/glm52_vllm_dsa_moe.md) on NVIDIA H200, worker `hp_unified`; sweeps `replicas` 1, 2; 2 configurations; captures: `ctx8k_out1k` (256 requests, mean 8,192 prompt and 1,024 output tokens)

### GLM-5.2 NVFP4

- [`GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe_chunked_prefill`](https://servingstudio.cs.washington.edu/simulate.html?preset=GLM-5.2-NVFP4%2Fglm52_sglang_nvfp4_tp_dsa_moe_chunked_prefill): unified deployment; main: [SGLang, TP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe.md) on NVIDIA B200, worker `chunked_prefill`; sweeps `replicas` 1, 2; `max_model_len` 8,192, 65,536, 131,072, 262,144, 524,288; 10 configurations; captures: `diverse_100` (100 requests, mean 20,617 prompt and 1,352 output tokens)
- [`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe_chunked_prefill`](https://servingstudio.cs.washington.edu/simulate.html?preset=GLM-5.2-NVFP4%2Fglm52_vllm_nvfp4_dp_attn_dsa_moe_chunked_prefill): unified deployment; main: [vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe.md) on NVIDIA B200, worker `chunked_prefill`; sweeps `replicas` 1, 2; `max_model_len` 8,192, 65,536, 131,072, 262,144, 524,288; `layout` dp4ep4, dp8ep8; 18 configurations; captures: `c32_long` (64 requests, mean 1,856 prompt and 192 output tokens)
- [`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_chunked_prefill`](https://servingstudio.cs.washington.edu/simulate.html?preset=GLM-5.2-NVFP4%2Fglm52_vllm_nvfp4_dsa_moe_chunked_prefill): unified deployment; main: [vLLM, TP and EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe.md) on NVIDIA B200, worker `chunked_prefill`; sweeps `replicas` 1, 2; `server` ep4_ctx8k, ep4_ctx64k, ep4_ctx128k, ep4_ctx256k, ep4_ctx512k, ep8_ctx8k, ep8_ctx64k, ep8_ctx128k, ep8_ctx256k, ep8_ctx512k; 20 configurations; captures: `c32_long` (64 requests, mean 1,856 prompt and 192 output tokens), `diverse_100` (100 requests, mean 20,617 prompt and 1,352 output tokens)
- [`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative`](https://servingstudio.cs.washington.edu/simulate.html?preset=GLM-5.2-NVFP4%2Fglm52_vllm_nvfp4_dsa_moe_speculative): unified deployment; main: [vLLM, TP and EP, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative.md) on NVIDIA B200, worker `speculative`; sweeps `replicas` 1, 2; `server` ctx8k, ctx16k, ctx64k, ctx128k, ctx128k_chunk8k, ctx256k, ctx512k; 14 configurations; captures: `diverse_100` (100 requests, mean 20,617 prompt and 1,352 output tokens), `diverse_100_tp8` (100 requests, mean 20,617 prompt and 1,352 output tokens), `quadrant_c48` (192 requests, mean 5,512 prompt and 1,600 output tokens)
- [`GLM-5.2-NVFP4/glm52_vllm_nvfp4_pp_dsa_moe_pipeline_chunked_prefill`](https://servingstudio.cs.washington.edu/simulate.html?preset=GLM-5.2-NVFP4%2Fglm52_vllm_nvfp4_pp_dsa_moe_pipeline_chunked_prefill): pp deployment; stage: [vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_pp_dsa_moe.md) on NVIDIA B200, worker `pipeline_chunked_prefill`; sweeps `replicas` 1, 2; `pp_size` 4, 8; `max_model_len` 8,192, 65,536, 131,072, 262,144, 524,288; 20 configurations; captures: `diverse_100` (100 requests, mean 20,617 prompt and 1,352 output tokens)

### GLM-5.3 NVFP4

- [`GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2_speculative`](https://servingstudio.cs.washington.edu/simulate.html?preset=GLM-5.3-NVFP4%2Fglm53_vllm_nvfp4_dsa_moe_dflash2_speculative): unified deployment; main: [vLLM, TP and EP, DFlash speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2.md) on NVIDIA B200, worker `speculative`; sweeps `replicas` 1, 2; `server` ctx8k, ctx8k_chunk4k, ctx16k, ctx16k_chunk8k, ctx128k; 10 configurations; captures: `diverse_100` (100 requests, mean 20,617 prompt and 1,352 output tokens)

### GLM-5.3-Flash

- [`GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe_chunked_prefill`](https://servingstudio.cs.washington.edu/simulate.html?preset=GLM-5.3-Flash%2Fglm53_flash_vllm_fp8_dp_attn_ep_moe_chunked_prefill): unified deployment; main: [vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe.md) on NVIDIA B200, worker `chunked_prefill`; sweeps `replicas` 1, 2; `ep_size` 4, 8; `max_model_len` 8,192, 131,072, 524,288; 12 configurations; captures: `balanced_c32` (256 requests, mean 1,856 prompt and 192 output tokens)
- [`GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe_chunked_prefill`](https://servingstudio.cs.washington.edu/simulate.html?preset=GLM-5.3-Flash%2Fglm53_flash_vllm_fp8_kda_dsa_moe_chunked_prefill): unified deployment; main: [vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe.md) on NVIDIA B200, worker `chunked_prefill`; sweeps `replicas` 1, 2; `server` ctx8k, ctx128k, ctx512k; 6 configurations; captures: `diverse_100` (100 requests, mean 20,617 prompt and 1,352 output tokens), `balanced_c32` (256 requests, mean 1,856 prompt and 192 output tokens)
- [`GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe_pipeline_chunked_prefill`](https://servingstudio.cs.washington.edu/simulate.html?preset=GLM-5.3-Flash%2Fglm53_flash_vllm_fp8_pp_kda_dsa_moe_pipeline_chunked_prefill): pp deployment; stage: [vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe.md) on NVIDIA B200, worker `pipeline_chunked_prefill`; sweeps `replicas` 1, 2; `pp_size` 4, 8; `max_model_len` 8,192, 131,072, 524,288; 12 configurations; captures: `balanced_c32` (256 requests, mean 1,856 prompt and 192 output tokens)

### GLM-5.3-Flash NVFP4

- [`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe_chunked_prefill`](https://servingstudio.cs.washington.edu/simulate.html?preset=GLM-5.3-Flash-NVFP4%2Fglm53_flash_vllm_nvfp4_dp_attn_ep_moe_chunked_prefill): unified deployment; main: [vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe.md) on NVIDIA B200, worker `chunked_prefill`; sweeps `replicas` 1, 2; `ep_size` 4, 8; `max_model_len` 8,192, 131,072, 524,288; 12 configurations; captures: `balanced_c32` (256 requests, mean 1,856 prompt and 192 output tokens)
- [`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe_chunked_prefill`](https://servingstudio.cs.washington.edu/simulate.html?preset=GLM-5.3-Flash-NVFP4%2Fglm53_flash_vllm_nvfp4_kda_dsa_moe_chunked_prefill): unified deployment; main: [vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe.md) on NVIDIA B200, worker `chunked_prefill`; sweeps `replicas` 1, 2; `tp_size` 4, 8; `enable_expert_parallel` on, off; `max_model_len` 8,192, 131,072, 524,288; 24 configurations; captures: `balanced_c32` (256 requests, mean 1,856 prompt and 192 output tokens)
- [`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe_pipeline_chunked_prefill`](https://servingstudio.cs.washington.edu/simulate.html?preset=GLM-5.3-Flash-NVFP4%2Fglm53_flash_vllm_nvfp4_pp_kda_dsa_moe_pipeline_chunked_prefill): pp deployment; stage: [vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe.md) on NVIDIA B200, worker `pipeline_chunked_prefill`; sweeps `replicas` 1, 2; `pp_size` 4, 8; `max_model_len` 8,192, 131,072, 524,288; 12 configurations; captures: `balanced_c32` (256 requests, mean 1,856 prompt and 192 output tokens)

### DeepSeek V4.1 Flash

- [`DeepSeek-V4.1-Flash/deepseek_v41_vllm_chunked_prefill`](https://servingstudio.cs.washington.edu/simulate.html?preset=DeepSeek-V4.1-Flash%2Fdeepseek_v41_vllm_chunked_prefill): unified deployment; main: [vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash/deepseek_v41_vllm.md) on NVIDIA B200, worker `chunked_prefill`; sweeps `replicas` 1, 2; `server` ctx128k, ctx1m; 4 configurations; captures: `diverse_100` (100 requests, mean 20,617 prompt and 1,352 output tokens)
- [`DeepSeek-V4.1-Flash/deepseek_v41_vllm_serial_streams_chunked_prefill`](https://servingstudio.cs.washington.edu/simulate.html?preset=DeepSeek-V4.1-Flash%2Fdeepseek_v41_vllm_serial_streams_chunked_prefill): unified deployment; main: [vLLM, TP attention, EP MoE, side streams serialized](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash/deepseek_v41_vllm_serial_streams.md) on NVIDIA B200, worker `chunked_prefill`; sweeps `replicas` 1, 2; `server` ctx128k, ctx1m; 4 configurations; captures: `diverse_100` (100 requests, mean 20,617 prompt and 1,352 output tokens)

### DeepSeek V4.1 Flash 1017

- [`DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dp_attn_ep_moe_dspark_speculative`](https://servingstudio.cs.washington.edu/simulate.html?preset=DeepSeek-V4.1-Flash-1017%2Fdeepseek_v41_vllm_dp_attn_ep_moe_dspark_speculative): unified deployment; main: [vLLM, DP attention, EP MoE, DSpark speculative decoding](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dp_attn_ep_moe_dspark.md) on NVIDIA B200, worker `speculative`; sweeps `replicas` 1, 2; 2 configurations; captures: `diverse_100` (100 requests, mean 20,617 prompt and 1,352 output tokens)
- [`DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dspark_speculative`](https://servingstudio.cs.washington.edu/simulate.html?preset=DeepSeek-V4.1-Flash-1017%2Fdeepseek_v41_vllm_dspark_speculative): unified deployment; main: [vLLM, TP attention, EP MoE, DSpark speculative decoding](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dspark.md) on NVIDIA B200, worker `speculative`; sweeps `replicas` 1, 2; 2 configurations; captures: `diverse_100` (100 requests, mean 20,617 prompt and 1,352 output tokens)

### Inkling NVFP4

- [`Inkling-NVFP4/inkling_vllm_nvfp4_rel_attn_moe_chunked_prefill`](https://servingstudio.cs.washington.edu/simulate.html?preset=Inkling-NVFP4%2Finkling_vllm_nvfp4_rel_attn_moe_chunked_prefill): unified deployment; main: [vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Inkling-NVFP4/inkling_vllm_nvfp4_rel_attn_moe.md) on NVIDIA B200, worker `chunked_prefill`; sweeps `replicas` 1, 2; `server` ctx128k, ctx1m; 4 configurations; captures: `diverse_100` (100 requests, mean 20,617 prompt and 1,352 output tokens)

### Inkling Small NVFP4

- [`Inkling-Small-NVFP4/inkling_vllm_nvfp4_rel_attn_moe_chunked_prefill`](https://servingstudio.cs.washington.edu/simulate.html?preset=Inkling-Small-NVFP4%2Finkling_vllm_nvfp4_rel_attn_moe_chunked_prefill): unified deployment; main: [vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Inkling-Small-NVFP4/inkling_vllm_nvfp4_rel_attn_moe.md) on NVIDIA B200, worker `chunked_prefill`; sweeps `replicas` 1, 2; 2 configurations; captures: `diverse_100` (100 requests, mean 20,617 prompt and 1,352 output tokens)

### MiMo V2.6 Flash RL

- [`MiMo-V2.6-Flash-RL/mimo_v2_vllm_fp8_swa_moe_chunked_prefill`](https://servingstudio.cs.washington.edu/simulate.html?preset=MiMo-V2.6-Flash-RL%2Fmimo_v2_vllm_fp8_swa_moe_chunked_prefill): unified deployment; main: [vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/MiMo-V2.6-Flash-RL/mimo_v2_vllm_fp8_swa_moe.md) on NVIDIA B200, worker `chunked_prefill`; sweeps `replicas` 1, 2; 2 configurations; captures: `diverse_100` (100 requests, mean 20,617 prompt and 1,352 output tokens)

## Workloads

A simulation's requests come from one of three sources:

- `capture`: A recorded trace a member offers: /simulations/presets lists each preset's captures. Its requests and its routing are the capture's.
- `generated`: A trace req-frontend's tracegen draws from `generator`: `type` and the generator's arguments, each a string as its flag takes it.
- `upload`: A CSV you send to POST /workloads (body: the file). Its format and tags are the ones its header fits, or the query's `format` and `tags` (comma-separated); the answer's workload_id is the simulation's workload.upload.

### Generator `synthetic`

Draw a session workload from distributions, with no corpus at all. Arguments:

- `sessions` (default `100`): How many sessions to draw
- `rounds` (default `uniform:1..8`): Rounds per session. `4`, `uniform:1..12`, `lognormal:3,0.9`
- `input_len` (default `lognormal:1024,0.8`): Fresh input tokens per round — the part not carried from the previous round. `512`, `uniform:256..2048`, `lognormal:1024,0.8`
- `output_len` (default `lognormal:256,0.7`): Output tokens per round
- `tool_wait_ms` (default `lognormal:500,1.0`): Milliseconds a session waits between one round completing and the next being submitted
- `compaction_probability` (default `0`): Chance that a round drops its carried context instead of reusing it, standing in for a compaction. Round 0 never carries anything, so this applies from round 1 onward
- `arrival_rate` (default `1`): Synthetic session arrival rate, in sessions per second
- `arrival_pattern` (default `poisson`; one of `poisson`, `constant`): Synthetic session arrival process
- `seed` (default `0`): Seed for every draw here. Same seed, same trace, byte for byte

### Upload formats

An uploaded CSV may be up to 1024 KiB and is kept for 24 hours.

- `text-generation-independent`: columns `id`, `arrival_time`, `input_len`, `output_len`; optional tags `session`, `slo`, `priority`, `speculative`
- `text-generation-session-execution-v2`: columns `request_id`, `session_id`, `round_idx`, `arrival_time_ms`, `prefix_len`, `input_len`, `output_len`, `tool_wait_after_ms`; optional tags `slo`, `priority`, `speculative`
- tag `session` adds columns `session_id`, `prefix_kv`, `tool_wait_after_ms`
- tag `slo` adds columns `ttft_slo_ms`, `tpot_slo_ms`, `e2e_slo_ms`
- tag `priority` adds columns `priority`
- tag `speculative` adds columns `accept_rate`

An MoE member routes every workload's tokens as one of its captures does: workload.capture names it, its first by default.

---

This page as Markdown: https://servingstudio.cs.washington.edu/simulate.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
