Simulate | ServingStudio
Send traffic to a deployment and read the latency and throughput its clients would see.
Interactive page: https://servingstudio.cs.washington.edu/simulate.html. The page reads its lists from ServingStudio Sim's public API at https://servingstudio.cs.washington.edu/api/public/v1 (OpenAPI: https://servingstudio.cs.washington.edu/api/public/v1/openapi.json). An agent can use it through the servingstudio-simulate skill.
This text was generated when the site was built, from Sim commit db977adc881595c5b48aa070bddc7cd6f33db130; the API has the current lists.
A simulation replays a stream of requests through one deployment and reports time to first token (TTFT), time per output token (TPOT), end-to-end latency and throughput. 32 deployments can be simulated. Start one with POST /api/public/v1/simulate and read it from /api/public/v1/simulations/{simulation_id}. At most 2 simulations run at once and 32 wait; a run may use up to 2,000 requests and is stopped after 10 minutes.
Deployments
Llama 3.1 8B
Llama-3.1-8B/llama3_dense_tp_barebone: unified deployment; main: TP on NVIDIA H200, workerbarebone; sweepstp_size1, 2, 4, 8;replicas1, 2, 4, 8; 10 configurations; captures:diverse_100(100 requests, mean 20,617 prompt and 1,352 output tokens),ctx8k_out1k/glm52_fp8(256 requests, mean 8,192 prompt and 1,024 output tokens),balanced_c32(256 requests, mean 1,856 prompt and 192 output tokens),c32_long(64 requests, mean 1,856 prompt and 192 output tokens),ctx8k_out1k/qwen36_35b_a3b_fp8(256 requests, mean 8,192 prompt and 1,024 output tokens),quadrant_c48(192 requests, mean 5,512 prompt and 1,600 output tokens),shape_grid_128(128 requests, mean 4,080 prompt and 424 output tokens)Llama-3.1-8B/llama3_dense_tp_pd: pd deployment; prefill: TP on NVIDIA H200, workerpd_prefill, decode: TP on NVIDIA H200, workerpd_decode; sweepsprefill_tp1, 2, 4, 8;prefill_replicas1, 2, 4;decode_tp1, 2, 4, 8;decode_replicas1, 2, 4, 8, 16, 32; 162 configurations; captures:diverse_100(100 requests, mean 20,617 prompt and 1,352 output tokens),ctx8k_out1k/glm52_fp8(256 requests, mean 8,192 prompt and 1,024 output tokens),balanced_c32(256 requests, mean 1,856 prompt and 192 output tokens),c32_long(64 requests, mean 1,856 prompt and 192 output tokens),ctx8k_out1k/qwen36_35b_a3b_fp8(256 requests, mean 8,192 prompt and 1,024 output tokens),quadrant_c48(192 requests, mean 5,512 prompt and 1,600 output tokens),shape_grid_128(128 requests, mean 4,080 prompt and 424 output tokens)
Qwen3 235B-A22B
Qwen3-235B-A22B/qwen3_moe_dp_attn_ep_ffn_hp_unified: unified deployment; main: DP attention, EP MoE on NVIDIA H200, workerhp_unified; sweepsep_size8, 32;replicas1, 2; 4 configurations; captures:ctx8k_out1k(256 requests, mean 8,192 prompt and 1,024 output tokens)
Qwen3 235B-A22B FP8
Qwen3-235B-A22B-FP8/qwen3_moe_fp8_dp_attn_ep_ffn_hp_unified: unified deployment; main: DP attention, EP MoE, DeepGEMM experts on NVIDIA H200, workerhp_unified; sweepsreplicas1, 2; 2 configurations; captures:ctx8k_out1k(256 requests, mean 8,192 prompt and 1,024 output tokens)Qwen3-235B-A22B-FP8/qwen3_vllm_moe_dp_attn_ep_ffn_hp_unified: unified deployment; main: vLLM, DP attention, EP MoE on NVIDIA H200, workerhp_unified; sweepsreplicas1, 2; 2 configurations; captures:ctx8k_out1k(256 requests, mean 8,192 prompt and 1,024 output tokens)
Qwen3 235B-A22B Thinking 2507 FP8
Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_moe_fp8_dp_attn_ep_ffn_hp_unified: unified deployment; main: DP attention, EP MoE, DeepGEMM experts on NVIDIA H200, workerhp_unified; sweepsreplicas1, 2; 2 configurations; captures:ctx8k_out1k(256 requests, mean 8,192 prompt and 1,024 output tokens)Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_vllm_moe_dp_attn_ep_ffn_hp_unified: unified deployment; main: vLLM, DP attention, EP MoE on NVIDIA H200, workerhp_unified; sweepsreplicas1, 2; 2 configurations; captures:ctx8k_out1k(256 requests, mean 8,192 prompt and 1,024 output tokens)
Qwen3.6 35B-A3B FP8
Qwen3.6-35B-A3B-FP8/qwen36_local_barebone: unified deployment; main: Single GPU on NVIDIA H200, workerbarebone; sweepsreplicas1, 2, 4, 8; 4 configurations; captures:ctx8k_out1k(256 requests, mean 8,192 prompt and 1,024 output tokens)
Qwen3.8 27B NVFP4
Qwen3.8-27B-NVFP4/qwen35_vllm_nvfp4_gdn_dense_chunked_prefill: unified deployment; main: vLLM, one GPU on NVIDIA B200, workerchunked_prefill; sweepsreplicas1, 2; 2 configurations; captures:diverse_100(100 requests, mean 20,617 prompt and 1,352 output tokens),ctx8k_out1k/glm52_fp8(256 requests, mean 8,192 prompt and 1,024 output tokens),balanced_c32(256 requests, mean 1,856 prompt and 192 output tokens),c32_long(64 requests, mean 1,856 prompt and 192 output tokens),ctx8k_out1k/qwen36_35b_a3b_fp8(256 requests, mean 8,192 prompt and 1,024 output tokens),quadrant_c48(192 requests, mean 5,512 prompt and 1,600 output tokens),shape_grid_128(128 requests, mean 4,080 prompt and 424 output tokens)
Qwen3.8 Flash Next NVFP4
Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative: unified deployment; main: vLLM, one GPU, MTP speculative decoding on NVIDIA B200, workerspeculative; sweepsreplicas1, 2; 2 configurations; captures:diverse_100(100 requests, mean 20,617 prompt and 1,352 output tokens)
DeepSeek V4 Flash 0731
DeepSeek-V4-Flash-0731/deepseek_v4_vllm_chunked_prefill: unified deployment; main: vLLM, DP attention, EP on NVIDIA H200, workerchunked_prefill; sweepsreplicas1, 2; 2 configurations; captures:shape_grid_128(128 requests, mean 4,080 prompt and 424 output tokens)DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams_chunked_prefill: unified deployment; main: vLLM, DP attention, EP, parallel streams serialized on NVIDIA H200, workerchunked_prefill; sweepsreplicas1, 2; 2 configurations; captures:shape_grid_128(128 requests, mean 4,080 prompt and 424 output tokens)
GLM-5.2 FP8
GLM-5.2-FP8/glm52_vllm_dsa_moe_hp_unified: unified deployment; main: vLLM, DP attention, EP on NVIDIA H200, workerhp_unified; sweepsreplicas1, 2; 2 configurations; captures:ctx8k_out1k(256 requests, mean 8,192 prompt and 1,024 output tokens)
GLM-5.2 NVFP4
GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe_chunked_prefill: unified deployment; main: SGLang, TP on NVIDIA B200, workerchunked_prefill; sweepsreplicas1, 2;max_model_len8,192, 65,536, 131,072, 262,144, 524,288; 10 configurations; captures:diverse_100(100 requests, mean 20,617 prompt and 1,352 output tokens)GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe_chunked_prefill: unified deployment; main: vLLM, DP attention, EP MoE on NVIDIA B200, workerchunked_prefill; sweepsreplicas1, 2;max_model_len8,192, 65,536, 131,072, 262,144, 524,288;layoutdp4ep4, dp8ep8; 18 configurations; captures:c32_long(64 requests, mean 1,856 prompt and 192 output tokens)GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_chunked_prefill: unified deployment; main: vLLM, TP and EP on NVIDIA B200, workerchunked_prefill; sweepsreplicas1, 2;serverep4_ctx8k, ep4_ctx64k, ep4_ctx128k, ep4_ctx256k, ep4_ctx512k, ep8_ctx8k, ep8_ctx64k, ep8_ctx128k, ep8_ctx256k, ep8_ctx512k; 20 configurations; captures:c32_long(64 requests, mean 1,856 prompt and 192 output tokens),diverse_100(100 requests, mean 20,617 prompt and 1,352 output tokens)GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative: unified deployment; main: vLLM, TP and EP, MTP speculative decoding on NVIDIA B200, workerspeculative; sweepsreplicas1, 2;serverctx8k, ctx16k, ctx64k, ctx128k, ctx128k_chunk8k, ctx256k, ctx512k; 14 configurations; captures:diverse_100(100 requests, mean 20,617 prompt and 1,352 output tokens),diverse_100_tp8(100 requests, mean 20,617 prompt and 1,352 output tokens),quadrant_c48(192 requests, mean 5,512 prompt and 1,600 output tokens)GLM-5.2-NVFP4/glm52_vllm_nvfp4_pp_dsa_moe_pipeline_chunked_prefill: pp deployment; stage: vLLM, PP on NVIDIA B200, workerpipeline_chunked_prefill; sweepsreplicas1, 2;pp_size4, 8;max_model_len8,192, 65,536, 131,072, 262,144, 524,288; 20 configurations; captures:diverse_100(100 requests, mean 20,617 prompt and 1,352 output tokens)
GLM-5.3 NVFP4
GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2_speculative: unified deployment; main: vLLM, TP and EP, DFlash speculative decoding on NVIDIA B200, workerspeculative; sweepsreplicas1, 2;serverctx8k, ctx8k_chunk4k, ctx16k, ctx16k_chunk8k, ctx128k; 10 configurations; captures:diverse_100(100 requests, mean 20,617 prompt and 1,352 output tokens)
GLM-5.3-Flash
GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe_chunked_prefill: unified deployment; main: vLLM, DP attention, EP MoE on NVIDIA B200, workerchunked_prefill; sweepsreplicas1, 2;ep_size4, 8;max_model_len8,192, 131,072, 524,288; 12 configurations; captures:balanced_c32(256 requests, mean 1,856 prompt and 192 output tokens)GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe_chunked_prefill: unified deployment; main: vLLM, TP attention, EP or TP MoE on NVIDIA B200, workerchunked_prefill; sweepsreplicas1, 2;serverctx8k, ctx128k, ctx512k; 6 configurations; captures:diverse_100(100 requests, mean 20,617 prompt and 1,352 output tokens),balanced_c32(256 requests, mean 1,856 prompt and 192 output tokens)GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe_pipeline_chunked_prefill: pp deployment; stage: vLLM, PP on NVIDIA B200, workerpipeline_chunked_prefill; sweepsreplicas1, 2;pp_size4, 8;max_model_len8,192, 131,072, 524,288; 12 configurations; captures:balanced_c32(256 requests, mean 1,856 prompt and 192 output tokens)
GLM-5.3-Flash NVFP4
GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe_chunked_prefill: unified deployment; main: vLLM, DP attention, EP MoE on NVIDIA B200, workerchunked_prefill; sweepsreplicas1, 2;ep_size4, 8;max_model_len8,192, 131,072, 524,288; 12 configurations; captures:balanced_c32(256 requests, mean 1,856 prompt and 192 output tokens)GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe_chunked_prefill: unified deployment; main: vLLM, TP attention, EP or TP MoE on NVIDIA B200, workerchunked_prefill; sweepsreplicas1, 2;tp_size4, 8;enable_expert_parallelon, off;max_model_len8,192, 131,072, 524,288; 24 configurations; captures:balanced_c32(256 requests, mean 1,856 prompt and 192 output tokens)GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe_pipeline_chunked_prefill: pp deployment; stage: vLLM, PP on NVIDIA B200, workerpipeline_chunked_prefill; sweepsreplicas1, 2;pp_size4, 8;max_model_len8,192, 131,072, 524,288; 12 configurations; captures:balanced_c32(256 requests, mean 1,856 prompt and 192 output tokens)
DeepSeek V4.1 Flash
DeepSeek-V4.1-Flash/deepseek_v41_vllm_chunked_prefill: unified deployment; main: vLLM, TP attention, EP MoE on NVIDIA B200, workerchunked_prefill; sweepsreplicas1, 2;serverctx128k, ctx1m; 4 configurations; captures:diverse_100(100 requests, mean 20,617 prompt and 1,352 output tokens)DeepSeek-V4.1-Flash/deepseek_v41_vllm_serial_streams_chunked_prefill: unified deployment; main: vLLM, TP attention, EP MoE, side streams serialized on NVIDIA B200, workerchunked_prefill; sweepsreplicas1, 2;serverctx128k, ctx1m; 4 configurations; captures:diverse_100(100 requests, mean 20,617 prompt and 1,352 output tokens)
DeepSeek V4.1 Flash 1017
DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dp_attn_ep_moe_dspark_speculative: unified deployment; main: vLLM, DP attention, EP MoE, DSpark speculative decoding on NVIDIA B200, workerspeculative; sweepsreplicas1, 2; 2 configurations; captures:diverse_100(100 requests, mean 20,617 prompt and 1,352 output tokens)DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dspark_speculative: unified deployment; main: vLLM, TP attention, EP MoE, DSpark speculative decoding on NVIDIA B200, workerspeculative; sweepsreplicas1, 2; 2 configurations; captures:diverse_100(100 requests, mean 20,617 prompt and 1,352 output tokens)
Inkling NVFP4
Inkling-NVFP4/inkling_vllm_nvfp4_rel_attn_moe_chunked_prefill: unified deployment; main: vLLM, TP attention, EP MoE on NVIDIA B200, workerchunked_prefill; sweepsreplicas1, 2;serverctx128k, ctx1m; 4 configurations; captures:diverse_100(100 requests, mean 20,617 prompt and 1,352 output tokens)
Inkling Small NVFP4
Inkling-Small-NVFP4/inkling_vllm_nvfp4_rel_attn_moe_chunked_prefill: unified deployment; main: vLLM, TP attention, EP MoE on NVIDIA B200, workerchunked_prefill; sweepsreplicas1, 2; 2 configurations; captures:diverse_100(100 requests, mean 20,617 prompt and 1,352 output tokens)
MiMo V2.6 Flash RL
MiMo-V2.6-Flash-RL/mimo_v2_vllm_fp8_swa_moe_chunked_prefill: unified deployment; main: vLLM, TP attention, EP MoE on NVIDIA B200, workerchunked_prefill; sweepsreplicas1, 2; 2 configurations; captures:diverse_100(100 requests, mean 20,617 prompt and 1,352 output tokens)
Workloads
A simulation's requests come from one of three sources:
capture: A recorded trace a member offers: /simulations/presets lists each preset's captures. Its requests and its routing are the capture's.generated: A trace req-frontend's tracegen draws fromgenerator:typeand the generator's arguments, each a string as its flag takes it.upload: A CSV you send to POST /workloads (body: the file). Its format and tags are the ones its header fits, or the query'sformatandtags(comma-separated); the answer's workload_id is the simulation's workload.upload.
Generator synthetic
Draw a session workload from distributions, with no corpus at all. Arguments:
sessions(default100): How many sessions to drawrounds(defaultuniform:1..8): Rounds per session.4,uniform:1..12,lognormal:3,0.9input_len(defaultlognormal:1024,0.8): Fresh input tokens per round — the part not carried from the previous round.512,uniform:256..2048,lognormal:1024,0.8output_len(defaultlognormal:256,0.7): Output tokens per roundtool_wait_ms(defaultlognormal:500,1.0): Milliseconds a session waits between one round completing and the next being submittedcompaction_probability(default0): Chance that a round drops its carried context instead of reusing it, standing in for a compaction. Round 0 never carries anything, so this applies from round 1 onwardarrival_rate(default1): Synthetic session arrival rate, in sessions per secondarrival_pattern(defaultpoisson; one ofpoisson,constant): Synthetic session arrival processseed(default0): Seed for every draw here. Same seed, same trace, byte for byte
Upload formats
An uploaded CSV may be up to 1024 KiB and is kept for 24 hours.
text-generation-independent: columnsid,arrival_time,input_len,output_len; optional tagssession,slo,priority,speculativetext-generation-session-execution-v2: columnsrequest_id,session_id,round_idx,arrival_time_ms,prefix_len,input_len,output_len,tool_wait_after_ms; optional tagsslo,priority,speculative- tag
sessionadds columnssession_id,prefix_kv,tool_wait_after_ms - tag
sloadds columnsttft_slo_ms,tpot_slo_ms,e2e_slo_ms - tag
priorityadds columnspriority - tag
speculativeadds columnsaccept_rate
An MoE member routes every workload's tokens as one of its captures does: workload.capture names it, its first by default.
This page as Markdown: https://servingstudio.cs.washington.edu/simulate.md. A guide to the whole site for AI agents: llms.txt.