Models | ServingStudio
Each model runs as an execution graph, which the simulator costs as a tree of measured kernels. Pick a model and how it is deployed to see how an iteration's time is put together, then time a batch of your own.
Interactive page: https://servingstudio.cs.washington.edu/models.html. The page reads its lists from ServingStudio Sim's public API at https://servingstudio.cs.washington.edu/api/public/v1 (OpenAPI: https://servingstudio.cs.washington.edu/api/public/v1/openapi.json). An agent can use it through the servingstudio-timing-predict skill.
This text was generated when the site was built, from Sim commit db977adc881595c5b48aa070bddc7cd6f33db130; the API has the current lists.
ServingStudio publishes 20 models. Each is listed with its deployments (public presets): how the model is split over GPUs, the GPU, the parameters the preset sweeps, and how many of its configurations could be timed when this text was generated. On the interactive page, a deployment shows its cost tree and times a batch you give it (POST /api/public/v1/predict).
Llama 3.1 8B
Family: Llama. Checkpoint: meta-llama/Llama-3.1-8B.
- Single GPU:
Llama-3.1-8B/llama3_dense; NVIDIA H200; 1 of 1 configuration can be timed; details: Markdown - TP:
Llama-3.1-8B/llama3_dense_tp; NVIDIA H200; sweepstp_size1, 2, 4, 8; 4 of 4 configurations can be timed; details: Markdown - DP attention, TP FFN:
Llama-3.1-8B/llama3_dp_attn_tp_ffn; NVIDIA H200; 1 of 1 configuration can be timed; details: Markdown
Qwen3 235B-A22B
Family: Qwen. Checkpoint: Qwen/Qwen3-235B-A22B.
- Attention side of an attention–FFN split:
Qwen3-235B-A22B/qwen3_attn_tp; NVIDIA H200; sweepsattn_tp_size2, 4; 2 of 2 configurations can be timed; details: Markdown - FFN side of an attention–FFN split:
Qwen3-235B-A22B/qwen3_ffn_moe; NVIDIA H200; sweepsworkloaduniform, random; 2 of 2 configurations can be timed; details: Markdown - DP attention, EP MoE:
Qwen3-235B-A22B/qwen3_moe_dp_attn_ep_ffn; NVIDIA H200; sweepsep_size8, 32;workloadctx8k_out1k, uniform; 4 of 4 configurations can be timed; details: Markdown
Qwen3 235B-A22B FP8
Family: Qwen. Checkpoint: Qwen/Qwen3-235B-A22B-FP8.
- Attention side of an attention–FFN split:
Qwen3-235B-A22B-FP8/qwen3_attn_tp; NVIDIA H200; sweepsattn_tp_size2, 4; 2 of 2 configurations can be timed; details: Markdown - FFN side of an attention–FFN split:
Qwen3-235B-A22B-FP8/qwen3_fp8_ffn_moe; NVIDIA H200; sweepsattn_tp_size2, 4;ep_size4, 8; 3 of 3 configurations can be timed; details: Markdown - DP attention, EP MoE, DeepGEMM experts:
Qwen3-235B-A22B-FP8/qwen3_moe_fp8_dp_attn_ep_ffn; NVIDIA H200; sweepsattn_tp_size1, 2, 4;ep_size4, 8;workloadctx8k_out1k, uniform; 10 of 10 configurations can be timed; details: Markdown - vLLM, DP attention, EP MoE:
Qwen3-235B-A22B-FP8/qwen3_vllm_moe_dp_attn_ep_ffn; NVIDIA H200; sweepsworkloadctx8k_out1k, uniform; 2 of 2 configurations can be timed; details: Markdown
Qwen3 235B-A22B Thinking 2507 FP8
Family: Qwen. Checkpoint: Qwen/Qwen3-235B-A22B-Thinking-2507-FP8.
- DP attention, EP MoE, DeepGEMM experts:
Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_moe_fp8_dp_attn_ep_ffn; NVIDIA H200; sweepsworkloadctx8k_out1k, uniform; 2 of 2 configurations can be timed; details: Markdown - vLLM, DP attention, EP MoE:
Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_vllm_moe_dp_attn_ep_ffn; NVIDIA H200; sweepsworkloadctx8k_out1k, uniform; 2 of 2 configurations can be timed; details: Markdown
Qwen3 Coder 480B-A35B
Family: Qwen. Checkpoint: Qwen/Qwen3-Coder-480B-A35B-Instruct.
- Attention side of an attention–FFN split:
Qwen3-Coder-480B-A35B-Instruct/qwen3_attn_tp; NVIDIA H200; 1 of 1 configuration can be timed; details: Markdown - FFN side of an attention–FFN split:
Qwen3-Coder-480B-A35B-Instruct/qwen3_fp8_ffn_moe; NVIDIA H200; sweepsattn_tp_size1, 4; 2 of 2 configurations can be timed; details: Markdown
Qwen3.6 35B-A3B FP8
Family: Qwen. Checkpoint: Qwen/Qwen3.6-35B-A3B-FP8.
- Single GPU:
Qwen3.6-35B-A3B-FP8/qwen36_local; NVIDIA H200; sweepsworkloadctx8k_out1k; 1 of 1 configuration can be timed; details: Markdown
Qwen3.8 27B NVFP4
Family: Qwen. Checkpoint: nvidia/Qwen3.8-27B-NVFP4.
- vLLM, one GPU:
Qwen3.8-27B-NVFP4/qwen35_vllm_nvfp4_gdn_dense; NVIDIA B200; 1 of 1 configuration can be timed; details: Markdown
Qwen3.8 Flash Next NVFP4
Family: Qwen. Checkpoint: nvidia/Qwen3.8-Flash-Next-NVFP4.
- vLLM, one GPU, MTP speculative decoding:
Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative; NVIDIA B200; sweepsworkloaddiverse_100; 1 of 1 configuration can be timed; details: Markdown
DeepSeek V4 Flash 0731
Family: DeepSeek. Checkpoint: deepseek-ai/DeepSeek-V4-Flash-0731.
- vLLM, DP attention, EP:
DeepSeek-V4-Flash-0731/deepseek_v4_vllm; NVIDIA H200; sweepsworkloadshape_grid_128, uniform; 2 of 2 configurations can be timed; details: Markdown - vLLM, DP attention, EP, parallel streams serialized:
DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams; NVIDIA H200; sweepsworkloadshape_grid_128, uniform; 2 of 2 configurations can be timed; details: Markdown
GLM-5.2
Family: GLM. Checkpoint: zai-org/GLM-5.2.
- vLLM, DP attention, EP:
GLM-5.2/glm52_vllm_dsa_moe; NVIDIA H200; sweepsworkloadctx8k_out1k, uniform; 2 of 2 configurations can be timed; details: Markdown
GLM-5.2 FP8
Family: GLM. Checkpoint: zai-org/GLM-5.2-FP8.
- vLLM, DP attention, EP:
GLM-5.2-FP8/glm52_vllm_dsa_moe; NVIDIA H200; sweepsworkloadctx8k_out1k, uniform; 2 of 2 configurations can be timed; details: Markdown
GLM-5.2 NVFP4
Family: GLM. Checkpoint: nvidia/GLM-5.2-NVFP4.
- SGLang, TP:
GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe; NVIDIA B200; sweepstp_size4, 8;max_model_len8,192, 65,536, 131,072, 262,144, 524,288, 1,048,576;workloaddiverse_100; 12 of 12 configurations can be timed; details: Markdown - vLLM, DP attention, EP MoE:
GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe; NVIDIA B200; sweepsep_size4, 8;max_model_len8,192, 65,536, 131,072, 262,144, 524,288, 1,048,576;workloadc32_long; 12 of 12 configurations can be timed; details: Markdown - vLLM, TP and EP:
GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe; NVIDIA B200; sweepsep_size4, 8;max_model_len8,192, 65,536, 131,072, 262,144, 524,288, 1,048,576;workloadc32_long, diverse_100; 24 of 24 configurations can be timed; details: Markdown - vLLM, TP and EP, MTP speculative decoding:
GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative; NVIDIA B200; sweepsep_size4, 8;max_model_len8,192, 16,384, 65,536, 131,072, 262,144, 524,288, 1,048,576;mtp_modefull_index, index_share;workloaddiverse_100, diverse_100_tp8, quadrant_c48; 27 of 27 configurations can be timed; details: Markdown - vLLM, PP:
GLM-5.2-NVFP4/glm52_vllm_nvfp4_pp_dsa_moe; NVIDIA B200; sweepspp_size4, 8;max_model_len8,192, 65,536, 131,072, 262,144, 524,288, 1,048,576;workloaddiverse_100; 12 of 12 configurations can be timed; details: Markdown
GLM-5.3 NVFP4
Family: GLM. Checkpoint: incoai/GLM-5.3-NVFP4.
- vLLM, TP and EP, DFlash speculative decoding:
GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2; NVIDIA B200; sweepsmax_model_len8,192, 16,384, 131,072, 1,048,576;workloaddiverse_100; 4 of 4 configurations can be timed; details: Markdown
GLM-5.3-Flash
Family: GLM. Checkpoint: zai-org/GLM-5.3-Flash.
- vLLM, DP attention, EP MoE:
GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe; NVIDIA B200; sweepsep_size4, 8;max_model_len8,192, 65,536, 131,072, 262,144, 524,288, 1,048,576;workloadbalanced_c32; 12 of 12 configurations can be timed; details: Markdown - vLLM, TP attention, EP or TP MoE:
GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe; NVIDIA B200; sweepstp_size4, 8;enable_expert_parallelon, off;max_model_len8,192, 65,536, 131,072, 262,144, 524,288, 1,048,576;workloaddiverse_100, balanced_c32; 30 of 30 configurations can be timed; details: Markdown - vLLM, PP:
GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe; NVIDIA B200; sweepspp_size4, 8;max_model_len8,192, 65,536, 131,072, 262,144, 524,288, 1,048,576;workloadbalanced_c32; 12 of 12 configurations can be timed; details: Markdown
GLM-5.3-Flash NVFP4
Family: GLM. Checkpoint: nvidia/GLM-5.3-Flash-NVFP4.
- vLLM, DP attention, EP MoE:
GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe; NVIDIA B200; sweepsep_size4, 8;max_model_len8,192, 65,536, 131,072, 262,144, 524,288, 1,048,576;workloadbalanced_c32; 12 of 12 configurations can be timed; details: Markdown - vLLM, TP attention, EP or TP MoE:
GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe; NVIDIA B200; sweepstp_size4, 8;enable_expert_parallelon, off;max_model_len8,192, 65,536, 131,072, 262,144, 524,288, 1,048,576;workloadbalanced_c32; 24 of 24 configurations can be timed; details: Markdown - vLLM, PP:
GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe; NVIDIA B200; sweepspp_size4, 8;max_model_len8,192, 65,536, 131,072, 262,144, 524,288, 1,048,576;workloadbalanced_c32; 12 of 12 configurations can be timed; details: Markdown
DeepSeek V4.1 Flash
Family: DeepSeek. Checkpoint: deepseek-ai/DeepSeek-V4.1-Flash.
- vLLM, TP attention, EP MoE:
DeepSeek-V4.1-Flash/deepseek_v41_vllm; NVIDIA B200; sweepsmax_model_len131,072, 1,048,576;workloaddiverse_100; 2 of 2 configurations can be timed; details: Markdown - vLLM, TP attention, EP MoE, side streams serialized:
DeepSeek-V4.1-Flash/deepseek_v41_vllm_serial_streams; NVIDIA B200; sweepsmax_model_len131,072, 1,048,576;workloaddiverse_100; 2 of 2 configurations can be timed; details: Markdown
DeepSeek V4.1 Flash 1017
Family: DeepSeek. Checkpoint: deepseek-ai/DeepSeek-V4.1-Flash-1017.
- vLLM, DP attention, EP MoE, DSpark speculative decoding:
DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dp_attn_ep_moe_dspark; NVIDIA B200; sweepsworkloaddiverse_100; 1 of 1 configuration can be timed; details: Markdown - vLLM, TP attention, EP MoE, DSpark speculative decoding:
DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dspark; NVIDIA B200; sweepsworkloaddiverse_100; 1 of 1 configuration can be timed; details: Markdown
Inkling NVFP4
Family: Inkling. Checkpoint: thinkingmachines/Inkling-NVFP4.
- vLLM, TP attention, EP MoE:
Inkling-NVFP4/inkling_vllm_nvfp4_rel_attn_moe; NVIDIA B200; sweepsmax_model_len131,072, 1,048,576;workloaddiverse_100; 2 of 2 configurations can be timed; details: Markdown
Inkling Small NVFP4
Family: Inkling. Checkpoint: thinkingmachines/Inkling-Small-NVFP4.
- vLLM, TP attention, EP MoE:
Inkling-Small-NVFP4/inkling_vllm_nvfp4_rel_attn_moe; NVIDIA B200; sweepsworkloaddiverse_100; 1 of 1 configuration can be timed; details: Markdown
MiMo V2.6 Flash RL
Family: MiMo. Checkpoint: XiaomiMiMo/MiMo-V2.6-Flash-RL.
- vLLM, TP attention, EP MoE:
MiMo-V2.6-Flash-RL/mimo_v2_vllm_fp8_swa_moe; NVIDIA B200; sweepsworkloaddiverse_100; 1 of 1 configuration can be timed; details: Markdown
This page as Markdown: https://servingstudio.cs.washington.edu/models.md. A guide to the whole site for AI agents: llms.txt.