Simulate | ServingStudio

Send traffic to a deployment and read the latency and throughput its clients would see.

Interactive page: https://servingstudio.cs.washington.edu/simulate.html. The page reads its lists from ServingStudio Sim's public API at https://servingstudio.cs.washington.edu/api/public/v1 (OpenAPI: https://servingstudio.cs.washington.edu/api/public/v1/openapi.json). An agent can use it through the servingstudio-simulate skill.

This text was generated when the site was built, from Sim commit db977adc881595c5b48aa070bddc7cd6f33db130; the API has the current lists.

A simulation replays a stream of requests through one deployment and reports time to first token (TTFT), time per output token (TPOT), end-to-end latency and throughput. 32 deployments can be simulated. Start one with POST /api/public/v1/simulate and read it from /api/public/v1/simulations/{simulation_id}. At most 2 simulations run at once and 32 wait; a run may use up to 2,000 requests and is stopped after 10 minutes.

Deployments

Llama 3.1 8B

Qwen3 235B-A22B

Qwen3 235B-A22B FP8

Qwen3 235B-A22B Thinking 2507 FP8

Qwen3.6 35B-A3B FP8

Qwen3.8 27B NVFP4

Qwen3.8 Flash Next NVFP4

DeepSeek V4 Flash 0731

GLM-5.2 FP8

GLM-5.2 NVFP4

GLM-5.3 NVFP4

GLM-5.3-Flash

GLM-5.3-Flash NVFP4

DeepSeek V4.1 Flash

DeepSeek V4.1 Flash 1017

Inkling NVFP4

Inkling Small NVFP4

MiMo V2.6 Flash RL

Workloads

A simulation's requests come from one of three sources:

Generator synthetic

Draw a session workload from distributions, with no corpus at all. Arguments:

Upload formats

An uploaded CSV may be up to 1024 KiB and is kept for 24 hours.

An MoE member routes every workload's tokens as one of its captures does: workload.capture names it, its first by default.


This page as Markdown: https://servingstudio.cs.washington.edu/simulate.md. A guide to the whole site for AI agents: llms.txt.