Features
Powerful simulation. Zero manual coding.
Sweep dense and MoE deployments in minutes, check the predictions against real serving frameworks, and trace any result down to the kernel and its lower bound—all in the browser, with the Agent.
Explore the features
ServingStudio’s key features.
Flexible configuration. From dense models to modern MoEs.
The simulator supports configurations from a dense model on one H200 to a routed MoE model across 144 GPUs. The catalog includes BF16, FP8, NVFP4, several parallelism strategies, and split attention–FFN serving.
Models
Llama 3 Qwen3 Qwen3.6 GLM-5.2 DeepSeek V4
Precision
BF16 FP8 NVFP4
Parallelism
TP EP DP PP
Serving
Unified Prefill / decode Attention / FFN Speculative decoding
Frameworks
vLLM SGLang
Hardware
H200 B200
Fast simulation. Simulate long workloads in minutes.
The Rust simulator advances modeled time without waiting for real hardware, so long workloads can reach steady state quickly. This makes broader configuration sweeps practical before deployment.
Llama 3.1 8B and Qwen3-235B. Simulator wall time on one host; it varies with the machine.
14× to 2,770× faster than real time. Each of the 21 runs simulates 33 minutes of serving traffic; the simulator finishes it in 0.7 to 143 seconds of wall time.
Single GPU, dense 1 GPU 2,770×
Tensor parallel 2 to 8 GPUs 1,652 to 1,936×
MoE expert parallel 8 to 32 GPUs 1,091 to 2,516×
Data and tensor parallel 4 to 16 GPUs 676 to 1,681×
Prefill and decode split 9 to 65 GPUs 122 to 780×
Attention and FFN split 24 to 144 GPUs 14 to 50×
Accurate predictions. Calibrated against real serving frameworks.
Kernel timings are profiled on real GPUs, and each higher layer composes those measured costs into a system prediction. The alignment suite then compares predictions with measurements from real serving frameworks across a range of cases.
GLM-5.2 NVFP4, 4 × B200, SGLang TP4.
15 of 15 cases passed all 195 alignment checks; 0 failures
Kernel timing 2.0–5.4% error
Kernel timing: 15 cases; 2.0–5.4% error relative to the measured framework result
Mean end-to-end latency within 7%
Mean end-to-end latency: 15 cases; within 7% relative to the measured framework result
Output throughput within 8%
Output throughput: 15 cases; within 8% relative to the measured framework result
Full observability. Inspect every detail in the run.
The simulator logs the run, each request, every scheduler step, and every kernel for analysis. You can move from end-to-end behavior to the operation that explains it without adding instrumentation overhead to a live deployment.
GLM-5.2 NVFP4, 4 × B200, SGLang TP4. 512 requests, saturated arrival at 256 concurrent, chunked prefill capped at 2,048 batch tokens. This is the same campaign used for the alignment figures above.
Level of detail
Requests in flight over 186.3 seconds, peaking at 256
0 s Requests in flight 186.3 s
Output
10,467 tok/s
Concurrent
204 of 256
GPU busy
97.67%
Optimization insights. Break down the gap to a model-derived lower bound.
Every simulated GPU second is assigned to a named category and compared with a theoretical lower bound derived from the model’s computation and data movement. The breakdown separates required work from modeled gaps in batching, kernel efficiency, power-limited clocks, distance from the hardware peak, fusion, communication, load balance, and idle time.
Simulated workload: Llama 3.1 8B BF16, 1 × NVIDIA H200. 512 requests, each with 1,024 input and 256 output tokens, arriving at 128 requests per second.
99.88% utilized 46.5% required work 53.5% of simulated GPU time sits above the model-required lower bound.
20.64 GPU seconds split into 10 named causes; 46 percent is work the model requires
GPU idle time
0.02 GPU‑s
Load imbalance
<0.01 GPU‑s
Batching gap
2.11 GPU‑s
Communication
0 GPU‑s
Kernel efficiency gap
0.16 GPU‑s
Power gap
1.55 GPU‑s
Hardware peak gap
2.71 GPU‑s
Redundant work gap
0.45 GPU‑s
Fusion gap
4.04 GPU‑s
Model-required lower bound
9.59 GPU‑s
Total GPU time
20.64 GPU‑s
Zero-code exploration. Describe the goal. Review the evidence.
Tell the Agent what you want to find out. It plans and runs the experiment, analyzes the results, and returns the tradeoff with supporting evidence—all on one browser page.
You
Find the request rate where Llama 3.1 8B stops keeping up on one H200.
- Agent Plans the experiment and decides what to compare
- Simulator Runs the selected configurations
- Analyzer Attributes GPU time and identifies the bottleneck
Back to you
Charts, tables, and a clear explanation of the tradeoff.
Start exploring your serving setup with ServingStudio.
ServingStudio Simulate, understand, and optimize LLM serving. Page source
This page as Markdown: https://servingstudio.cs.washington.edu/features.md. A guide to the whole site for AI agents: llms.txt.