[ServingStudio](https://servingstudio.cs.washington.edu/)

# Bridge Simulation and Agents for Speed-of-Light Serving

Compare configurations using performance models grounded in real GPU measurements, then let an agent build and validate the strongest candidates in real serving frameworks.

See the Agent at work [Explore ServingStudio features](https://servingstudio.cs.washington.edu/features.html)

- [GitHub](https://github.com/SyFI-ServingStudio/ServingStudio)
- [Blog](https://servingstudio.cs.washington.edu/blog.html)

## Run the study with the Agent.

Begin with one serving question. The first answer comes from real B200 measurements; the Agent then uses the Simulator and Analyzer to explain the kernel costs, compare them with the hardware limit, and choose an operating point.

### You

For GLM-5.2 on four B200 GPUs with TP4 + EP4, which speculative depth gives the best single-request decode throughput on enwik9 input?

### ServingStudio Agent

I have a completed depth sweep from real vLLM serving on four B200 GPUs with TP4 + EP4 (DP1). It uses enwik9, derived from an English Wikipedia dump, and covers nine settings, from speculation off to eight draft tokens.

Experiment plan

- Replay the same eight 16K-input, 2K-output requests at each depth
- Measure accepted draft tokens at each depth
- Compare median decode throughput after one warm-up request

Running depth sweep… Depth sweep complete

Nine depth settings measured on the same model, hardware, and request set.

Depth 3 is the best tested point at 218.5 tok/s, or 2.14× the throughput without speculation. It accepts 1.83 draft tokens per round on average.

GLM-5.2 NVFP4 4 × B200 · TP4 + EP4 (DP1)

### Throughput versus speculative depth

Single-request decode throughput · output tok/s Higher is better

Single-request decode throughput by speculative depth

Draft tokens per round

Decode throughput

218.5 tok/s

Time per output token

4.58 ms

Accepted draft tokens

1.83 / round

GLM-5.2 NVFP4 · 4 × B200 · TP4 + EP4 (DP1) · enwik9 · one request at a time.

View the depth sweep and method

| Draft tokens | Time per output token (ms) | Accepted draft tokens / round | Throughput (tok/s) |
| --- | --- | --- | --- |
| Off | 9.787 | 0 | 102.175 |
| 1 | 5.633 | 0.925 | 177.515 |
| 2 | 4.757 | 1.460 | 210.235 |
| 3 | 4.577 | 1.834 | 218.473 |
| 4 | 4.891 | 2.017 | 204.468 |
| 5 | 5.098 | 1.907 | 196.151 |
| 6 | 5.553 | 2.143 | 180.080 |
| 7 | 5.670 | 2.063 | 176.353 |
| 8 | 5.657 | 2.335 | 176.788 |

Each request has 16,384 input and 2,048 output tokens. The first request warms up the engine; the median of the next seven determines throughput. Prefill and the first output token are excluded. Acceptance is measured independently at each depth across all eight requests, including the warm-up.

[Download experiment data](https://servingstudio.cs.washington.edu/data/glm52-spec-depth-evidence.json)

Cumulative reach rate **90.6% · 62.4% · 30.4%** Accepted at positions 1, 2, and 3 ÷ all draft rounds

Conditional acceptance **90.6% · 68.9% · 48.6%** Accepted at each position ÷ rounds that reached that position

In the later concurrency sweep, I reuse the conditional acceptance rates measured here. More accepted drafts do not guarantee higher throughput: depth 4 accepts 2.02 draft tokens per round, compared with 1.83 at depth 3, but is slower at 204.5 versus 218.5 tok/s.

### You

With three draft tokens and a 128K-token input, where does one decode iteration spend its time?

### ServingStudio Agent

Now I’ll switch from measurement to simulation. The Simulator predicts one target-model verification pass and three draft passes at 128K context; I rank the kernel call sites by time.

Running time prediction… Time prediction complete

One speculative iteration: three drafted positions followed by the final verification step.

GLM-5.2 NVFP4 4 × B200 · TP4 + EP4 (DP1)

### Kernel-time breakdown at 128K context

**14.03** ms per speculative iteration

fused\_moe **2.55 ms**

18.2%

fused\_moe **0.85 ms**

6.1%

fused\_qkv\_a\_proj **0.75 ms**

5.3%

o\_proj **0.65 ms**

4.6%

tp\_allreduce **0.54 ms**

3.8%

decode **0.53 ms**

3.8%

decode\_topk **0.46 ms**

3.3%

gate\_up\_proj **0.41 ms**

2.9%

q\_b\_proj **0.39 ms**

2.8%

decode\_logits **0.36 ms**

2.5%

The final verification step sees 131,075 KV positions: 131,072 input tokens plus three drafted positions.

The 10 largest kernel call sites account for 53.3% of the modeled iteration. The two routed-expert MoE call sites are the largest contributors, together accounting for 24.2%.

### You

How close are these kernels to the B200 hardware limit?

### ServingStudio Agent

I’ll hold the iteration shape fixed and compare the modeled cost of each kernel with its B200 hardware limit.

Running optimality analysis… Optimality analysis complete

The Analyzer split the 128K iteration’s GPU time into the hardware limit and the five sources of overhead above it.

GLM-5.2 NVFP4 4 × B200 · TP4 + EP4 (DP1)

### Distance from the hardware limit at this batch shape

**25.4%** hardware limit

GPU time divided among the hardware limit, hardware peak gap, power gap, communication, kernel efficiency gap, and rank imbalance

Hardware limit

25.4%

Hardware peak gap

3.3%

Power gap

0.5%

Communication

9.2%

Kernel efficiency gap

56.8%

Rank imbalance

4.7%

#### Hardware-limit share by kernel call site

Hardware limit as a share of load-balanced time

`fused_moe`

unified.body.sparse\_cycle\_index\_share.moe.routed\_experts.fused\_moe: 50.9% hardware limit, 6.7% hardware peak gap, 1.1% power gap, 0.0% communication, 41.3% kernel efficiency gap

**50.9%**

`fused_qkv_a_proj`

unified.body.sparse\_cycle\_index\_share.attention.fused\_qkv\_a\_proj: 29.1% hardware limit, 3.8% hardware peak gap, 0.6% power gap, 0.0% communication, 66.5% kernel efficiency gap

**29.1%**

`fused_moe`

unified.body.sparse\_cycle\_full\_index.moe.routed\_experts.fused\_moe: 50.9% hardware limit, 6.7% hardware peak gap, 1.1% power gap, 0.0% communication, 41.3% kernel efficiency gap

**50.9%**

`o_proj`

unified.body.sparse\_cycle\_index\_share.attention.o\_proj: 52.8% hardware limit, 6.9% hardware peak gap, 1.1% power gap, 0.0% communication, 39.2% kernel efficiency gap

**52.8%**

`tp_allreduce`

unified.body.sparse\_cycle\_index\_share.moe.tp\_allreduce: 0.0% hardware limit, 0.0% hardware peak gap, 0.0% power gap, 100.0% communication, 0.0% kernel efficiency gap

**0.0%**

`decode`

unified.body.sparse\_cycle\_index\_share.attention.sparse\_mla.decode: 6.2% hardware limit, 0.8% hardware peak gap, 0.1% power gap, 0.0% communication, 92.8% kernel efficiency gap

**6.2%**

`decode_topk`

unified.body.sparse\_cycle\_full\_index.attention.indexer.decode\_topk: 1.0% hardware limit, 0.1% hardware peak gap, 0.0% power gap, 0.0% communication, 98.8% kernel efficiency gap

**1.0%**

`gate_up_proj`

unified.body.sparse\_cycle\_index\_share.moe.shared\_expert.gate\_up\_proj: 20.8% hardware limit, 2.7% hardware peak gap, 0.4% power gap, 0.0% communication, 76.1% kernel efficiency gap

**20.8%**

`q_b_proj`

unified.body.sparse\_cycle\_index\_share.attention.q\_b\_proj: 28.9% hardware limit, 3.8% hardware peak gap, 0.6% power gap, 0.0% communication, 66.7% kernel efficiency gap

**28.9%**

`decode_logits`

unified.body.sparse\_cycle\_full\_index.attention.indexer.decode\_logits: 45.2% hardware limit, 5.9% hardware peak gap, 0.9% power gap, 0.0% communication, 48.0% kernel efficiency gap

**45.2%**

The value at right divides each call site’s hardware limit by its load-balanced time—its time with rank imbalance removed. These 10 call sites cover 51.5% of the iteration’s load-balanced kernel time.

At this shape, the model-derived hardware limit accounts for 25.4% of GPU time. The largest opportunity is kernel efficiency: the gap between the modeled kernels and a GEMM or memory copy running back to back on a B200 accounts for 56.8%. The power limit, which keeps that sustained rate below the clock a short measured kernel reaches, adds 0.5%, and the gap from the fastest measured kernels to the spec-sheet peak is only 3.3%. Communication accounts for 9.2% and uneven work across GPU ranks for 4.7%.

### You

With three draft tokens, 32K input tokens, and 16K output tokens, which concurrency maximizes total throughput while keeping each request above 80 tok/s?

### ServingStudio Agent

I’ll sweep concurrency with the request shape and acceptance profile fixed, discard points at or below 80 tok/s per request, and choose the remaining point with the highest total output throughput.

Running concurrency sweep… Concurrency sweep complete

The Simulator ran 8 operating points, each with 96 fixed-length requests.

GLM-5.2 NVFP4 4 × B200 · TP4 + EP4 (DP1)

### Throughput with an 80 tok/s per-request target

Total output throughput · tok/s Per-request target: \>80 tok/s

Output throughput and per-request decode speed by concurrency

Concurrent requests

Total output throughput

1,919.3 tok/s

Per-request decode speed

82.7 tok/s

Mean time per output token

12.09 ms/token

View every tested point

| Concurrency | Total output throughput (tok/s) | Per-request decode speed (tok/s) | Mean TPOT (ms/token) |
| --- | --- | --- | --- |
| 1 | 201.94 | 205.20 | 4.873 |
| 2 | 346.63 | 175.84 | 5.687 |
| 4 | 585.58 | 148.46 | 6.736 |
| 8 | 921.35 | 116.88 | 8.556 |
| 12 | 1,246.83 | 105.91 | 9.442 |
| 16 | 1,487.63 | 95.01 | 10.525 |
| 24 | 1,919.27 | 82.71 | 12.091 |
| 32 | 2,242.64 | 73.31 | 13.641 |

At each operating point, the Simulator ran 96 identical requests, each with 32,768 input and 16,384 output tokens, keeping the concurrency limit saturated. The run used three draft tokens and the measured conditional acceptance rates: 90.62%, 68.89%, and 48.63%. Per-request decode speed is 1,000 divided by mean TPOT; total output throughput is the rate of completed decode tokens. All 96 requests finished at every point.

[Download experiment data](https://servingstudio.cs.washington.edu/data/glm52-spec3-study.json)

Among the tested points, concurrency 24 is the highest-throughput choice that stays above the 80 tok/s per-request target: 1,919.3 tok/s total at 82.7 tok/s per request. Concurrency 32 reaches 2,242.6 tok/s, but per-request speed drops to 73.3 tok/s—below the target.

## The Simulator predicts. The Agent acts.

Grounded in measured GPU kernel timings, the Simulator predicts performance across models, hardware, and serving configurations. The Agent runs the optimization loop: it designs the experiment, sweeps configurations, explains the result, implements the change in vLLM or SGLang, and validates it on real hardware.

### Simulator

Predicts serving performance.

Models

Dense and mixture-of-experts architectures in BF16, FP8, and NVFP4

Deployments

H200 and B200 GPUs with TP, EP, DP, and PP

Execution

Unified serving, prefill–decode or attention–FFN disaggregation, and speculative decoding

Results

Throughput, TTFT, TPOT, and per-kernel timing

Calibration

Checked against measurements from vLLM and SGLang runs

### Agent

Runs the optimization loop.

Designs

Turns a serving question into an executable experiment

Searches

Sweeps configurations and compares their tradeoffs

Explains

Traces performance back to specific kernels and system behavior

Builds

Implements the selected change in vLLM or SGLang

Validates

Benchmarks the change on real hardware and explains any remaining gap

## Close the loop between simulation and serving.

Each pass starts with a measured baseline, tests one simulation-guided change in a real serving system, and returns the result to the Simulator. The Agent uses that evidence to keep, reject, or refine the change and decide what to test next.

### Set the target and baseline

Define the model, request mix, hardware, and performance target, then measure the current deployment to establish a baseline.

### Explore in simulation

Sweep serving configurations and inspect predicted kernel costs to select one change worth testing.

### Implement one change

Apply the selected change to a real serving system, whether it is a kernel optimization or support for a new model. Pass correctness checks before measuring performance.

### Profile the change

Capture a GPU trace of the modified implementation. Break down execution time across kernels, communication, and idle gaps.

### Compare measurement with prediction

Line up the measured breakdown against the Simulator's prediction, then trace each gap to the performance model, implementation, or measurement setup.

### Validate end-to-end performance

Measure throughput and latency on the full serving workload and compare with the baseline. Keep, reject, or refine the change, then use the evidence to guide the next simulation pass.

## From simulation to faster serving systems.

By comparing simulated and measured performance, we found opportunities to improve frontier serving systems (SGLang, vLLM), as well as both integrate and optimize new MoEs into Mini-SGLang. These case studies show what we changed and how much faster each system ran.

vLLM · GLM-5.2 MTP

### Restore CUDA graph replay for speculative decoding.

With five MTP draft tokens, vLLM ran a common 2,048-token batch eagerly because its graph-size filter excluded that batch. Raising the scheduler and graph limits to 2,052 tokens restored CUDA graph replay. The updated configuration delivered 10.8% higher output throughput in a follow-up benchmark.

**+10.8%** higher output throughput than the 2,048-token baseline

NVFP4 · 4 × B200 · TP4 + EP4

Mini-SGLang · Qwen3-235B

### Build and optimize Qwen3-235B in Mini-SGLang.

Mini-SGLang did not support MoE models, so we built a Qwen3-235B serving path and used ServingStudio Sim to guide kernel selection, communication design, and operator fusion. On the tested prefill-heavy workload, the final implementation delivered 25.6% higher output throughput than vLLM.

**+25.6%** higher output throughput than vLLM

FP8 · 4 × H200 · TP4 + EP4

SGLang · GLM-5.2

### Autotune MoE kernels for prefill graphs.

ServingStudio Sim exposed slower-than-expected MoE kernels during SGLang prefill. Autotuning was enabled, but it did not cover the CUDA graph path used by prefill. Tuning that path increased input throughput by 5.6%.

**+5.6%** higher input throughput than the unpatched baseline

NVFP4 · 4 × B200 · TP4

## ServingStudio’s key features.

From a serving question to the evidence behind the answer.

[Explore all features](https://servingstudio.cs.washington.edu/features.html)

### Flexible configuration

Explore dense and MoE models across precisions, GPU layouts, and serving strategies.

[https://servingstudio.cs.washington.edu/features.html#support](https://servingstudio.cs.washington.edu/features.html#support)

### Fast simulation

Reach steady state and compare more configurations with a simulator built for speed.

[https://servingstudio.cs.washington.edu/features.html#speed](https://servingstudio.cs.washington.edu/features.html#speed)

### Accurate predictions

Predictions draw on real GPU measurements and are calibrated against vLLM and SGLang.

[https://servingstudio.cs.washington.edu/features.html#accuracy](https://servingstudio.cs.washington.edu/features.html#accuracy)

### Full observability

Follow performance from the whole run down to individual requests, steps, and kernels.

[https://servingstudio.cs.washington.edu/features.html#observability](https://servingstudio.cs.washington.edu/features.html#observability)

### Optimization insights

Compare simulated GPU time with model-required work to identify where time could be saved.

[https://servingstudio.cs.washington.edu/features.html#optimization](https://servingstudio.cs.washington.edu/features.html#optimization)

### Zero-code exploration

Give the Agent a goal and receive experiments, analysis, tradeoffs, and supporting evidence.

[https://servingstudio.cs.washington.edu/features.html#agent](https://servingstudio.cs.washington.edu/features.html#agent)

## Start exploring your serving setup with ServingStudio.

[Get started on GitHub](https://github.com/SyFI-ServingStudio/ServingStudio)

[ServingStudio](https://servingstudio.cs.washington.edu/) Simulate, understand, and optimize LLM serving. [Page source](https://github.com/SyFI-ServingStudio/ServingStudioIntro)

---

This page as Markdown: https://servingstudio.cs.washington.edu/index.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
