---
name: servingstudio-simulate
description: >-
  Simulate an LLM serving deployment (model, GPU, parallelism, worker, replicas)
  on a request workload through the ServingStudio public API: a recorded
  capture, a generated trace or an uploaded CSV, with TTFT, TPOT, end-to-end
  latency and throughput. Not for the time of one batch (that is a timing
  prediction).
---

# ServingStudio Simulation

A ServingStudio simulation replays a stream of requests through one deployment
and reports what a client would see: time to first token (TTFT), time per
output token (TPOT), end-to-end latency, and throughput. A deployment is one
member of a sim preset: a checkpoint, a deployment shape (`unified` or `pd`),
its pools of GPUs, the worker each pool runs (the scheduler: batching, chunked
prefill, speculative decoding) and one value of each parameter the preset
sweeps, such as `replicas` or `tp_size`. Every kernel time comes from
measurements. Use this skill to answer questions such as "what TTFT and TPOT
does GLM-5.2 NVFP4 on 4 B200s give at 2 requests/s with 4k prompts?"

The API is JSON over HTTPS. Set the base URL once:

```bash
API=${SERVINGSTUDIO_API:-https://servingstudio.cs.washington.edu/api/public/v1}
curl -s "$API/health"
```

If `/health` does not return `{"status": "ok", ...}`, ask the user for the base
URL; do not guess another host. `$API/openapi.json` lists every route.

A simulation runs in the background: `POST /simulate` queues it and answers at
once, and you read it with `GET /simulations/{id}` until it is done. Two run at
a time, each for at most 10 minutes of wall clock and 2000 requests; results
are kept 24 hours. `POST /simulate` and `POST /workloads` take 6 requests per
minute per client; a refused request does not count.

## 1. Find a deployment

`/simulations/presets` lists every sim preset with its `axes` and `members`:

```bash
curl -s "$API/simulations/presets" > presets.json
jq -r '.presets[] | .id as $id | ([.pools[].gpu] | unique | join("+")) as $g | .members[]
  | [$id, $g, (.params | tojson), .gpus, (.unavailable | keys | join(","))] | @tsv' presets.json
```

A preset gives its `deployment`, its `pools` (per role, the arch preset, arch,
GPU and worker type) and `captures`, the recorded workloads its members can
replay. A capture's `routing` says how it routes experts (`popularity`,
`corpus`, or null for a dense model, which can replay any published trace,
including one recorded on another model). Each member gives:

- `params`: its axis values. A simulation names the member by these, exactly.
- `gpus`: the GPUs it occupies, every pool and replica included.
- `pools`: per role, `replicas`, the arch preset's `arch_params`,
  `max_model_len` (the longest request the pool serves; `arch_params` may leave
  it to the model's own limit), `gpus_per_replica`, the `worker` block, and
  `kv_cache`: the pool's KV cache as its build sizes it, the first
  partition's entry of a run's `raw/kv_cache.json` plus `partitions` (per
  replica). `result.max_concurrency_at_max_model_len` is vLLM's "Maximum
  concurrency" for one partition, `result.request_parts` what one request of
  `max_model_len` holds per part, and `layout` the arch's declared parts;
  bytes are over `attention_shards` GPUs.
- `unavailable`: per capture, why the member cannot run it: `{"error": ...}`
  when it does not build, `{"missing": {<kernel role>: <rows>}}` when
  profile.db lacks rows, or `{"misfit": {"reason", "max_model_len", ...}}`
  when the capture's requests do not fit its pools' `max_model_len` (that
  blocks a replay only; the capture still serves as routing). A member can run
  a capture missing from this map.
- `error`: why the member does not build, or null.

An axis can carry `rows`, what each of its values sets: `server: ep4_ctx128k`
is `ep_size` 4 with `max_model_len` 131072. The members that can replay one
capture:

```bash
jq '.presets[] | select(.id == "GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_chunked_prefill")
  | {deployment, pools, axes, captures}' presets.json
jq -c '.presets[] | select(.id == "GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_chunked_prefill")
  | .members[] | select(.error == null and (.unavailable | has("diverse_100") | not)) | .params' presets.json
```

A worker of type `speculative` drafts `draft_tokens` per request per step; its
member needs an `accept_rate` (step 2).

## 2. Choose the workload

`workload.source` says where the requests come from. `GET /workloads`
describes the three sources:

```bash
curl -s "$API/workloads" | jq '.sources | map_values(.summary)'
```

**Routing.** An MoE model's experts are routed as one of the member's captures
routes them, whatever the source: `workload.capture` names the capture, the
preset's first by default. The service never falls back to a synthetic
routing, and the answer states the routing it used (step 4). A dense model
routes nothing.

**A capture** (`"source": "capture"`, the default). The requests are the
capture's recorded trace, and its routing is the capture's own. Each capture's
`facts` say what it holds: `requests`, mean `prompt_tokens` and
`output_tokens`, `rate`, its recorded requests per second, and
`input_file_tags`. A capture tagged `speculative` carries the acceptance it
recorded, per request, so a speculative member needs no `accept_rate` on it.
A capture whose `rate` is null had every request arrive at once; it takes a
concurrency, not a rate:

```json
{"source": "capture", "capture": "diverse_100", "load": {"rate": 4}}
{"source": "capture", "capture": "c32_long", "load": {"concurrency": 32}}
```

**A generated trace** (`"source": "generated"`). req-frontend's `tracegen`
draws it from `generator`: `type` and the generator's arguments. List them:

```bash
curl -s "$API/workloads" | jq -r '.sources.generated.generators[] | .name as $g
  | .arguments[] | [$g, .name, .default, (.choices | join("|")), .help] | @tsv'
```

Each length argument (`input_len`, `output_len`, `rounds`, `tool_wait_ms`)
takes a distribution as a string: `"512"` for a constant, `"uniform:256..1024"`
for an even range, `"lognormal:2048,0.8"` for a long right tail (median 2048).
`sessions` is how many sessions to draw and `rounds` how many requests each
session sends; a later round carries the conversation so far as its prefix.
`arrival_rate` is sessions per second and `seed` fixes the draw. For
independent requests, give `"rounds": "1"`:

```json
{"source": "generated",
 "generator": {"type": "synthetic", "sessions": 200, "rounds": "1",
               "input_len": "lognormal:4096,0.6", "output_len": "lognormal:512,0.5",
               "arrival_rate": 2, "seed": 1}}
```

`arrival_pattern` is `poisson` (the default) or `constant`. Poisson
arrivals drawn at `arrival_rate` 2 have a mean near 2, not exactly 2 (the
upload's or trace's `rate` says what was drawn). To run at exactly r per
second, keep the draw and add `"load": {"rate": r}` (below), which rescales
the arrivals to that mean.

A session trace's rounds always chain: each round waits for the previous round
to finish and for its tool wait.

**An uploaded CSV** (`"source": "upload"`). Send the file to `POST /workloads`
as the raw request body (`--data-binary`, not a multipart `-F` form), with
`format` and comma-separated `tags` as query parameters when you give them,
then name the answer's `workload_id`:

```bash
curl -s "$API/workloads" | jq '.sources.upload | {formats, tags, max_bytes}'
cat > trace.csv <<'EOF'
id,arrival_time,input_len,output_len
r0,0,2048,256
r1,250,1024,128
r2,500,4096,512
EOF
UP=$(curl -s -X POST "$API/workloads" \
  -H 'content-type: text/csv' --data-binary @trace.csv | jq -r .workload_id)
# or say the format: "$API/workloads?format=text-generation-independent"
```

A format's `columns` are the header the file must have, exactly, in any order;
each tag adds its own columns. With no `format` (and no `tags`), the service
reads the file as the format and tags its header fits; give `format` (and
`tags`) to say it yourself. In `text-generation-independent`,
`arrival_time` is in milliseconds and must not decrease down the file. The
service reads the file as a run will and answers 400 with the reason when it
cannot; the answer gives `workload_id`, the `input_file_format` and
`input_file_tags` it read, the same facts a capture has (`requests`,
`sessions`, true when rounds chain, mean `prompt_tokens` and `output_tokens`,
`rate`) and `expires_at` (24 hours). An upload cannot be deleted; it
expires. An upload
tagged `speculative` carries each request's acceptance in its `accept_rate`
column (one probability, or a JSON list of `draft_tokens` of them); a
speculative member then takes no `accept_rate` in the simulation.

**Knobs for every source.** Every request of the source runs.

| Field | Default | Meaning |
| --- | --- | --- |
| `load` | the trace's own arrival times | One of `{"rate": r}`: r requests (sessions, when rounds chain) per second, the trace's spacing scaled to that mean; or `{"concurrency": c}`: every request ready at once, at most c in flight. A trace whose requests all arrive at once (`rate` null) takes only a concurrency. |
| `run_to_end` | true | Run until every request finishes; with false, `duration_ms` of simulated time. |
| `duration_ms` | none | Simulated milliseconds to run; required when `run_to_end` is false. |
| `accept_rate` | none | Speculative workers only, and then required unless the upload carries its own column: the chance each draft token is accepted, one number for every position or a list of `draft_tokens` numbers, one per position. |

`/simulations/presets` also gives the full JSON schema of the workload under
`workload`.

## 3. Start the simulation

Post the preset id, the member's `params`, and the workload. Each `params`
value must equal the member's value; a number given as a string also matches.

```bash
SIM=$(curl -s -X POST "$API/simulate" -H 'content-type: application/json' -d '{
  "preset": "GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_chunked_prefill",
  "params": {"replicas": 1, "server": "ep4_ctx128k"},
  "workload": {"source": "generated",
               "generator": {"type": "synthetic", "sessions": 100, "rounds": "1",
                             "input_len": "lognormal:4096,0.6", "output_len": "256",
                             "arrival_rate": 2, "seed": 1}}
}')
echo "$SIM" | jq .
ID=$(echo "$SIM" | jq -r .simulation_id)
```

The answer is 202 with `simulation_id`, `status` (`queued`) and `routing`:
`capture` (null for a dense model) and `label`, such as "requests from your
workload, routing from capture c32_long".

A speculative member, with one acceptance per draft position:

```json
{"preset": "GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative",
 "params": {"replicas": 1, "server": "ctx256k"},
 "workload": {"source": "upload", "upload": "<workload_id>",
              "accept_rate": [0.9, 0.8, 0.7, 0.6, 0.5]}}
```

The request is checked before it queues:

| Status | Cause | `detail` |
| --- | --- | --- |
| 400 | `params` miss an axis, name an unknown one, or match no member | `{message, choices}`; `choices` lists every member's `params` |
| 400 | A bad workload: an unknown capture or upload, a generator argument tracegen rejects, a trace the simulator cannot read, more than 2000 requests, a request longer than a pool's `max_model_len` (prefix + input + output; a capture's own requests included, which `/simulations/presets` lists as its `misfit`), a missing or wrong `accept_rate` (both the request's and an upload's column, or a list that is not `draft_tokens` long) | the reason, naming the request or argument; for a request too long, `{message, too_long: {pool, max_model_len, requests, total}}`, `pool` naming the pool whose limit it broke |
| 404 | Unknown preset id | `no public preset '...'` |
| 409 | The member does not build or lacks profile.db rows for that capture | the reason; pick another member or capture |
| 413 | An upload over `max_bytes` | the limit |
| 429 | The client's rate limit (`Retry-After` header) or 32 simulations already waiting | the limit |

## 4. Read the result

Poll every few seconds; most runs take seconds to a minute, and a run stops
itself at 10 minutes, so give up after about 12:

```bash
for i in $(seq 144); do S=$(curl -s "$API/simulations/$ID")
  echo "$S" | jq -e '.status | test("done|failed|timed_out")' >/dev/null && break
  echo "$S" | jq -c '{status, queue_position}'; sleep 5; done
echo "$S" | jq '{status, error, routing, workload, gpus, summary}'
```

`status` is `queued` (with `queue_position`), `running`, `done`, `failed`
(with `error`) or `timed_out`. The record repeats the request: `preset`,
`params`, `workload` (with the `capture` used, `trace`, its facts as
recorded, before any `load` rescales its rate), `routing` and `gpus`. Once
done, `summary` has:

- `cause`: why the run stopped;
- `requests`: `total` and `finished`;
- `sim_ms`: simulated time, the drain after the last arrival included;
- `num_gpus`: the GPUs the run counted;
- `throughput`: `total_tok_s`, `prefill_tok_s`, `decode_tok_s`,
  `total_tok_s_per_gpu`, and `completed_req_s`, finished requests over
  `sim_ms`; each is over all of `sim_ms`, not only the busy part;
- `ttft_ms`, `tpot_ms`, `e2e_ms`: `mean`, `p50`, `p90`, `p99`, `max` and `n`, in
  milliseconds. `n` counts the requests the metric covers: a request with one
  output token has no TPOT, so `tpot_ms.n` can be below `finished`.

```bash
echo "$S" | jq '.summary | {cause, requests, sim_ms, throughput, ttft: .ttft_ms, tpot: .tpot_ms, e2e: .e2e_ms}'
```

For more, the Analyzer serves the run under `$API/analyzer/runs/{run_id}/`,
with the record's `run_id`:

```bash
R="$API/analyzer/runs/$(echo "$S" | jq -r .run_id)"
curl -s "$R/descriptor" | jq .
curl -s "$R/subjects/slo-general/report" | jq .
```

Delete a simulation you no longer need, or stop one still running, with
`curl -s -X DELETE "$API/simulations/$ID"`.

## 5. Interpret and report

The simulation is an event-driven model of the serving engine's scheduler
over measured kernel times: queueing, batching, chunked prefill, KV-cache
admission and, for speculative workers, drafting with acceptance drawn from
`accept_rate`. It models no tokenizer, network or client. A generated trace is
not a model of real traffic; it tells how the deployment responds to the shape
you asked for, so say which shape you simulated.

When you answer, name the preset, the member's `params`, the GPUs, the
workload (its source, the generator arguments or upload, its request count,
the `load`, `accept_rate`), the
`routing.label`, and the numbers from `summary` you quote, with `requests`
finished out of total. Quote only numbers the API returns; label any
arithmetic on them as your own.
