Architecture
How kernel measurements become serving predictions.
A kernel is measured once on the target GPU. Higher layers compose that measurement into a model, then into the workers and pools that serve requests. The top layer predicts a whole serving run.
Explore the architecture
Every prediction starts with a measurement.
Kernel timings come from measurements on real hardware. During simulation, cached costs are evaluated for the current shapes and composed into a serving prediction. That prediction can be traced back to the kernels that support it.
A real kernel, on a real GPU
Run a real kernel with concrete inputs, time it, and record the result on the target GPU.
One row in the database
The row stores the measured timing for reuse across experiments.
An interpolating cache
The cache interpolates measured samples for fast cost queries.
The cost at this shape
The simulator evaluates the cache for each input shape as the run progresses.
Assemble kernels into a model.
A cost tree captures how kernels run in sequence, across parallel ranks, and through repeated layers. Build it once, then evaluate its leaves for each new batch.
Cost tree model
Llama 3.1 8B · recorded cost treeScroll within the tree to inspect every operation.
Sum Llama 3.1 8B · one iteration
Leaf Embedding
0.005 ms Triton
Scale 32 decoder layers
Sum Decoder layer
Sum Pre-attention
Leaf Input RMSNorm
0.006 ms FlashInfer
Leaf QKV projection
0.074 ms PyTorch
Leaf Attention
0.040 ms vLLM CUDA · FA3
Sum Post-attention
Leaf Output projection
0.046 ms PyTorch
Leaf RMSNorm
0.006 ms FlashInfer
Leaf Gate/up projection
0.347 ms PyTorch
Leaf SiLU activation
0.023 ms Triton
Leaf Down projection
0.167 ms PyTorch
Leaf Final RMSNorm
0.006 ms FlashInfer
Leaf LM head
0.238 ms PyTorch
Cost tree phase
The model’s operations are assembled into a cost tree once. Sum combines sequential work, Max combines parallel ranks, and Scale represents repeated layers.
Predicted iteration time
22.942 ms
1,024-token prefill BF16 · 1 × H200
Recorded simulation · iteration 1. One prefill request, no cached prefix, 32 decoder layers.
This is the recorded model tree, from embedding through the decoder layers to the LM head. Operations shown as single leaves include their underlying kernel costs. Timings are per invocation; the original tree determines the iteration total.
Organize workers into a serving system.
Workers manage local requests. Pools organize workers. The orchestrator coordinates work across pools.
Serving hierarchy
A worker advances its assigned requests.
A worker tracks local request and KV state, forms batches, and queries the model’s cost tree to determine when work finishes. One worker can span one GPU or a parallel group.
Worker A
Shell
Sets the execution cadence by forming a batch, building its input, and completing the iteration.
Admission
Chooses which pending request starts next and manages its lifecycle.
KV
Owns cache capacity and each request’s residency, from reservation to release.
Execution
Turns admitted work and KV state into the model input for this iteration.
Cost Tree
Returns the predicted cost of the model input, which sets the next compute completion time.
Sweep parameters systematically.
Define a serving setup in YAML, then vary request rates, GPU allocation, and kernel backends. The launcher expands the combinations and applies your constraints.
Launcher demonstrations
Launcher YAML Illustrative excerpt
One parameter, three arrival rates.
Seven layers. Explicit responsibilities.
Timing flows from layer 1, the kernel, to layer 7, the run. State starts at layer 5, the worker. Select a layer to inspect its contract.
Measured foundation
Cost composition
Stateful execution
L1 Kernel / Profiling
Measured foundation
Measure kernel costs on hardware.
Python profiles real kernels on real GPUs. Rust builds an interpolating timing cache from those samples and evaluates it for each kernel shape.
Examples
Single GEMM
Use matrix dimensions, precision, and GPU configuration to query a cached kernel cost.
RMSNorm
Query normalization cost by token count for a configured hidden size and precision.
Start exploring your serving setup with ServingStudio.
ServingStudio Simulate, understand, and optimize LLM serving. Page source
This page as Markdown: https://servingstudio.cs.washington.edu/architecture.md. A guide to the whole site for AI agents: llms.txt.