ServingStudio

Architecture

How kernel measurements become serving predictions.

A kernel is measured once on the target GPU. Higher layers compose that measurement into a model, then into the workers and pools that serve requests. The top layer predicts a whole serving run.

Explore the architecture

Every prediction starts with a measurement.

Kernel timings come from measurements on real hardware. During simulation, cached costs are evaluated for the current shapes and composed into a serving prediction. That prediction can be traced back to the kernels that support it.

A real kernel, on a real GPU

Run a real kernel with concrete inputs, time it, and record the result on the target GPU.

One row in the database

The row stores the measured timing for reuse across experiments.

An interpolating cache

The cache interpolates measured samples for fast cost queries.

The cost at this shape

The simulator evaluates the cache for each input shape as the run progresses.

Assemble kernels into a model.

A cost tree captures how kernels run in sequence, across parallel ranks, and through repeated layers. Build it once, then evaluate its leaves for each new batch.

Cost tree model

Llama 3.1 8B · recorded cost treeScroll within the tree to inspect every operation.

Sum Llama 3.1 8B · one iteration

Leaf Embedding

0.005 ms Triton

Scale 32 decoder layers

Sum Decoder layer

Sum Pre-attention

Leaf Input RMSNorm

0.006 ms FlashInfer

Leaf QKV projection

0.074 ms PyTorch

Leaf Attention

0.040 ms vLLM CUDA · FA3

Sum Post-attention

Leaf Output projection

0.046 ms PyTorch

Leaf RMSNorm

0.006 ms FlashInfer

Leaf Gate/up projection

0.347 ms PyTorch

Leaf SiLU activation

0.023 ms Triton

Leaf Down projection

0.167 ms PyTorch

Leaf Final RMSNorm

0.006 ms FlashInfer

Leaf LM head

0.238 ms PyTorch

Cost tree phase

The model’s operations are assembled into a cost tree once. Sum combines sequential work, Max combines parallel ranks, and Scale represents repeated layers.

Predicted iteration time

22.942 ms

1,024-token prefill BF16 · 1 × H200

Recorded simulation · iteration 1. One prefill request, no cached prefix, 32 decoder layers.

This is the recorded model tree, from embedding through the decoder layers to the LM head. Operations shown as single leaves include their underlying kernel costs. Timings are per invocation; the original tree determines the iteration total.

Organize workers into a serving system.

Workers manage local requests. Pools organize workers. The orchestrator coordinates work across pools.

Serving hierarchy

A worker advances its assigned requests.

A worker tracks local request and KV state, forms batches, and queries the model’s cost tree to determine when work finishes. One worker can span one GPU or a parallel group.

Worker A

Shell

Sets the execution cadence by forming a batch, building its input, and completing the iteration.

Admission

Chooses which pending request starts next and manages its lifecycle.

KV

Owns cache capacity and each request’s residency, from reservation to release.

Execution

Turns admitted work and KV state into the model input for this iteration.

Cost Tree

Returns the predicted cost of the model input, which sets the next compute completion time.

Sweep parameters systematically.

Define a serving setup in YAML, then vary request rates, GPU allocation, and kernel backends. The launcher expands the combinations and applies your constraints.

Launcher demonstrations

Launcher YAML Illustrative excerpt

One parameter, three arrival rates.

Seven layers. Explicit responsibilities.

Timing flows from layer 1, the kernel, to layer 7, the run. State starts at layer 5, the worker. Select a layer to inspect its contract.

Measured foundation

Cost composition

Stateful execution

L1 Kernel / Profiling

Measured foundation

Measure kernel costs on hardware.

Python profiles real kernels on real GPUs. Rust builds an interpolating timing cache from those samples and evaluates it for each kernel shape.

Examples

Single GEMM

Use matrix dimensions, precision, and GPU configuration to query a cached kernel cost.

RMSNorm

Query normalization cost by token count for a configured hidden size and precision.

Start exploring your serving setup with ServingStudio.

Get started on GitHub

ServingStudio Simulate, understand, and optimize LLM serving. Page source


This page as Markdown: https://servingstudio.cs.washington.edu/architecture.md. A guide to the whole site for AI agents: llms.txt.