[ServingStudio](https://servingstudio.cs.washington.edu/)

Architecture

# How kernel measurements become serving predictions.

A kernel is measured once on the target GPU. Higher layers compose that measurement into a model, then into the workers and pools that serve requests. The top layer predicts a whole serving run.

Explore the architecture

## Every prediction starts with a measurement.

Kernel timings come from measurements on real hardware. During simulation, cached costs are evaluated for the current shapes and composed into a serving prediction. That prediction can be traced back to the kernels that support it.

### A real kernel, on a real GPU

Run a real kernel with concrete inputs, time it, and record the result on the target GPU.

### One row in the database

The row stores the measured timing for reuse across experiments.

### An interpolating cache

The cache interpolates measured samples for fast cost queries.

### The cost at this shape

The simulator evaluates the cache for each input shape as the run progresses.

## Assemble kernels into a model.

A cost tree captures how kernels run in sequence, across parallel ranks, and through repeated layers. Build it once, then evaluate its leaves for each new batch.

Cost tree model

Llama 3.1 8B · recorded cost treeScroll within the tree to inspect every operation.

**Sum** **Llama 3.1 8B · one iteration**

**Leaf** **Embedding**

0.005 ms Triton

**Scale** **32 decoder layers**

**Sum** **Decoder layer**

**Sum** **Pre-attention**

**Leaf** **Input RMSNorm**

0.006 ms FlashInfer

**Leaf** **QKV projection**

0.074 ms PyTorch

**Leaf** **Attention**

0.040 ms vLLM CUDA · FA3

**Sum** **Post-attention**

**Leaf** **Output projection**

0.046 ms PyTorch

**Leaf** **RMSNorm**

0.006 ms FlashInfer

**Leaf** **Gate/up projection**

0.347 ms PyTorch

**Leaf** **SiLU activation**

0.023 ms Triton

**Leaf** **Down projection**

0.167 ms PyTorch

**Leaf** **Final RMSNorm**

0.006 ms FlashInfer

**Leaf** **LM head**

0.238 ms PyTorch

Cost tree phase

The model’s operations are assembled into a cost tree once. Sum combines sequential work, Max combines parallel ranks, and Scale represents repeated layers.

Predicted iteration time

**22.942 ms**

1,024-token prefill BF16 · 1 × H200

Recorded simulation · iteration 1. One prefill request, no cached prefix, 32 decoder layers.

This is the recorded model tree, from embedding through the decoder layers to the LM head. Operations shown as single leaves include their underlying kernel costs. Timings are per invocation; the original tree determines the iteration total.

## Organize workers into a serving system.

Workers manage local requests. Pools organize workers. The orchestrator coordinates work across pools.

Serving hierarchy

### A worker advances its assigned requests.

A worker tracks local request and KV state, forms batches, and queries the model’s cost tree to determine when work finishes. One worker can span one GPU or a parallel group.

**Worker A**

**Shell**

Sets the execution cadence by forming a batch, building its input, and completing the iteration.

**Admission**

Chooses which pending request starts next and manages its lifecycle.

**KV**

Owns cache capacity and each request’s residency, from reservation to release.

**Execution**

Turns admitted work and KV state into the model input for this iteration.

**Cost Tree**

Returns the predicted cost of the model input, which sets the next compute completion time.

## Sweep parameters systematically.

Define a serving setup in YAML, then vary request rates, GPU allocation, and kernel backends. The launcher expands the combinations and applies your constraints.

Launcher demonstrations

Launcher YAML Illustrative excerpt

One parameter, three arrival rates.

## Seven layers. Explicit responsibilities.

Timing flows from layer 1, the kernel, to layer 7, the run. State starts at layer 5, the worker. Select a layer to inspect its contract.

Measured foundation

Cost composition

Stateful execution

L1 **Kernel / Profiling**

Measured foundation

### Measure kernel costs on hardware.

Python profiles real kernels on real GPUs. Rust builds an interpolating timing cache from those samples and evaluates it for each kernel shape.

Examples

**Single GEMM**

Use matrix dimensions, precision, and GPU configuration to query a cached kernel cost.

**RMSNorm**

Query normalization cost by token count for a configured hidden size and precision.

## Start exploring your serving setup with ServingStudio.

[Get started on GitHub](https://github.com/SyFI-ServingStudio/ServingStudio)

[ServingStudio](https://servingstudio.cs.washington.edu/) Simulate, understand, and optimize LLM serving. [Page source](https://github.com/SyFI-ServingStudio/ServingStudioIntro)

---

This page as Markdown: https://servingstudio.cs.washington.edu/architecture.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
