# ServingStudio > Use measured simulation, detailed performance analysis, and an Agent to improve LLM serving systems. Compare configurations using performance models grounded in real GPU measurements, then let an agent build and validate the strongest candidates in real serving frameworks. A kernel is measured once on the target GPU. Higher layers compose that measurement into a model, then into the workers and pools that serve requests. The top layer predicts a whole serving run. Every page below has a Markdown version, linked here. The Models, Kernels and Simulate pages read ServingStudio Sim's public JSON API at https://servingstudio.cs.washington.edu/api/public/v1; their Markdown, and one page per kernel kind and per model deployment, were generated from it at build time (Sim commit db977adc881595c5b48aa070bddc7cd6f33db130). Source code: https://github.com/SyFI-ServingStudio/ServingStudio. ## Pages - [Overview](https://servingstudio.cs.washington.edu/index.md): Use measured simulation, detailed performance analysis, and an Agent to improve LLM serving systems. - [Features](https://servingstudio.cs.washington.edu/features.md): Explore ServingStudio’s simulation, analysis, configuration, and Agent capabilities for LLM serving. - [Architecture](https://servingstudio.cs.washington.edu/architecture.md): See how ServingStudio turns measured GPU kernel timings into LLM serving predictions and actionable analysis. - [Models](https://servingstudio.cs.washington.edu/models.md): The models ServingStudio simulates and how each is deployed: the cost tree that composes one iteration's time from measured kernels, and a live prediction of a batch of your own. - [Simulate](https://servingstudio.cs.washington.edu/simulate.md): Simulate an LLM serving deployment on recorded, generated or uploaded requests, and read the TTFT, TPOT, end-to-end latency and throughput it gives. - [Kernels](https://servingstudio.cs.washington.edu/kernels.md): Measured GPU time for the kernels LLM serving runs, on NVIDIA H200 and B200. Browse, plot, and download the rows. - [Blog](https://servingstudio.cs.washington.edu/blog.md): Research, engineering, and updates from ServingStudio. ## Public API - [API base](https://servingstudio.cs.washington.edu/api/public/v1): JSON over HTTPS; `/health`, `/models`, `/kernels`, `/simulations/presets` and `/workloads` list what it serves; `POST /predict` and `POST /simulate` run work - [OpenAPI schema](https://servingstudio.cs.washington.edu/api/public/v1/openapi.json): every route - [Interactive API docs](https://servingstudio.cs.washington.edu/api/public/v1/docs) ## Agent skills Install them all with `npx skills add https://github.com/SyFI-ServingStudio/ServingStudioSim/tree/main/public_api`. - [servingstudio-kernel-performance](https://servingstudio.cs.washington.edu/skills/servingstudio-kernel-performance.md): Look up measured GPU kernel performance for LLM serving (GEMM, attention, MoE, normalization, communication) from the ServingStudio Kernel Library API. Not for end-to-end serving latency. - [servingstudio-simulate](https://servingstudio.cs.washington.edu/skills/servingstudio-simulate.md): Simulate an LLM serving deployment (model, GPU, parallelism, worker, replicas) on a request workload through the ServingStudio public API: a recorded capture, a generated trace or an uploaded CSV, with TTFT, TPOT, end-to-end latency and throughput. Not for the time of one batch (that is a timing prediction). - [servingstudio-timing-predict](https://servingstudio.cs.washington.edu/skills/servingstudio-timing-predict.md): Predict how long one LLM serving iteration takes for given batch shapes on a published deployment (model, GPU, parallelism, MoE routing) through the ServingStudio public API, with a per-kernel breakdown. Not for request queueing or end-to-end latency. ## Model deployments - [Llama 3.1 8B: Single GPU](https://servingstudio.cs.washington.edu/models/Llama-3.1-8B/llama3_dense.md) - [Llama 3.1 8B: TP](https://servingstudio.cs.washington.edu/models/Llama-3.1-8B/llama3_dense_tp.md) - [Llama 3.1 8B: DP attention, TP FFN](https://servingstudio.cs.washington.edu/models/Llama-3.1-8B/llama3_dp_attn_tp_ffn.md) - [Qwen3 235B-A22B: Attention side of an attention–FFN split](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B/qwen3_attn_tp.md) - [Qwen3 235B-A22B: FFN side of an attention–FFN split](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B/qwen3_ffn_moe.md) - [Qwen3 235B-A22B: DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B/qwen3_moe_dp_attn_ep_ffn.md) - [Qwen3 235B-A22B FP8: Attention side of an attention–FFN split](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_attn_tp.md) - [Qwen3 235B-A22B FP8: FFN side of an attention–FFN split](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_fp8_ffn_moe.md) - [Qwen3 235B-A22B FP8: DP attention, EP MoE, DeepGEMM experts](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_moe_fp8_dp_attn_ep_ffn.md) - [Qwen3 235B-A22B FP8: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_vllm_moe_dp_attn_ep_ffn.md) - [Qwen3 235B-A22B Thinking 2507 FP8: DP attention, EP MoE, DeepGEMM experts](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_moe_fp8_dp_attn_ep_ffn.md) - [Qwen3 235B-A22B Thinking 2507 FP8: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_vllm_moe_dp_attn_ep_ffn.md) - [Qwen3 Coder 480B-A35B: Attention side of an attention–FFN split](https://servingstudio.cs.washington.edu/models/Qwen3-Coder-480B-A35B-Instruct/qwen3_attn_tp.md) - [Qwen3 Coder 480B-A35B: FFN side of an attention–FFN split](https://servingstudio.cs.washington.edu/models/Qwen3-Coder-480B-A35B-Instruct/qwen3_fp8_ffn_moe.md) - [Qwen3.6 35B-A3B FP8: Single GPU](https://servingstudio.cs.washington.edu/models/Qwen3.6-35B-A3B-FP8/qwen36_local.md) - [Qwen3.8 27B NVFP4: vLLM, one GPU](https://servingstudio.cs.washington.edu/models/Qwen3.8-27B-NVFP4/qwen35_vllm_nvfp4_gdn_dense.md) - [Qwen3.8 Flash Next NVFP4: vLLM, one GPU, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative.md) - [DeepSeek V4 Flash 0731: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm.md) - [DeepSeek V4 Flash 0731: vLLM, DP attention, EP, parallel streams serialized](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams.md) - [GLM-5.2: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2/glm52_vllm_dsa_moe.md) - [GLM-5.2 FP8: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-FP8/glm52_vllm_dsa_moe.md) - [GLM-5.2 NVFP4: SGLang, TP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe.md) - [GLM-5.2 NVFP4: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe.md) - [GLM-5.2 NVFP4: vLLM, TP and EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe.md) - [GLM-5.2 NVFP4: vLLM, TP and EP, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative.md) - [GLM-5.2 NVFP4: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_pp_dsa_moe.md) - [GLM-5.3 NVFP4: vLLM, TP and EP, DFlash speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2.md) - [GLM-5.3-Flash: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe.md) - [GLM-5.3-Flash: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe.md) - [GLM-5.3-Flash: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe.md) - [GLM-5.3-Flash NVFP4: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe.md) - [GLM-5.3-Flash NVFP4: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe.md) - [GLM-5.3-Flash NVFP4: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe.md) - [DeepSeek V4.1 Flash: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash/deepseek_v41_vllm.md) - [DeepSeek V4.1 Flash: vLLM, TP attention, EP MoE, side streams serialized](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash/deepseek_v41_vllm_serial_streams.md) - [DeepSeek V4.1 Flash 1017: vLLM, DP attention, EP MoE, DSpark speculative decoding](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dp_attn_ep_moe_dspark.md) - [DeepSeek V4.1 Flash 1017: vLLM, TP attention, EP MoE, DSpark speculative decoding](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dspark.md) - [Inkling NVFP4: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Inkling-NVFP4/inkling_vllm_nvfp4_rel_attn_moe.md) - [Inkling Small NVFP4: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Inkling-Small-NVFP4/inkling_vllm_nvfp4_rel_attn_moe.md) - [MiMo V2.6 Flash RL: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/MiMo-V2.6-Flash-RL/mimo_v2_vllm_fp8_swa_moe.md) ## Kernel kinds - [All-reduce (all_reduce)](https://servingstudio.cs.washington.edu/kernels/all_reduce.md): Sum one buffer across the GPUs of a tensor-parallel group. - [All-reduce, FlashInfer kernels (all_reduce_fusion)](https://servingstudio.cs.washington.edu/kernels/all_reduce_fusion.md): Sum a [num_tokens, hidden_dim] tensor across the GPUs of a tensor-parallel group with FlashInfer's all-reduce. - [All-reduce + residual add + RMSNorm (all_reduce_residual_rms_norm)](https://servingstudio.cs.washington.edu/kernels/all_reduce_residual_rms_norm.md): Sum a tensor across GPUs, add the residual and RMS-normalize the result in a fused call. - [Batched GEMM (batched_gemm)](https://servingstudio.cs.washington.edu/kernels/batched_gemm.md): Multiply one activation and weight matrix pair per attention head (MLA) or per head group (grouped output projection). - [BF16 fused MoE (bf16_fused_moe)](https://servingstudio.cs.washington.edu/kernels/bf16_fused_moe.md): Route tokens through BF16 experts and combine their gated MLP outputs. - [Clamped SwiGLU (clamped_swiglu)](https://servingstudio.cs.washington.edu/kernels/clamped_swiglu.md): Clamp the gate and up projections, then multiply SiLU of the gate by the up projection. - [Compressed sparse MLA decode (compressed_sparse_mla_decode)](https://servingstudio.cs.washington.edu/kernels/compressed_sparse_mla_decode.md): Attend from each decode query to sliding-window and optional compressed keys. - [Compressed sparse MLA prefill (compressed_sparse_mla_prefill)](https://servingstudio.cs.washington.edu/kernels/compressed_sparse_mla_prefill.md): Attend from prefill queries to sliding-window and optional compressed keys. - [Compressed sparse MLA with fused RoPE and FP8 cast (compressed_sparse_mla_rope_cast)](https://servingstudio.cs.washington.edu/kernels/compressed_sparse_mla_rope_cast.md): Rotate the queries, attend to sliding-window and selected compressed keys, then un-rotate the output and cast it to FP8, in one attention segment. - [Compressed prefill indexer logits (dsa_compressed_mqa_logits_prefill)](https://servingstudio.cs.washington.edu/kernels/dsa_compressed_mqa_logits_prefill.md): Score prefill queries against compressed keys for sparse token selection. - [Compressed prefill indexer top-k (dsa_compressed_topk_prefill)](https://servingstudio.cs.washington.edu/kernels/dsa_compressed_topk_prefill.md): Select compressed cache positions from indexer logits for each prefill query. - [Indexer key cache append (dsa_index_cache_append)](https://servingstudio.cs.washington.edu/kernels/dsa_index_cache_append.md): Quantize new DSA index keys and write them into a paged FP8 cache. - [Indexer query RoPE and FP8 quantization (dsa_indexer_q_rope_quant)](https://servingstudio.cs.washington.edu/kernels/dsa_indexer_q_rope_quant.md): Rotate indexer queries and quantize them to FP8 while producing per-head weights. - [Indexer query RoPE and FP8 quantization, scales in weights (dsa_indexer_q_rope_quant_weight_fold)](https://servingstudio.cs.washington.edu/kernels/dsa_indexer_q_rope_quant_weight_fold.md): Rotate and quantize indexer queries while folding their FP8 scales into head weights. - [Prefill indexer logits (dsa_mqa_logits_prefill)](https://servingstudio.cs.washington.edu/kernels/dsa_mqa_logits_prefill.md): Score each prefill query against causal index keys and reduce across indexer heads. - [Decode indexer logits (dsa_paged_mqa_logits_decode)](https://servingstudio.cs.washington.edu/kernels/dsa_paged_mqa_logits_decode.md): Score decode queries against paged index keys and reduce across indexer heads. - [Decode indexer top-k (dsa_persistent_topk_decode)](https://servingstudio.cs.washington.edu/kernels/dsa_persistent_topk_decode.md): Select request-local token positions from indexer logits for each DSA decode query. - [Sparse index remap (dsa_sparse_index_remap)](https://servingstudio.cs.washington.edu/kernels/dsa_sparse_index_remap.md): Map DSA-selected request-local token indices to cache or prefill-workspace positions. - [Sparse MLA attention (dsa_sparse_mla_attention)](https://servingstudio.cs.washington.edu/kernels/dsa_sparse_mla_attention.md): Attend to the DSA-selected cache tokens for each query head. - [Sparse MLA prefill (dsa_sparse_mla_prefill)](https://servingstudio.cs.washington.edu/kernels/dsa_sparse_mla_prefill.md): Attend to selected compressed KV tokens for a batch of prefill queries. - [Prefill indexer top-k (dsa_topk_prefill)](https://servingstudio.cs.washington.edu/kernels/dsa_topk_prefill.md): Choose the highest-scoring index-key positions for each prefill query. - [Elementwise byte pass (elementwise)](https://servingstudio.cs.washington.edu/kernels/elementwise.md): Read a whole multiple of the output size, or nothing, and write the output. - [Embedding row gather (embedding_row_gather)](https://servingstudio.cs.washington.edu/kernels/embedding_row_gather.md): Copy num_tokens × num_heads randomly addressed rows of a large table, kept in pinned host memory or HBM, into a device buffer. - [Engram n-gram embedding lookup (engram_lookup)](https://servingstudio.cs.washington.edu/kernels/engram_lookup.md): Gather hashed n-gram rows from a large FP8 table, dequantize them and write BF16 rows for the layers that consume them. - [FlashAttention-4 varlen attention (fa4_varlen_attention)](https://servingstudio.cs.washington.edu/kernels/fa4_varlen_attention.md): Paged causal or sliding-window GQA attention with optional attention sinks and a value head dim that may differ from the query/key head dim, over one mixed decode and prefill batch. - [Paged decode attention (flashinfer_attn_decode)](https://servingstudio.cs.washington.edu/kernels/flashinfer_attn_decode.md): One query token per request attending over its paged KV cache, for a whole decode batch. - [Causal prefill attention (flashinfer_attn_prefill)](https://servingstudio.cs.washington.edu/kernels/flashinfer_attn_prefill.md): Attend a block of new query tokens over its preceding context and itself. - [Non-causal rectangular attention (flashinfer_attn_rect)](https://servingstudio.cs.washington.edu/kernels/flashinfer_attn_rect.md): Attend every query token over every key and value token in one request. - [FP8 block quantization (fp8_block_quant)](https://servingstudio.cs.washington.edu/kernels/fp8_block_quant.md): Quantize BF16 expert inputs to FP8 E4M3 in 128-element blocks. - [FP8 block-scale grouped GEMM (fp8_blockscale_grouped_gemm)](https://servingstudio.cs.washington.edu/kernels/fp8_blockscale_grouped_gemm.md): Multiply routed fp8 token rows by block-scaled expert weights and write bf16 output. - [FP8 per-token-group quantization (fp8_per_token_group_quant)](https://servingstudio.cs.washington.edu/kernels/fp8_per_token_group_quant.md): Quantize BF16 activation rows to FP8 E4M3 with a scale per 128-element group. - [Decode causal convolution (gdn_causal_conv_decode)](https://servingstudio.cs.washington.edu/kernels/gdn_causal_conv_decode.md): Apply a depthwise causal convolution and SiLU to one token per request, updating the convolution state. - [Prefill causal convolution (gdn_causal_conv_prefill)](https://servingstudio.cs.washington.edu/kernels/gdn_causal_conv_prefill.md): Apply a depthwise causal convolution and SiLU to fresh sequences and save their final input samples as state. - [Fused chunked delta rule (gdn_chunk_delta_rule)](https://servingstudio.cs.washington.edu/kernels/gdn_chunk_delta_rule.md): Compute Gated DeltaNet prefill output and final recurrent states in one launch. - [Gated DeltaNet output RMSNorm (gdn_gated_rms_norm)](https://servingstudio.cs.washington.edu/kernels/gdn_gated_rms_norm.md): RMS-normalize each recurrent output row and multiply it by a SiLU gate. - [Prefill post-convolution split (gdn_prefill_post_conv)](https://servingstudio.cs.washington.edu/kernels/gdn_prefill_post_conv.md): Split the convolution output into normalized queries and keys, values, the log decay and beta. - [Recurrent decode (gdn_recurrent_decode)](https://servingstudio.cs.washington.edu/kernels/gdn_recurrent_decode.md): Update the gated delta-rule state and read one output token per request. - [Verify recurrence with output norm (gdn_recurrent_decode_gated_norm)](https://servingstudio.cs.washington.edu/kernels/gdn_recurrent_decode_gated_norm.md): Run the gated delta rule over each request's verify tokens and apply the gated RMSNorm, in one launch. - [FP32-output GEMM (gemm_fp32_output)](https://servingstudio.cs.washington.edu/kernels/gemm_fp32_output.md): Multiply bf16 or fp32 activations and weights while writing an fp32 result. - [Grouped GEMM (grouped_gemm)](https://servingstudio.cs.washington.edu/kernels/grouped_gemm.md): Multiply routed token batches by their respective MoE expert weights. - [Grouped RMSNorm (grouped_rms_norm)](https://servingstudio.cs.washington.edu/kernels/grouped_rms_norm.md): RMS-normalize each contiguous group of a row separately. - [HC combine (hc_combine)](https://servingstudio.cs.washington.edu/kernels/hc_combine.md): Add a block's output into every residual stream with a learned gate. - [HC combine + per-stream norm (hc_combine_norm)](https://servingstudio.cs.washington.edu/kernels/hc_combine_norm.md): Add a block's output into every residual stream with a learned gate, then RMS-normalize each stream, in one launch. - [HC gate mix (hc_gate_mix)](https://servingstudio.cs.washington.edu/kernels/hc_gate_mix.md): Collapse the normalized residual streams into one block input with a per-feature sigmoid gate. - [Indexed row copy (indexed_row_copy)](https://servingstudio.cs.washington.edu/kernels/indexed_row_copy.md): Gather, scatter or zero whole rows by index with eager PyTorch. - [Inverse RoPE and FP8 quantization (inv_rope_fp8_quant)](https://servingstudio.cs.washington.edu/kernels/inv_rope_fp8_quant.md): Undo RoPE on attention outputs and quantize them in groups for the output projection. - [KDA chunked prefill (kda_chunk_prefill)](https://servingstudio.cs.washington.edu/kernels/kda_chunk_prefill.md): Compute KDA prefill output and final recurrent states, with a decay gate per key channel, in one call. - [KDA recurrent decode (kda_recurrent_decode)](https://servingstudio.cs.washington.edu/kernels/kda_recurrent_decode.md): Update each request's KDA state, gated per key channel, and read one output token. - [Paged KV cache append (kv_cache_append)](https://servingstudio.cs.washington.edu/kernels/kv_cache_append.md): Write the keys and values of new tokens into their slots in a paged KV cache. - [KV compression and store (kv_compress_store)](https://servingstudio.cs.washington.edu/kernels/kv_compress_store.md): Save partial states, then compress and store keys at compression boundaries. - [Logits top-k (logits_topk)](https://servingstudio.cs.washington.edu/kernels/logits_topk.md): Select the largest top_k scores and their column indices from each score row. - [mHC post-block, pre-block and RMSNorm (mhc_fused_post_pre_rms_norm)](https://servingstudio.cs.washington.edu/kernels/mhc_fused_post_pre_rms_norm.md): Mix a block's output into the mHC residual streams, then compute the next block's mixing weights and normalized input, in one call. - [mHC pre-block with RMSNorm (mhc_pre_rms_norm)](https://servingstudio.cs.washington.edu/kernels/mhc_pre_rms_norm.md): From a token's parallel residual streams, compute the mHC mixing weights and the RMS-normalized input of the next block. - [Terminal mHC head (mhc_terminal_head)](https://servingstudio.cs.washington.edu/kernels/mhc_terminal_head.md): Collapse the final mHC residual channels into one hidden state and RMS-normalize it for the LM head. - [MLA cache append (mla_cache_append)](https://servingstudio.cs.washington.edu/kernels/mla_cache_append.md): Write compressed KV latents and rotary keys into paged MLA cache entries. - [MLA RoPE and FP8 quantization (mla_rope_quantize_fp8)](https://servingstudio.cs.washington.edu/kernels/mla_rope_quantize_fp8.md): Rotate MLA queries and keys, quantize them to FP8, and assemble each query. - [MoE block alignment (moe_align_block_size)](https://servingstudio.cs.washington.edu/kernels/moe_align_block_size.md): Group routed token IDs by expert and pad each group to a block boundary. - [MoE all-to-all transfer (moe_alltoall)](https://servingstudio.cs.washington.edu/kernels/moe_alltoall.md): Move token rows to expert GPUs or return expert outputs to their source GPUs. - [MoE all-to-all preparation (moe_alltoall_prepare)](https://servingstudio.cs.washington.edu/kernels/moe_alltoall_prepare.md): Build and exchange the row indices needed for expert-parallel dispatch. - [MoE all-gather of hidden states and router logits (moe_ep_all_gather)](https://servingstudio.cs.washington.edu/kernels/moe_ep_all_gather.md): Gather hidden states and router logits from every data-parallel GPU. - [MoE all-gather of quantized activations and top-k routing (moe_ep_quantized_all_gather)](https://servingstudio.cs.washington.edu/kernels/moe_ep_quantized_all_gather.md): Gather NVFP4 or MXFP8 activations, scales and top-k routing from every data-parallel GPU. - [MoE output reduce-scatter (moe_ep_reduce_scatter)](https://servingstudio.cs.washington.edu/kernels/moe_ep_reduce_scatter.md): Sum expert outputs across GPUs and return each GPU's token shard. - [MoE finalization with shared output (moe_finalize_fuse_shared)](https://servingstudio.cs.washington.edu/kernels/moe_finalize_fuse_shared.md): Gather and weight routed expert outputs, optionally adding the shared-expert row. - [MoE routing finalization (moe_finalize_routing)](https://servingstudio.cs.washington.edu/kernels/moe_finalize_routing.md): Unpermute, scale and sum local expert outputs into one row per original token. - [Fused MoE top-k routing (moe_fused_topk)](https://servingstudio.cs.washington.edu/kernels/moe_fused_topk.md): Select top-k experts from softmax router scores and renormalize their weights. - [Sink-renormalized MoE gate select (moe_sink_gate_select)](https://servingstudio.cs.washington.edu/kernels/moe_sink_gate_select.md): Pick each token's top-k routed experts by sigmoid score plus bias, then renormalize their logits together with the sink experts' logits. - [MoE expert sum (moe_sum)](https://servingstudio.cs.washington.edu/kernels/moe_sum.md): Sum the routed expert output rows for each token. - [Softplus-sqrt MoE routing (moe_topk_softplus_sqrt)](https://servingstudio.cs.washington.edu/kernels/moe_topk_softplus_sqrt.md): Select experts with sqrt-softplus scores or a token hash table and produce scaled weights. - [Summed RMSNorm with fused outputs (multi_add_rms_norm)](https://servingstudio.cs.washington.edu/kernels/multi_add_rms_norm.md): Sum one to three residual-stream tensors, RMS-normalize, and write any of the sum, the normalized rows and their FP8 quantization in one kernel. - [MXFP4 Marlin MoE GEMM (mxfp4_marlin_moe_gemm)](https://servingstudio.cs.washington.edu/kernels/mxfp4_marlin_moe_gemm.md): Multiply routed BF16 activations by packed MXFP4 expert weights in one Marlin launch. - [Block-scaled fused MoE (nvfp4_fused_moe)](https://servingstudio.cs.washington.edu/kernels/nvfp4_fused_moe.md): Route block-quantized tokens through packed expert weights and produce their MoE outputs. - [NVFP4 activation quantization (nvfp4_quant)](https://servingstudio.cs.washington.edu/kernels/nvfp4_quant.md): Quantize BF16 activation rows to packed NVFP4 values and E4M3 group scales. - [Inter-domain point-to-point transfer (p2p_inter)](https://servingstudio.cs.washington.edu/kernels/p2p_inter.md): Modeled time for one transfer between GPUs in different NVLink domains. - [Intra-domain point-to-point transfer (p2p_intra)](https://servingstudio.cs.washington.edu/kernels/p2p_intra.md): Send one payload between two GPUs in the same NVLink domain. - [Packed KV cache gather (packed_kv_cache_gather)](https://servingstudio.cs.washington.edu/kernels/packed_kv_cache_gather.md): Gather and dequantize paged FP8 keys into a BF16 prefill workspace. - [Paged batch decode attention (paged_attn_batch_decode)](https://servingstudio.cs.washington.edu/kernels/paged_attn_batch_decode.md): One-token-per-request GQA attention over a paged KV cache for a whole decode batch, keyed by the batch's longest context and total context. - [Paged batch prefill attention (paged_attn_batch_prefill)](https://servingstudio.cs.washington.edu/kernels/paged_attn_batch_prefill.md): Causal GQA attention of several requests' new tokens over their cached prefixes and themselves, through a paged KV cache, in one call. - [PLE similarity gate (ple_gate)](https://servingstudio.cs.washington.edu/kernels/ple_gate.md): Gate a looked-up n-gram value by its normalized key's similarity to each residual stream, and RMS-normalize the result. - [PLE n-gram hash ids (ple_ngram_ids)](https://servingstudio.cs.washington.edu/kernels/ple_ngram_ids.md): Hash each token's preceding n-grams into per-head embedding-table ids. - [PLE dilated short convolution (ple_short_conv)](https://servingstudio.cs.washington.edu/kernels/ple_short_conv.md): Depthwise dilated causal convolution with SiLU over the PLE stream, added into the residual, with the per-request conv state carried across steps. - [Query and KV RMSNorm (q_kv_rms_norm)](https://servingstudio.cs.washington.edu/kernels/q_kv_rms_norm.md): RMS-normalize query-side and KV-side projections with separate weights. - [Query padding, KV RoPE and MXFP8 window insert (q_pad_kv_rope_mxfp8_insert)](https://servingstudio.cs.washington.edu/kernels/q_pad_kv_rope_mxfp8_insert.md): Copy the live query heads into a zero-padded buffer while rotating, quantizing to MXFP8 and inserting KV rows into a sliding-window cache. - [QK RMSNorm + RoPE + gate split (qk_rms_norm_rope_gate)](https://servingstudio.cs.washington.edu/kernels/qk_rms_norm_rope_gate.md): RMS-normalize every query and key head, rotate its leading rotary_dim elements, and split the query projection's output gate off, in one launch. - [Query norm, RoPE and KV insert (qnorm_rope_kv_insert)](https://servingstudio.cs.washington.edu/kernels/qnorm_rope_kv_insert.md): Normalize and rotate queries while rotating and inserting mixed FP8/bf16 KV rows into a packed MLA cache. - [Relative-position attention (rel_bias_attention)](https://servingstudio.cs.washington.edu/kernels/rel_bias_attention.md): Paged causal or sliding-window GQA attention with a per-query relative-position logit added to every score, over one mixed decode and prefill batch. - [Residual add + RMSNorm (residual_rms_norm)](https://servingstudio.cs.washington.edu/kernels/residual_rms_norm.md): Add the residual stream, then normalize: the fused step between transformer blocks. - [RMSNorm (rms_norm)](https://servingstudio.cs.washington.edu/kernels/rms_norm.md): Normalize each token's hidden vector by its root mean square and scale it by a weight. - [Reduce-scatter + short conv + all-gather + residual + RMSNorm (rs_sconv_ag_residual_rms_norm)](https://servingstudio.cs.washington.edu/kernels/rs_sconv_ag_residual_rms_norm.md): Sum sublayer partials across GPUs, run a causal short convolution on each GPU's hidden shard, gather it back, add the residual and RMS-normalize. - [Scaled SiLU (scaled_silu)](https://servingstudio.cs.washington.edu/kernels/scaled_silu.md): Apply SiLU to every element after dividing it by a constant. - [Short-conv QKVR prep (sconv_qkvr_prep)](https://servingstudio.cs.washington.edu/kernels/sconv_qkvr_prep.md): Short-conv K and V, RMS-normalize q and k, scale by tau, insert K/V into the paged cache and project the relative-position logits, in one call. - [SiLU and mul (silu_and_mul)](https://servingstudio.cs.washington.edu/kernels/silu_and_mul.md): Multiply SiLU of the gate half of each row by its up half. - [Dense GEMM (single_gemm)](https://servingstudio.cs.washington.edu/kernels/single_gemm.md): One matrix multiply: an activation block of m tokens times a weight matrix. - [Sparse GQA paged attention (sparse_gqa_paged_attention)](https://servingstudio.cs.washington.edu/kernels/sparse_gqa_paged_attention.md): Attend each query row to its indexer-selected tokens in a paged K/V cache and apply the sigmoid output gate. - [Sparse index expansion (sparse_index_expand)](https://servingstudio.cs.washington.edu/kernels/sparse_index_expand.md): Turn selected compressed groups into packed logical token ids plus a count. - [Sparse indexer decode logits (sparse_indexer_logits_decode)](https://servingstudio.cs.washington.edu/kernels/sparse_indexer_logits_decode.md): Score decode rows against every visible compressed key with a ReLU head sum. - [Sparse indexer prefill logits (sparse_indexer_logits_prefill)](https://servingstudio.cs.washington.edu/kernels/sparse_indexer_logits_prefill.md): Score a prefill chunk's rows against their causally visible compressed keys. - [Sparse indexer prepare (sparse_indexer_prepare)](https://servingstudio.cs.washington.edu/kernels/sparse_indexer_prepare.md): Normalize and rotate indexer queries and pool completed key groups into the cache. - [Static FP8 quantization (static_fp8_quant)](https://servingstudio.cs.washington.edu/kernels/static_fp8_quant.md): Scale an activation by a fixed per-tensor scale and round it to FP8 E4M3. - [vLLM fused MoE GEMM (vllm_fused_moe)](https://servingstudio.cs.washington.edu/kernels/vllm_fused_moe.md): Gather selected token rows and multiply them by FP8 expert weights in one Triton launch. - [vLLM MLA query RoPE (vllm_mla_rope)](https://servingstudio.cs.washington.edu/kernels/vllm_mla_rope.md): Rotate the MLA query's positional columns and materialize the full query tensor. ## Optional - [SSEngine-GLM5.3-Flash: Pipeline-Parallel Prefill with Tiered Prefix Caching for Agentic Workloads](https://servingstudio.cs.washington.edu/blog/ssengine-glm53-flash/index.md): An open-source prefill engine for GLM-5.3-Flash on four B200 GPUs that combines pipeline parallelism, a prefix cache across HBM, DRAM and SSD, and a shortest-remaining-prefill-first scheduler. - [Beyond GQA: Characterizing Modern Attention Variants with ServingStudio](https://servingstudio.cs.washington.edu/blog/attention-variants/index.md): How the attention designs of today's open models trade memory, bytes read per step and compute against GQA, with measured B200 kernel times. - [Introducing ServingStudio: An Integrated Workbench for Simulating, Analyzing, and Optimizing LLM Serving Systems](https://servingstudio.cs.washington.edu/blog/introducing-servingstudio/index.md): Compare serving configurations using measured GPU timings, then let an agent implement and validate promising optimizations in real serving frameworks. - [Source code](https://github.com/SyFI-ServingStudio/ServingStudio)