Kernel Library | ServingStudio
Explore measured GPU performance across the kernels, shapes, and backends that power LLM serving.
Interactive page: https://servingstudio.cs.washington.edu/kernels.html. The page reads its lists from ServingStudio Sim's public API at https://servingstudio.cs.washington.edu/api/public/v1 (OpenAPI: https://servingstudio.cs.washington.edu/api/public/v1/openapi.json). An agent can use it through the servingstudio-kernel-performance skill.
This text was generated when the site was built, from Sim commit db977adc881595c5b48aa070bddc7cd6f33db130; the API has the current lists.
The Kernel Library lists 100 kernel kinds measured on NVIDIA B200, NVIDIA H200, last measured 2026-11-07. Each kind is one operation; its rows are measurements, one per combination of argument values, GPU and backend. Read the rows of a kind from /api/public/v1/kernels/{kind}/rows (?format=csv for CSV).
GEMM
- Batched GEMM (
batched_gemm): Multiply one activation and weight matrix pair per attention head (MLA) or per head group (grouped output projection). backends:torch_mla_q_absorb,torch_mla_v_up,torch_mla_q_absorb_no_rope,torch_mla_v_up_unpadded,deepgemm_mxfp8_einsum_grouped_o_proj; precisions:bf16,mxfp8_e4m3; GPUs: NVIDIA B200, NVIDIA H200; 1,360 measured rows. Details: Markdown - FP32-output GEMM (
gemm_fp32_output): Multiply bf16 or fp32 activations and weights while writing an fp32 result. backends:torch_cublas,sglang_router_auto,vllm_ll_bf16_auto; precisions:bf16,fp32; GPUs: NVIDIA B200, NVIDIA H200; 1,048 measured rows. Details: Markdown - Dense GEMM (
single_gemm): One matrix multiply: an activation block of m tokens times a weight matrix. backends:torch,torch_linear_vllm,torch_linear,sglang_bf16_auto,sglang_fused_a_auto,deepgemm,flashinfer_fp8_tensor,flashinfer_cutedsl,flashinfer_mxfp8,flashinfer_mxfp8_prequantized; precisions:bf16,fp16,fp8_e4m3,mxfp8_e4m3,nvfp4_e2m1; GPUs: NVIDIA B200, NVIDIA H200; 16,451 measured rows. Details: Markdown
Attention
- Compressed sparse MLA decode (
compressed_sparse_mla_decode): Attend from each decode query to sliding-window and optional compressed keys. group: Compressed sparse MLA; backends:vllm_flashmla_fp8_cudagraph; precisions:bf16; GPUs: NVIDIA H200; 386 measured rows. Details: Markdown - Compressed sparse MLA prefill (
compressed_sparse_mla_prefill): Attend from prefill queries to sliding-window and optional compressed keys. group: Compressed sparse MLA; backends:vllm_flashmla_bf16; precisions:bf16; GPUs: NVIDIA H200; 2,259 measured rows. Details: Markdown - Compressed sparse MLA with fused RoPE and FP8 cast (
compressed_sparse_mla_rope_cast): Rotate the queries, attend to sliding-window and selected compressed keys, then un-rotate the output and cast it to FP8, in one attention segment. group: Compressed sparse MLA; backends:torch,flashmla_mega,swapab_cutedsl; precisions:bf16; GPUs: NVIDIA B200; 5,673 measured rows. Details: Markdown - Compressed prefill indexer logits (
dsa_compressed_mqa_logits_prefill): Score prefill queries against compressed keys for sparse token selection. group: DSA; backends:vllm_deepgemm_fp8; precisions:fp8_e4m3; GPUs: NVIDIA H200; 622 measured rows. Details: Markdown - Compressed prefill indexer top-k (
dsa_compressed_topk_prefill): Select compressed cache positions from indexer logits for each prefill query. group: DSA; backends:vllm_cuda; precisions:fp32; GPUs: NVIDIA H200; 622 measured rows. Details: Markdown - Indexer key cache append (
dsa_index_cache_append): Quantize new DSA index keys and write them into a paged FP8 cache. group: DSA; backends:torch,vllm_cuda,sglang_fused_norm_rope_store; precisions:bf16; GPUs: NVIDIA B200, NVIDIA H200; 279 measured rows. Details: Markdown - Indexer query RoPE and FP8 quantization (
dsa_indexer_q_rope_quant): Rotate indexer queries and quantize them to FP8 while producing per-head weights. group: DSA; backends:sglang_cuda; precisions:bf16; GPUs: NVIDIA B200; 68 measured rows. Details: Markdown - Indexer query RoPE and FP8 quantization, scales in weights (
dsa_indexer_q_rope_quant_weight_fold): Rotate and quantize indexer queries while folding their FP8 scales into head weights. group: DSA; backends:vllm_cutedsl_fp8; precisions:bf16; GPUs: NVIDIA H200; 40 measured rows. Details: Markdown - Prefill indexer logits (
dsa_mqa_logits_prefill): Score each prefill query against causal index keys and reduce across indexer heads. group: DSA; backends:torch,deepgemm_fp8; precisions:fp8_e4m3; GPUs: NVIDIA B200, NVIDIA H200; 2,057 measured rows. Details: Markdown - Decode indexer logits (
dsa_paged_mqa_logits_decode): Score decode queries against paged index keys and reduce across indexer heads. group: DSA; backends:torch,deepgemm_fp8; precisions:fp8_e4m3; GPUs: NVIDIA B200, NVIDIA H200; 18,569 measured rows. Details: Markdown - Decode indexer top-k (
dsa_persistent_topk_decode): Select request-local token positions from indexer logits for each DSA decode query. group: DSA; backends:torch,vllm_cuda,vllm_fork_cuda,vllm_cuda_cooperative; precisions:fp32; GPUs: NVIDIA B200, NVIDIA H200; 24,369 measured rows. Details: Markdown - Sparse index remap (
dsa_sparse_index_remap): Map DSA-selected request-local token indices to cache or prefill-workspace positions. group: DSA; backends:torch,vllm_triton; GPUs: NVIDIA B200; 4,227 measured rows. Details: Markdown - Sparse MLA attention (
dsa_sparse_mla_attention): Attend to the DSA-selected cache tokens for each query head. group: DSA; backends:torch,flashinfer_trtllm_fp8,vllm_flashmla_bf16; precisions:bf16,fp8_e4m3; GPUs: NVIDIA B200, NVIDIA H200; 21,272 measured rows. Details: Markdown - Sparse MLA prefill (
dsa_sparse_mla_prefill): Attend to selected compressed KV tokens for a batch of prefill queries. group: DSA; backends:flashinfer_trtllm_fp8; precisions:fp8_e4m3; GPUs: NVIDIA B200; 1,630 measured rows. Details: Markdown - Prefill indexer top-k (
dsa_topk_prefill): Choose the highest-scoring index-key positions for each prefill query. group: DSA; backends:torch,vllm_cuda,deep_select,sglang_cuda; precisions:fp32; GPUs: NVIDIA B200, NVIDIA H200; 2,115 measured rows. Details: Markdown - FlashAttention-4 varlen attention (
fa4_varlen_attention): Paged causal or sliding-window GQA attention with optional attention sinks and a value head dim that may differ from the query/key head dim, over one mixed decode and prefill batch. group: MHA / GQA; backends:vllm_fa4; precisions:bf16; GPUs: NVIDIA B200; 1,136 measured rows. Details: Markdown - Paged decode attention (
flashinfer_attn_decode): One query token per request attending over its paged KV cache, for a whole decode batch. group: MHA / GQA; backends:fa2,fa2_cudagraph,fa3,cudnn; precisions:bf16; GPUs: NVIDIA H200; 3,888 measured rows. Details: Markdown - Causal prefill attention (
flashinfer_attn_prefill): Attend a block of new query tokens over its preceding context and itself. group: MHA / GQA; backends:fa2,fa3,cudnn; precisions:bf16,fp8_e4m3; GPUs: NVIDIA H200; 2,835 measured rows. Details: Markdown - Non-causal rectangular attention (
flashinfer_attn_rect): Attend every query token over every key and value token in one request. group: MHA / GQA; backends:fa2,fa3,cudnn; precisions:bf16; GPUs: NVIDIA B200; 3,969 measured rows. Details: Markdown - Decode causal convolution (
gdn_causal_conv_decode): Apply a depthwise causal convolution and SiLU to one token per request, updating the convolution state. group: Gated DeltaNet; backends:torch,vllm_triton; precisions:bf16; GPUs: NVIDIA B200, NVIDIA H200; 93 measured rows. Details: Markdown - Prefill causal convolution (
gdn_causal_conv_prefill): Apply a depthwise causal convolution and SiLU to fresh sequences and save their final input samples as state. group: Gated DeltaNet; backends:torch,vllm_triton,dao_channellast; precisions:bf16; GPUs: NVIDIA B200, NVIDIA H200; 1,216 measured rows. Details: Markdown - Fused chunked delta rule (
gdn_chunk_delta_rule): Compute Gated DeltaNet prefill output and final recurrent states in one launch. group: Gated DeltaNet; backends:flashinfer; precisions:bf16; GPUs: NVIDIA B200, NVIDIA H200; 84 measured rows. Details: Markdown - Prefill post-convolution split (
gdn_prefill_post_conv): Split the convolution output into normalized queries and keys, values, the log decay and beta. group: Gated DeltaNet; backends:torch,vllm_triton; precisions:bf16; GPUs: NVIDIA B200, NVIDIA H200; 71 measured rows. Details: Markdown - Recurrent decode (
gdn_recurrent_decode): Update the gated delta-rule state and read one output token per request. group: Gated DeltaNet; backends:torch,vllm_triton,vllm_triton_sigmoid_gating; precisions:bf16; GPUs: NVIDIA B200, NVIDIA H200; 75 measured rows. Details: Markdown - Verify recurrence with output norm (
gdn_recurrent_decode_gated_norm): Run the gated delta rule over each request's verify tokens and apply the gated RMSNorm, in one launch. group: Gated DeltaNet; backends:vllm_cuda; precisions:bf16; GPUs: NVIDIA B200; 104 measured rows. Details: Markdown - KDA chunked prefill (
kda_chunk_prefill): Compute KDA prefill output and final recurrent states, with a decay gate per key channel, in one call. group: Gated DeltaNet; backends:vllm_triton,flashinfer_tirx,flashinfer_cute_persistent,flashkda; precisions:bf16; GPUs: NVIDIA B200; 4,096 measured rows. Details: Markdown - KDA recurrent decode (
kda_recurrent_decode): Update each request's KDA state, gated per key channel, and read one output token. group: Gated DeltaNet; backends:vllm_triton; precisions:bf16; GPUs: NVIDIA B200; 36 measured rows. Details: Markdown - Paged KV cache append (
kv_cache_append): Write the keys and values of new tokens into their slots in a paged KV cache. group: MHA / GQA; backends:torch,vllm_cuda,vllm_triton_diffkv; precisions:bf16; GPUs: NVIDIA B200, NVIDIA H200; 1,131 measured rows. Details: Markdown - KV compression and store (
kv_compress_store): Save partial states, then compress and store keys at compression boundaries. group: Compressed sparse MLA; backends:vllm_cutedsl,vllm_triton; precisions:fp32; GPUs: NVIDIA H200; 361 measured rows. Details: Markdown - MLA cache append (
mla_cache_append): Write compressed KV latents and rotary keys into paged MLA cache entries. group: MLA; backends:torch,vllm_cuda,sglang_cuda; precisions:bf16,fp8_e4m3; GPUs: NVIDIA B200, NVIDIA H200; 419 measured rows. Details: Markdown - MLA RoPE and FP8 quantization (
mla_rope_quantize_fp8): Rotate MLA queries and keys, quantize them to FP8, and assemble each query. group: MLA; backends:flashinfer; precisions:bf16; GPUs: NVIDIA B200; 137 measured rows. Details: Markdown - Packed KV cache gather (
packed_kv_cache_gather): Gather and dequantize paged FP8 keys into a BF16 prefill workspace. group: Compressed sparse MLA; backends:vllm_cutedsl; precisions:bf16; GPUs: NVIDIA H200; 2 measured rows. Details: Markdown - Paged batch decode attention (
paged_attn_batch_decode): One-token-per-request GQA attention over a paged KV cache for a whole decode batch, keyed by the batch's longest context and total context. group: MHA / GQA; backends:flashinfer_trtllm_gen; precisions:fp8_e4m3; GPUs: NVIDIA B200; 701 measured rows. Details: Markdown - Paged batch prefill attention (
paged_attn_batch_prefill): Causal GQA attention of several requests' new tokens over their cached prefixes and themselves, through a paged KV cache, in one call. group: MHA / GQA; backends:flashinfer_trtllm_gen; precisions:fp8_e4m3; GPUs: NVIDIA B200; 1,080 measured rows. Details: Markdown - Query padding, KV RoPE and MXFP8 window insert (
q_pad_kv_rope_mxfp8_insert): Copy the live query heads into a zero-padded buffer while rotating, quantizing to MXFP8 and inserting KV rows into a sliding-window cache. group: Compressed sparse MLA; backends:vllm_cuda; precisions:bf16; GPUs: NVIDIA B200; 208 measured rows. Details: Markdown - QK RMSNorm + RoPE + gate split (
qk_rms_norm_rope_gate): RMS-normalize every query and key head, rotate its leading rotary_dim elements, and split the query projection's output gate off, in one launch. group: MHA / GQA; backends:vllm_triton; precisions:bf16; GPUs: NVIDIA B200; 136 measured rows. Details: Markdown - Query norm, RoPE and KV insert (
qnorm_rope_kv_insert): Normalize and rotate queries while rotating and inserting mixed FP8/bf16 KV rows into a packed MLA cache. group: MLA; backends:vllm_cuda; precisions:bf16; GPUs: NVIDIA H200; 339 measured rows. Details: Markdown - Relative-position attention (
rel_bias_attention): Paged causal or sliding-window GQA attention with a per-query relative-position logit added to every score, over one mixed decode and prefill batch. group: MHA / GQA; backends:tml_fa4; precisions:bf16; GPUs: NVIDIA B200; 6,472 measured rows. Details: Markdown - Short-conv QKVR prep (
sconv_qkvr_prep): Short-conv K and V, RMS-normalize q and k, scale by tau, insert K/V into the paged cache and project the relative-position logits, in one call. group: MHA / GQA; backends:vllm_triton; precisions:bf16; GPUs: NVIDIA B200; 147 measured rows. Details: Markdown - Sparse GQA paged attention (
sparse_gqa_paged_attention): Attend each query row to its indexer-selected tokens in a paged K/V cache and apply the sigmoid output gate. group: Sparse GQA; backends:vllm_triton; precisions:bf16; GPUs: NVIDIA B200; 868 measured rows. Details: Markdown - Sparse index expansion (
sparse_index_expand): Turn selected compressed groups into packed logical token ids plus a count. group: Sparse GQA; backends:vllm_triton; GPUs: NVIDIA B200; 68 measured rows. Details: Markdown - Sparse indexer decode logits (
sparse_indexer_logits_decode): Score decode rows against every visible compressed key with a ReLU head sum. group: Sparse GQA; backends:vllm_triton; precisions:fp8_e4m3; GPUs: NVIDIA B200; 964 measured rows. Details: Markdown - Sparse indexer prefill logits (
sparse_indexer_logits_prefill): Score a prefill chunk's rows against their causally visible compressed keys. group: Sparse GQA; backends:vllm_triton; precisions:fp8_e4m3; GPUs: NVIDIA B200; 80 measured rows. Details: Markdown - Sparse indexer prepare (
sparse_indexer_prepare): Normalize and rotate indexer queries and pool completed key groups into the cache. group: Sparse GQA; backends:vllm_triton; precisions:bf16; GPUs: NVIDIA B200; 68 measured rows. Details: Markdown - vLLM MLA query RoPE (
vllm_mla_rope): Rotate the MLA query's positional columns and materialize the full query tensor. group: MLA; backends:vllm_inductor; precisions:bf16; GPUs: NVIDIA B200, NVIDIA H200; 272 measured rows. Details: Markdown
MoE
- BF16 fused MoE (
bf16_fused_moe): Route tokens through BF16 experts and combine their gated MLP outputs. group: Expert compute; backends:flashinfer_trtllm_sm100; precisions:bf16; GPUs: NVIDIA B200; 6,324 measured rows. Details: Markdown - Clamped SwiGLU (
clamped_swiglu): Clamp the gate and up projections, then multiply SiLU of the gate by the up projection. group: Expert compute; backends:vllm_inductor; precisions:bf16; GPUs: NVIDIA H200; 68 measured rows. Details: Markdown - FP8 block-scale grouped GEMM (
fp8_blockscale_grouped_gemm): Multiply routed fp8 token rows by block-scaled expert weights and write bf16 output. group: Expert compute; backends:flashinfer_trtllm; precisions:fp8_e4m3; GPUs: NVIDIA H200; 3,678 measured rows. Details: Markdown - Grouped GEMM (
grouped_gemm): Multiply routed token batches by their respective MoE expert weights. group: Expert compute; backends:torch,deepgemm; precisions:bf16,fp8_e4m3; GPUs: NVIDIA H200; 9,652 measured rows. Details: Markdown - MoE block alignment (
moe_align_block_size): Group routed token IDs by expert and pad each group to a block boundary. group: Routing and combine; backends:vllm_cuda; GPUs: NVIDIA H200; 198 measured rows. Details: Markdown - MoE finalization with shared output (
moe_finalize_fuse_shared): Gather and weight routed expert outputs, optionally adding the shared-expert row. group: Routing and combine; backends:sglang_cuda; precisions:bf16; GPUs: NVIDIA B200; 68 measured rows. Details: Markdown - MoE routing finalization (
moe_finalize_routing): Unpermute, scale and sum local expert outputs into one row per original token. group: Routing and combine; backends:flashinfer_trtllm; precisions:bf16; GPUs: NVIDIA B200, NVIDIA H200; 2,137 measured rows. Details: Markdown - Fused MoE top-k routing (
moe_fused_topk): Select top-k experts from softmax router scores and renormalize their weights. group: Routing and combine; backends:torch,vllm_cuda; precisions:bf16; GPUs: NVIDIA H200; 42 measured rows. Details: Markdown - Sink-renormalized MoE gate select (
moe_sink_gate_select): Pick each token's top-k routed experts by sigmoid score plus bias, then renormalize their logits together with the sink experts' logits. group: Routing and combine; backends:vllm_triton; precisions:fp32; GPUs: NVIDIA B200; 152 measured rows. Details: Markdown - MoE expert sum (
moe_sum): Sum the routed expert output rows for each token. group: Routing and combine; backends:vllm_cuda; precisions:bf16; GPUs: NVIDIA H200; 68 measured rows. Details: Markdown - Softplus-sqrt MoE routing (
moe_topk_softplus_sqrt): Select experts with sqrt-softplus scores or a token hash table and produce scaled weights. group: Routing and combine; backends:vllm_cuda; precisions:fp32; GPUs: NVIDIA H200; 136 measured rows. Details: Markdown - MXFP4 Marlin MoE GEMM (
mxfp4_marlin_moe_gemm): Multiply routed BF16 activations by packed MXFP4 expert weights in one Marlin launch. group: Expert compute; backends:vllm_marlin; precisions:bf16; GPUs: NVIDIA H200; 313 measured rows. Details: Markdown - Block-scaled fused MoE (
nvfp4_fused_moe): Route block-quantized tokens through packed expert weights and produce their MoE outputs. group: Expert compute; backends:flashinfer_trtllm_sm100,flashinfer_trtllm_routed_sm100,flashinfer_trtllm_routed_sm100_fork,flashinfer_trtllm_sm100_deferred_finalize,flashinfer_trtllm_fp8_block_sm100,flashinfer_trtllm_sm100_mxfp4,flashinfer_trtllm_sm100_mxfp4_bf16; precisions:fp8_e4m3,mxfp4_e2m1,nvfp4_e2m1; GPUs: NVIDIA B200; 13,358 measured rows. Details: Markdown - vLLM fused MoE GEMM (
vllm_fused_moe): Gather selected token rows and multiply them by FP8 expert weights in one Triton launch. group: Expert compute; backends:vllm_triton; precisions:fp8_e4m3; GPUs: NVIDIA H200; 194 measured rows. Details: Markdown
Communication
- All-reduce (
all_reduce): Sum one buffer across the GPUs of a tensor-parallel group. backends:nccl,nvshmem; GPUs: NVIDIA B200, NVIDIA H200; 171 measured rows. Details: Markdown - All-reduce, FlashInfer kernels (
all_reduce_fusion): Sum a [num_tokens, hidden_dim] tensor across the GPUs of a tensor-parallel group with FlashInfer's all-reduce. backends:flashinfer_trtllm,flashinfer_mnnvl; precisions:bf16; GPUs: NVIDIA B200; 150 measured rows. Details: Markdown - All-reduce + residual add + RMSNorm (
all_reduce_residual_rms_norm): Sum a tensor across GPUs, add the residual and RMS-normalize the result in a fused call. backends:flashinfer_trtllm,flashinfer_mnnvl; precisions:bf16; GPUs: NVIDIA B200, NVIDIA H200; 143 measured rows. Details: Markdown - MoE all-to-all transfer (
moe_alltoall): Move token rows to expert GPUs or return expert outputs to their source GPUs. backends:flashinfer_mnnvl; GPUs: NVIDIA H200; 470 measured rows. Details: Markdown - MoE all-to-all preparation (
moe_alltoall_prepare): Build and exchange the row indices needed for expert-parallel dispatch. backends:flashinfer_mnnvl; GPUs: NVIDIA H200; 68 measured rows. Details: Markdown - MoE all-gather of hidden states and router logits (
moe_ep_all_gather): Gather hidden states and router logits from every data-parallel GPU. backends:vllm_pynccl; precisions:bf16,fp8_e4m3; GPUs: NVIDIA B200, NVIDIA H200; 62 measured rows. Details: Markdown - MoE all-gather of quantized activations and top-k routing (
moe_ep_quantized_all_gather): Gather NVFP4 or MXFP8 activations, scales and top-k routing from every data-parallel GPU. backends:vllm_pynccl; precisions:mxfp8_e4m3,nvfp4_e2m1; GPUs: NVIDIA B200; 130 measured rows. Details: Markdown - MoE output reduce-scatter (
moe_ep_reduce_scatter): Sum expert outputs across GPUs and return each GPU's token shard. backends:vllm_pynccl; precisions:bf16; GPUs: NVIDIA B200, NVIDIA H200; 127 measured rows. Details: Markdown - Inter-domain point-to-point transfer (
p2p_inter): Modeled time for one transfer between GPUs in different NVLink domains. backends:nccl,nvshmem; GPUs: NVIDIA H200; 38 measured rows. Details: Markdown - Intra-domain point-to-point transfer (
p2p_intra): Send one payload between two GPUs in the same NVLink domain. backends:nccl,nvshmem; GPUs: NVIDIA B200, NVIDIA H200; 57 measured rows. Details: Markdown - Reduce-scatter + short conv + all-gather + residual + RMSNorm (
rs_sconv_ag_residual_rms_norm): Sum sublayer partials across GPUs, run a causal short convolution on each GPU's hidden shard, gather it back, add the residual and RMS-normalize. backends:vllm_lamport_triton; precisions:bf16; GPUs: NVIDIA B200; 254 measured rows. Details: Markdown
Normalization
- Gated DeltaNet output RMSNorm (
gdn_gated_rms_norm): RMS-normalize each recurrent output row and multiply it by a SiLU gate. group: RMSNorm; backends:torch,vllm_triton; precisions:bf16; GPUs: NVIDIA B200, NVIDIA H200; 44 measured rows. Details: Markdown - Grouped RMSNorm (
grouped_rms_norm): RMS-normalize each contiguous group of a row separately. group: RMSNorm; backends:vllm_triton; precisions:bf16; GPUs: NVIDIA B200; 68 measured rows. Details: Markdown - HC combine (
hc_combine): Add a block's output into every residual stream with a learned gate. group: Hyper-connections; backends:vllm_triton; precisions:bf16; GPUs: NVIDIA B200; 68 measured rows. Details: Markdown - HC combine + per-stream norm (
hc_combine_norm): Add a block's output into every residual stream with a learned gate, then RMS-normalize each stream, in one launch. group: Hyper-connections; backends:vllm_triton; precisions:bf16; GPUs: NVIDIA B200; 68 measured rows. Details: Markdown - HC gate mix (
hc_gate_mix): Collapse the normalized residual streams into one block input with a per-feature sigmoid gate. group: Hyper-connections; backends:vllm_triton; precisions:bf16; GPUs: NVIDIA B200; 68 measured rows. Details: Markdown - mHC post-block, pre-block and RMSNorm (
mhc_fused_post_pre_rms_norm): Mix a block's output into the mHC residual streams, then compute the next block's mixing weights and normalized input, in one call. group: Hyper-connections; backends:vllm_tilelang,deepgemm_mega,deepgemm_mega_nonshifted; precisions:bf16; GPUs: NVIDIA B200, NVIDIA H200; 393 measured rows. Details: Markdown - mHC pre-block with RMSNorm (
mhc_pre_rms_norm): From a token's parallel residual streams, compute the mHC mixing weights and the RMS-normalized input of the next block. group: Hyper-connections; backends:vllm_tilelang; precisions:bf16; GPUs: NVIDIA B200, NVIDIA H200; 204 measured rows. Details: Markdown - Terminal mHC head (
mhc_terminal_head): Collapse the final mHC residual channels into one hidden state and RMS-normalize it for the LM head. group: Hyper-connections; backends:vllm_tilelang; precisions:bf16; GPUs: NVIDIA H200; 68 measured rows. Details: Markdown - Summed RMSNorm with fused outputs (
multi_add_rms_norm): Sum one to three residual-stream tensors, RMS-normalize, and write any of the sum, the normalized rows and their FP8 quantization in one kernel. group: RMSNorm; backends:vllm_inductor; precisions:bf16; GPUs: NVIDIA B200; 340 measured rows. Details: Markdown - Query and KV RMSNorm (
q_kv_rms_norm): RMS-normalize query-side and KV-side projections with separate weights. group: RMSNorm; backends:vllm_triton; precisions:bf16; GPUs: NVIDIA B200, NVIDIA H200; 136 measured rows. Details: Markdown - Residual add + RMSNorm (
residual_rms_norm): Add the residual stream, then normalize: the fused step between transformer blocks. group: RMSNorm; backends:torch,vllm_cuda; precisions:bf16; GPUs: NVIDIA B200, NVIDIA H200; 272 measured rows. Details: Markdown - RMSNorm (
rms_norm): Normalize each token's hidden vector by its root mean square and scale it by a weight. group: RMSNorm; backends:flashinfer,vllm_cuda,vllm_gemma_native; precisions:bf16; GPUs: NVIDIA B200, NVIDIA H200; 756 measured rows. Details: Markdown
Quantization
- FP8 block quantization (
fp8_block_quant): Quantize BF16 expert inputs to FP8 E4M3 in 128-element blocks. backends:flashinfer_trtllm; precisions:bf16; GPUs: NVIDIA H200; 748 measured rows. Details: Markdown - FP8 per-token-group quantization (
fp8_per_token_group_quant): Quantize BF16 activation rows to FP8 E4M3 with a scale per 128-element group. backends:vllm_cuda; precisions:bf16; GPUs: NVIDIA B200, NVIDIA H200; 1,224 measured rows. Details: Markdown - Inverse RoPE and FP8 quantization (
inv_rope_fp8_quant): Undo RoPE on attention outputs and quantize them in groups for the output projection. backends:vllm_triton; GPUs: NVIDIA H200; 57 measured rows. Details: Markdown - NVFP4 activation quantization (
nvfp4_quant): Quantize BF16 activation rows to packed NVFP4 values and E4M3 group scales. backends:vllm_cuda,flashinfer_cutedsl; precisions:bf16; GPUs: NVIDIA B200; 748 measured rows. Details: Markdown - Static FP8 quantization (
static_fp8_quant): Scale an activation by a fixed per-tensor scale and round it to FP8 E4M3. backends:vllm_inductor; precisions:bf16; GPUs: NVIDIA B200; 136 measured rows. Details: Markdown
Other
- Elementwise byte pass (
elementwise): Read a whole multiple of the output size, or nothing, and write the output. backends:triton,torch,torch_strided_copy,torch_cat,torch_vllm,torch_strided_copy_vllm,torch_cat_vllm; GPUs: NVIDIA B200, NVIDIA H200; 11,023 measured rows. Details: Markdown - Embedding row gather (
embedding_row_gather): Copy num_tokens × num_heads randomly addressed rows of a large table, kept in pinned host memory or HBM, into a device buffer. backends:vllm_triton,torch; precisions:bf16,fp8_e4m3; GPUs: NVIDIA B200; 78 measured rows. Details: Markdown - Engram n-gram embedding lookup (
engram_lookup): Gather hashed n-gram rows from a large FP8 table, dequantize them and write BF16 rows for the layers that consume them. backends:vllm_triton; precisions:fp8_e4m3; GPUs: NVIDIA B200; 146 measured rows. Details: Markdown - Indexed row copy (
indexed_row_copy): Gather, scatter or zero whole rows by index with eager PyTorch. backends:torch,torch_vllm; precisions:bf16,fp32; GPUs: NVIDIA B200; 165 measured rows. Details: Markdown - Logits top-k (
logits_topk): Select the largest top_k scores and their column indices from each score row. backends:torch,flashinfer; precisions:bf16; GPUs: NVIDIA B200; 68 measured rows. Details: Markdown - PLE similarity gate (
ple_gate): Gate a looked-up n-gram value by its normalized key's similarity to each residual stream, and RMS-normalize the result. backends:vllm_triton; precisions:bf16; GPUs: NVIDIA B200; 25 measured rows. Details: Markdown - PLE n-gram hash ids (
ple_ngram_ids): Hash each token's preceding n-grams into per-head embedding-table ids. backends:vllm_triton; GPUs: NVIDIA B200; 40 measured rows. Details: Markdown - PLE dilated short convolution (
ple_short_conv): Depthwise dilated causal convolution with SiLU over the PLE stream, added into the residual, with the per-request conv state carried across steps. backends:vllm_triton; precisions:bf16; GPUs: NVIDIA B200; 195 measured rows. Details: Markdown - Scaled SiLU (
scaled_silu): Apply SiLU to every element after dividing it by a constant. backends:vllm_triton; precisions:bf16; GPUs: NVIDIA B200; 68 measured rows. Details: Markdown - SiLU and mul (
silu_and_mul): Multiply SiLU of the gate half of each row by its up half. backends:vllm_inductor,vllm_cuda; precisions:bf16; GPUs: NVIDIA B200; 136 measured rows. Details: Markdown
This page as Markdown: https://servingstudio.cs.washington.edu/kernels.md. A guide to the whole site for AI agents: llms.txt.