# Query padding, KV RoPE and MXFP8 window insert (`q_pad_kv_rope_mxfp8_insert`) | ServingStudio Kernel Library

Copy the live query heads into a zero-padded buffer while rotating, quantizing to MXFP8 and inserting KV rows into a sliding-window cache.

Between the attention projections and a fused attention kernel that applies query RoPE itself, one CUDA call prepares both inputs. Query side: the num_heads live heads are copied and zero-filled up to padded_heads in the attention kernel's 16-element chunk-interleaved layout, with no norm and no rotation. KV side: each 512-element row gets GPT-J RoPE on its last 64 elements and MXFP8 quantization (FP8 E4M3, one UE8M0 scale per 32 elements), and the first num_insert_tokens rows are written to the paged cache, each page holding its 512-byte data rows and then its 16-byte scale rows. padded_heads = 0 is the KV-only launch.

Category: Attention; group: Compressed sparse MLA. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=q_pad_kv_rope_mxfp8_insert. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `q_out[:, :num_heads] = q; q_out[:, num_heads:padded_heads] = 0`
- `cache[slot] = MXFP8(RoPE(kv)), 528 bytes per inserted row`
- `GB/s = [(2 · 512 · num_tokens · (num_heads + padded_heads) if padded_heads > 0) + num_insert_tokens · (2 · 512 + 16 + 4 · 64 + 528)] / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. The call is one launch: a one-CTA-per-token grid when num_tokens ≥ 1024 and padded_heads \> 0, a warp per (token, slot) otherwise. Before timing, the padded query must match bit for bit and every inserted record must decode to the rotated KV row within FP8 tolerance.

## Caveats

- The runner takes head_dim 512, rope_dim 64, 32-token pages, bf16 inputs and an MXFP8 cache (528 B per token), with 8, 16, 32, 64 or 128 live heads padded to 64 or 128 and at most 65,536 tokens.
- Positions and insert slots are random and distinct, as for decode tokens of different requests.
- TFLOPS is not computed. GB/s counts logical query, KV, position, RoPE-table and record bytes.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_tokens` | number | tokens | Query and KV rows the call reads. |
| `num_insert_tokens` | number | tokens | Leading KV rows inserted into the sliding-window cache. |
| `num_heads` | number | heads | Live query heads per token on this GPU. |
| `padded_heads` | number | heads | Query heads in the zero-padded output; 0 when no padding is written. |
| `block_size` | number | tokens | Tokens per sliding-window cache page. |
| `input_dtype` | dtype |  | Element type of the query and KV inputs. |
| `swa_cache_format` | label |  | Record format of the sliding-window KV cache. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_cuda`: vLLM's fused_deepseek_v4_qnorm_rope_kv_rope_quant_insert CUDA op with query norm and query RoPE off, chunk-interleaved query and MXFP8 KV records. dtypes `bf16`, KV dtypes `fp8_e4m3`, compute capability 8.0 or newer, GPUs A100-SXM4-40GB, A100-SXM4-80GB, A100-PCIe-40GB, A100-PCIe-80GB, H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip, H800-SXM5-80GB, H20-SXM-96GB, A40, L40, L40S, L4. Source: https://github.com/vllm-project/vllm/blob/main/csrc/libtorch_stable/fused_qnorm_rope_kv_insert_kernel.cu

## Measured rows

208 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `vllm_cuda` | `bf16` | 208 |

## Used by these deployments

- [DeepSeek V4.1 Flash 1017: vLLM, DP attention, EP MoE, DSpark speculative decoding](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dp_attn_ep_moe_dspark.md) (`DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dp_attn_ep_moe_dspark`)
- [DeepSeek V4.1 Flash 1017: vLLM, TP attention, EP MoE, DSpark speculative decoding](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dspark.md) (`DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dspark`)
- [DeepSeek V4.1 Flash: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash/deepseek_v41_vllm.md) (`DeepSeek-V4.1-Flash/deepseek_v41_vllm`)
- [DeepSeek V4.1 Flash: vLLM, TP attention, EP MoE, side streams serialized](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash/deepseek_v41_vllm_serial_streams.md) (`DeepSeek-V4.1-Flash/deepseek_v41_vllm_serial_streams`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/q_pad_kv_rope_mxfp8_insert
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/q_pad_kv_rope_mxfp8_insert/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/q_pad_kv_rope_mxfp8_insert/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/q_pad_kv_rope_mxfp8_insert.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
