# QK RMSNorm + RoPE + gate split (`qk_rms_norm_rope_gate`) | ServingStudio Kernel Library

RMS-normalize every query and key head, rotate its leading rotary_dim elements, and split the query projection's output gate off, in one launch.

A gated attention projects each query head together with an output-gate head. vLLM's fused Triton kernel reads the query-and-gate and key slices of the QKV projection output in place, RMS-normalizes each head over head_dim, applies NeoX rotary embedding to the first rotary_dim elements, and writes contiguous query, key and gate tensors. One program handles one head of one token. Value heads are not touched.

Category: Attention; group: MHA / GQA. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=qk_rms_norm_rope_gate. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `q, k ← x / √(mean(x²) + ε) · weight per head; RoPE on [0, rotary_dim)`
- `GB/s = 2 · num_tokens · head_dim · (4·num_q_heads + 2·num_kv_heads) / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. Three warm-up calls and a comparison with an independent PyTorch reference precede capture; every launch of the call is counted.

## Caveats

- GB/s counts the query, gate and key heads read and written once; the cos/sin rows and the rotated halves' reload hit cache.
- The cos/sin table length and the position values do not change the time (measured: under 1% between 4K and 1M rows, sequential or random positions).

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_tokens` | number | tokens | Tokens whose query and key heads are prepared. |
| `num_q_heads` | number | heads | Query heads per token (each with a gate head). |
| `num_kv_heads` | number | heads | Key heads per token. |
| `head_dim` | number | elements | Width of each head. |
| `rotary_dim` | number | elements | Leading elements of each head that are rotated. |
| `mrope` | label |  | Whether positions are multimodal (three rows: temporal, height, width) rather than one row; each rotated pair then loads its section's position. |
| `dtype` | dtype |  | Element type of the projections and outputs. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_triton`: vLLM's fused_qk_rmsnorm_rope_gate Triton kernel. dtypes `bf16`, compute capability 8.0 or newer, GPUs A100-SXM4-40GB, A100-SXM4-80GB, A100-PCIe-40GB, A100-PCIe-80GB, H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip, H800-SXM5-80GB, H20-SXM-96GB, A40, L40, L40S, L4. Source: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/fused_qk_norm_rope.py

## Measured rows

136 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `vllm_triton` | `bf16` | 136 |

## Used by these deployments

- [Qwen3.8 27B NVFP4: vLLM, one GPU](https://servingstudio.cs.washington.edu/models/Qwen3.8-27B-NVFP4/qwen35_vllm_nvfp4_gdn_dense.md) (`Qwen3.8-27B-NVFP4/qwen35_vllm_nvfp4_gdn_dense`)
- [Qwen3.8 Flash Next NVFP4: vLLM, one GPU, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative.md) (`Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/qk_rms_norm_rope_gate
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/qk_rms_norm_rope_gate/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/qk_rms_norm_rope_gate/configs

## Torch reference

What the kind computes, as Torch code (`profiling/runners/attention/qk_rms_norm_rope_gate_reference.py` in ServingStudio Sim):

```python
"""Independent value reference for the fused QK RMSNorm + partial NeoX RoPE + gate split."""

from typing import Any


def qk_rms_norm_rope_gate_reference(
    torch: Any,
    q_gate: Any,
    k: Any,
    q_weight: Any,
    k_weight: Any,
    cos_sin_cache: Any,
    positions: Any,
    eps: float,
    head_dim: int,
    rotary_dim: int,
) -> tuple[Any, Any, Any]:
    """Return contiguous (q, k, gate); ``positions`` is one row per token."""
    num_tokens = q_gate.shape[0]
    half = rotary_dim // 2
    cos = cos_sin_cache[positions, :half].float()[:, None, :]
    sin = cos_sin_cache[positions, half:rotary_dim].float()[:, None, :]

    def norm_rope(heads: Any, weight: Any) -> Any:
        values = heads.float()
        normed = values * torch.rsqrt(values.square().mean(-1, keepdim=True) + eps)
        # The kernel stores the normalized head in the input dtype before rotating it.
        normed = (normed * weight.float()).to(heads.dtype).float()
        first, second = normed[..., :half], normed[..., half:rotary_dim]
        rotated = torch.cat(
            (first * cos - second * sin, second * cos + first * sin, normed[..., rotary_dim:]),
            dim=-1,
        )
        return rotated.to(heads.dtype).reshape(num_tokens, -1)

    q_and_gate = q_gate.reshape(num_tokens, -1, 2 * head_dim)
    q_out = norm_rope(q_and_gate[..., :head_dim], q_weight)
    k_out = norm_rope(k.reshape(num_tokens, -1, head_dim), k_weight)
    gate_out = q_and_gate[..., head_dim:].reshape(num_tokens, -1)
    return q_out, k_out, gate_out


__all__ = ["qk_rms_norm_rope_gate_reference"]
```

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/qk_rms_norm_rope_gate.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
