# Short-conv QKVR prep (`sconv_qkvr_prep`) | ServingStudio Kernel Library

Short-conv K and V, RMS-normalize q and k, scale by tau, insert K/V into the paged cache and project the relative-position logits, in one call.

Between the qkvr projection and relative attention, one call turns each token's q | k | v | r row into the attention inputs. K and V get a depthwise causal short convolution with residual, reading earlier tokens' taps from a paged conv-state cache and writing the raw K/V back to it. q and the convolved k are RMS-normalized per head; on global layers q and the rel logits are multiplied by the log-length scale tau. The normalized k and convolved v go into the paged attention cache, and r is projected to rel_extent relative-position logits per head. Below 128 tokens this is one kernel; from 128 tokens the KV work runs on a second stream in parallel with the q and rel-projection kernels.

Category: Attention; group: MHA / GQA. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=sconv_qkvr_prep. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `k' = bf16(k + Σ_i w_k[i] · k[p − W + 1 + i]), v' likewise`
- `q̂ = RMSNorm(q) · tau, k̂ = RMSNorm(k'), rel = (r @ W_rel) · tau`
- `cache[slot] = (k̂, v'); conv_cache[conv_slot] = (k, v)`
- `GB/s = 2 · [num_tokens · (q_w + 2·kv_w + H_q·d_rel + q_w + H_q·rel_extent + 4·kv_w) + d_rel·rel_extent] / time, q_w = H_q·head_dim, kv_w = H_kv·head_dim`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. One call is captured into a CUDA graph, as production runs it, and each graph replay is one launch, priced by the interval union of its kernels on both streams (the overlap counts once), not summed. Before timing, q, rel logits, the attention-cache K/V and the conv-cache writes must match an independent PyTorch reference, with every other cache slot untouched.

## Caveats

- Every token is a decode token of its own request, so all W − 1 conv taps come from the conv-state cache; prefill-heavy batches of the same size run 5-20% faster on B200.
- tau values do not change the work; the runner draws them in \[1, 1.3).
- head_dim 128, d_rel 16 and bf16 only, as the production call asserts.
- Rows were measured on one conv-state layout that the args do not carry: a per-head packed width of 2048, which is 2·head_dim + 2·hidden_size / (the model's KV heads) padded to a power of two, measured with 512 hidden columns per KV head. The width sets only the cache stride.
- TFLOPS is not computed. GB/s counts the qkvr read, q and rel-logit writes and both caches' K/V writes, not the cached conv taps.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_tokens` | number | tokens | Rows of the qkvr input in this call. |
| `num_q_heads` | number | heads | Query heads on this GPU. |
| `num_kv_heads` | number | heads | KV heads on this GPU. |
| `head_dim` | number | elements | Width of one q, k or v head. |
| `d_rel` | number | elements | Width of one head's relative-position branch r. |
| `rel_extent` | number | positions | Relative distances the rel logits cover (columns of W_rel). |
| `apply_log_scaling` | label |  | Whether q and the rel logits are multiplied by tau (global layers). |
| `sconv_kernel_size` | number | tokens | Taps of the K/V short convolution. |
| `conv_block_size` | number | tokens | Tokens per conv-state cache block. |
| `page_size` | number | tokens | Tokens per attention KV cache page. |
| `dtype` | dtype |  | Element type of qkvr, the weights, the outputs and both caches. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_triton`: The alignment fork's fused_qkvr_prep Triton call: one kernel below 128 tokens, a two-stream KV ∥ q + rel-projection fork above. dtypes `bf16`, KV dtypes `bf16`, compute capability 8.0 or newer, GPUs A100-SXM4-40GB, A100-SXM4-80GB, A100-PCIe-40GB, A100-PCIe-80GB, H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip, H800-SXM5-80GB, H20-SXM-96GB, A40, L40, L40S, L4.

## Measured rows

147 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `vllm_triton` | `bf16` | 147 |

## Used by these deployments

- [Inkling NVFP4: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Inkling-NVFP4/inkling_vllm_nvfp4_rel_attn_moe.md) (`Inkling-NVFP4/inkling_vllm_nvfp4_rel_attn_moe`)
- [Inkling Small NVFP4: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Inkling-Small-NVFP4/inkling_vllm_nvfp4_rel_attn_moe.md) (`Inkling-Small-NVFP4/inkling_vllm_nvfp4_rel_attn_moe`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/sconv_qkvr_prep
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/sconv_qkvr_prep/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/sconv_qkvr_prep/configs

## Torch reference

What the kind computes, as Torch code (`profiling/runners/attention/sconv_qkvr_prep_reference.py` in ServingStudio Sim):

```python
"""Independent Torch reference for the fused short-conv QKVR prep.

Written from the layer math in the alignment fork
(``vllm/models/inkling/nvidia/attention.py``: ``InklingShortConv`` on K and V,
``InklingRMSNorm`` on q and k, the tau multiply, ``RelLogitsProj``)
and not from ``ops/qkvr_prep.py``. It shares no code with the Triton kernels:
the tap source is decided by absolute position (a tap at or after the
request's first new position comes from this forward's ``qkvr``, an earlier one
from the conv-state cache), where the kernels compare row indices.

One call per layer, per token ``t`` of request ``r`` at position ``p``:

- ``k' = bf16(k_t + sum_{i<W} w_k[:, i] * k(p - W + 1 + i))`` and the same for
  ``v``, with zero taps before position 0 (depthwise causal conv + residual);
- ``k_hat = bf16(rmsnorm(k') * k_norm_w)`` and
  ``q_hat = bf16(rmsnorm(q) * q_norm_w)``, then on global layers
  ``q_hat = bf16(q_hat * tau_t)``;
- ``rel[t, h, :] = bf16(r[t, h, :] @ W_rel)`` and on global layers
  ``bf16(rel * tau_t)``;
- the raw ``k_t, v_t`` go to the conv-state cache at the token's conv slot and
  ``k_hat, v'`` go to the paged attention cache at its attention slot.
"""

from __future__ import annotations

from dataclasses import dataclass
from typing import Any


@dataclass(frozen=True)
class PrepShape:
    num_q_heads: int
    num_kv_heads: int
    head_dim: int
    d_rel: int
    rel_extent: int
    sconv_kernel_size: int

    @property
    def q_width(self) -> int:
        return self.num_q_heads * self.head_dim

    @property
    def kv_width(self) -> int:
        return self.num_kv_heads * self.head_dim

    @property
    def row_width(self) -> int:
        """Elements of one ``qkvr`` row: q | k | v | r."""
        return self.q_width + 2 * self.kv_width + self.num_q_heads * self.d_rel


def _rmsnorm(torch: Any, x: Any, weight: Any, eps: float) -> Any:
    return x * torch.rsqrt(x.pow(2).mean(dim=-1, keepdim=True) + eps) * weight.float()


def reference_conv(
    torch: Any,
    shape: PrepShape,
    qkvr: Any,
    k_weight: Any,
    v_weight: Any,
    positions: Any,
    first_new_position: Any,
    first_row: Any,
    cached_tap: Any,
) -> tuple[Any, Any]:
    """K and V after the short conv and residual, bf16, shape ``[T, kv_width]``.

    ``first_new_position[t]`` / ``first_row[t]`` are the position and row of the
    first token its request contributes to this forward. ``cached_tap(pos, t)``
    returns the pre-forward conv-cache ``(k, v)`` rows at absolute positions
    ``pos`` of token ``t``'s request (only called where ``pos`` is cached).
    """

    q_w, kv_w, width = shape.q_width, shape.kv_width, shape.sconv_kernel_size
    k = qkvr[:, q_w : q_w + kv_w].float()
    v = qkvr[:, q_w + kv_w : q_w + 2 * kv_w].float()
    acc_k = torch.zeros_like(k)
    acc_v = torch.zeros_like(v)
    rows = torch.arange(qkvr.shape[0], device=qkvr.device)
    for tap in range(width):
        source = positions - (width - 1) + tap
        exists = source >= 0
        in_forward = exists & (source >= first_new_position)
        cached = exists & ~in_forward
        src_k = torch.zeros_like(k)
        src_v = torch.zeros_like(v)
        if bool(in_forward.any()):
            src_row = first_row + (source - first_new_position)
            idx = rows[in_forward]
            src_k[idx] = k[src_row[in_forward]]
            src_v[idx] = v[src_row[in_forward]]
        if bool(cached.any()):
            idx = rows[cached]
            ck, cv = cached_tap(source[cached], idx)
            src_k[idx] = ck.float()
            src_v[idx] = cv.float()
        acc_k += src_k * k_weight[:, tap].float()
        acc_v += src_v * v_weight[:, tap].float()
    return (acc_k + k).to(torch.bfloat16), (acc_v + v).to(torch.bfloat16)


def reference_prep(
    torch: Any,
    shape: PrepShape,
    qkvr: Any,
    k_conv: Any,
    v_conv: Any,
    q_norm_weight: Any,
    k_norm_weight: Any,
    rel_proj: Any,
    tau: Any | None,
    eps: float,
) -> dict[str, Any]:
    """q, rel logits and the attention-cache K/V rows for every token."""

    tokens = qkvr.shape[0]
    hq, hkv, hd = shape.num_q_heads, shape.num_kv_heads, shape.head_dim
    q = qkvr[:, : shape.q_width].float().view(tokens, hq, hd)
    q_hat = _rmsnorm(torch, q, q_norm_weight, eps).to(torch.bfloat16)
    if tau is not None:
        q_hat = (q_hat.float() * tau.float()[:, None, None]).to(torch.bfloat16)

    k_hat = _rmsnorm(torch, k_conv.float().view(tokens, hkv, hd), k_norm_weight, eps).to(
        torch.bfloat16
    )

    r_off = shape.q_width + 2 * shape.kv_width
    r = qkvr[:, r_off : r_off + hq * shape.d_rel].float().view(tokens, hq, shape.d_rel)
    rel = torch.einsum("thd,de->the", r, rel_proj.float()).to(torch.bfloat16)
    if tau is not None:
        rel = (rel.float() * tau.float()[:, None, None]).to(torch.bfloat16)
    return {
        "q": q_hat.reshape(tokens, shape.q_width),
        "rel": rel,
        "k_cache_rows": k_hat,
        "v_cache_rows": v_conv.view(tokens, hkv, hd),
    }


def compare(torch: Any, got: Any, want: Any) -> dict[str, float]:
    """Max absolute error and max error relative to the reference's max magnitude."""
    g = got.float()
    w = want.float()
    diff = (g - w).abs()
    scale = float(w.abs().max().item()) or 1.0
    return {
        "max_abs": float(diff.max().item()) if diff.numel() else 0.0,
        "max_rel": (float(diff.max().item()) / scale) if diff.numel() else 0.0,
        "mismatch_fraction": float((diff > 0).float().mean().item()) if diff.numel() else 0.0,
    }


__all__ = [
    "PrepShape",
    "compare",
    "reference_conv",
    "reference_prep",
]
```

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/sconv_qkvr_prep.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
