# KDA chunked prefill (`kda_chunk_prefill`) | ServingStudio Kernel Library

Compute KDA prefill output and final recurrent states, with a decay gate per key channel, in one call.

KDA is a gated delta rule like Gated DeltaNet, but its decay gate has one value per key channel instead of one per head. In prefill vLLM makes one call: it copies the strided q, k and v views, L2-normalizes q and k, computes the gate from the raw projection, and writes every token's output and each sequence's final state. The vllm_triton backend is FLA's chunked Triton kernels (chunk_kda_with_fused_gate); flashkda is FlashKDA, vLLM's default on SM90, SM10x and SM12x. Decode requests scheduled in the same step join the call as one-token sequences, placed first; the remaining tokens form full max_sequence_length sequences plus one shorter remainder. Decode sequences start from random FP32 states, prefill sequences from zero. vllm_triton takes beta already passed through a sigmoid; flashkda takes the raw logits and applies the sigmoid inside the call.

Category: Attention; group: Gated DeltaNet. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=kda_chunk_prefill. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `decay = exp(−5·sigmoid(exp(A_log)·(raw_g + dt_bias))), per head and key channel`
- `q, k = L2Norm(q, k); q = q·head_dim⁻¹ᐟ²`
- `S′ = diag(decay)·S + k·(beta·(v − kᵀ(diag(decay)·S)))ᵀ; y = qᵀS′, token by token`
- `N = num_decode_sequences + ⌈(num_tokens − num_decode_sequences) / max_sequence_length⌉`
- `FLOPs = 2·num_tokens·num_heads·(3·head_dim² + 4·64·head_dim)`
- `bytes = 10·num_tokens·num_heads·head_dim + 4·num_tokens·num_heads + 8·N·num_heads·head_dim²`
- `TFLOPS = FLOPs / time`
- `GB/s = bytes / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. Five warm-up calls run first; every launch of the call is counted, the q, k and v copies included. vllm_triton's Triton autotuning runs once per worker, at 2048 tokens (one 2019-token prefill and 29 decodes); its keys omit the token count, so those configs serve every later shape; FlashKDA has no autotuning. Before timing, the shape cut to at most 8 decodes and 1024-token sequences is checked against a per-token PyTorch reference.

## Caveats

- FLA's chunk kernels take head_dim up to 256; FlashKDA takes only 128.
- Each one-token decode sequence occupies a whole chunk in the chunk-parallel kernels: 64 tokens in FLA, 16 in FlashKDA.
- FlashKDA splits each head's value dimension across two blocks only while 2·num_heads·sequences blocks fit one wave of SMs (148 on B200), so its time steps up where that stops: about 1.4x at 8 heads from 9 to 17 sequences of one 8192-token prefill.
- The timed calls reuse one cu_seqlens tensor, so FLA's cached chunk-index setup, which the first layer of a step pays, is not in the vllm_triton time.
- The gather and scatter of recurrent states run outside the call, and so does vllm_triton's beta sigmoid.
- FLOPs are logical counts at a 64-token chunk width; bytes count each input once and leave out the copies and intermediates.
- The flashinfer_* backends fuse the beta sigmoid into the call and need head_dim 128 and num_heads divisible by 8.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_tokens` | number | tokens | Tokens in the call, decode tokens included. |
| `max_sequence_length` | number | tokens | Tokens in the longest prefill sequence. |
| `num_decode_sequences` | number | requests | Decode requests in the same call, one token each. |
| `num_heads` | number | heads | Heads, each with its own query, key, value and state. |
| `head_dim` | number | elements | Features in each query, key and value head. |
| `dtype` | dtype |  | Element type of queries, keys, values, gate input and output. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_triton`: vLLM's chunk_kda_with_fused_gate: q, k and v copies, L2 norms, the fused gate and chunk cumsum, then FLA's chunked Triton kernels. dtypes `bf16`, any CUDA GPU. Source: https://github.com/vllm-project/vllm/blob/main/vllm/models/glm5next/nvidia/ops/third_party/kda/kernels.py
- `flashinfer_tirx`: FlashInfer recurrent_kda(backend='tirx'), the NVlabs KDA-for-KDA TIRx kernel: three q/k/v copies, a host-built work plan that uploads its work list and clears handoff flags (two small fills) on every call, then one persistent BT64 kernel that does the L2 norms, gate, beta sigmoid and recurrence, handing FP32 state between CTAs. dtypes `bf16`, SM targets `sm_100a`, `sm_103a`, GPUs B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip. Source: https://github.com/flashinfer-ai/flashinfer/blob/main/flashinfer/kda.py
- `flashinfer_cute_persistent`: FlashInfer recurrent_kda(backend='cute-dsl-persistent'), the CuTe DSL persistent prefill kernel: three q/k/v copies, then one kernel that does the L2 norms, gate, beta sigmoid and recurrence. Its work plan is cached per sequence layout and adds no launch. dtypes `bf16`, SM targets `sm_100a`, `sm_103a`, GPUs B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip. Source: https://github.com/flashinfer-ai/flashinfer/blob/main/flashinfer/kda.py
- `flashkda`: FlashKDA (vLLM's _flashkda_C), as vLLM's _flashkda_prefill calls it: q, k and v copies, then one fwd that transposes beta, runs a prepare kernel (L2 norms, gate, 16-token chunk factors) and a recurrence kernel. Only head_dim 128. Unlike vllm_triton, it takes raw beta logits and applies the sigmoid inside the call, so no separate beta sigmoid runs outside it. dtypes `bf16`, SM targets `sm_100f`, `sm_120f`, `sm_90a`, GPUs H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip, H800-SXM5-80GB, H20-SXM-96GB. Source: https://github.com/vllm-project/vllm/blob/main/vllm/models/glm5next/common/kda.py

## Measured rows

4,096 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `flashinfer_cute_persistent` | `bf16` | 1,024 |
| NVIDIA B200 | `flashkda` | `bf16` | 1,536 |
| NVIDIA B200 | `vllm_triton` | `bf16` | 1,536 |

## Used by these deployments

- [GLM-5.3-Flash NVFP4: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe`)
- [GLM-5.3-Flash: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe`)
- [GLM-5.3-Flash: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe`)
- [GLM-5.3-Flash: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/kda_chunk_prefill
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/kda_chunk_prefill/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/kda_chunk_prefill/configs

## Torch reference

What the kind computes, as Torch code (`profiling/runners/attention/kda_chunk_prefill_reference.py` in ServingStudio Sim):

```python
"""Naive per-token Torch oracle for ``kda_chunk_prefill``.

Recomputes what ``chunk_kda_with_fused_gate`` computes on the GLM-5.3-Flash
prefill path. It does not use the chunked algorithm or any FLA kernel:

1. l2-normalize q and k per head (fp32 math, ``eps=1e-6``). The result is
   rounded back to the input dtype because ``l2norm_fwd`` returns the input
   dtype and the chunk kernels read that rounded tensor.
2. Compute the per-channel safe gate in natural-log space:
   ``g = lower_bound * sigmoid(exp(A_log[h]) * (raw_g + dt_bias[h, :]))``.
3. Run the gated delta rule once per token, one sequence at a time, on an fp32
   state ``S[h, k, v]``. This is the recurrence in FLA's ``naive_recurrent_kda``:
   ``S *= exp(g_t)``, ``S += beta_t * k_t (v_t - k_t^T S)``,
   ``o_t = (scale * q_t)^T S``.

Production stores recurrent states as ``[N, H, V, K]``, the transpose of the
math layout, so the oracle takes and returns that layout.

Only torch is imported, so the unit tests run this module on CPU.
"""

from __future__ import annotations

from collections.abc import Sequence

import torch

L2NORM_EPS = 1e-6


def l2norm_like_fla(x: torch.Tensor, eps: float = L2NORM_EPS) -> torch.Tensor:
    """``x / sqrt(sum(x^2) + eps)`` over the last dim, rounded to ``x.dtype``."""
    xf = x.float()
    return (xf * torch.rsqrt(xf.square().sum(-1, keepdim=True) + eps)).to(x.dtype)


def kda_safe_gate(
    raw_g: torch.Tensor,
    a_log: torch.Tensor,
    dt_bias: torch.Tensor,
    lower_bound: float,
) -> torch.Tensor:
    """Per-channel log-decay ``[T, H, K]`` in ``(lower_bound, 0)``."""
    _, heads, head_dim = raw_g.shape
    decay_rate = a_log.reshape(-1).float().exp().view(1, heads, 1)
    bias = dt_bias.reshape(heads, head_dim).float()
    return lower_bound * torch.sigmoid(decay_rate * (raw_g.float() + bias))


def kda_chunk_prefill_reference(
    q: torch.Tensor,
    k: torch.Tensor,
    v: torch.Tensor,
    raw_g: torch.Tensor,
    beta: torch.Tensor,
    a_log: torch.Tensor,
    dt_bias: torch.Tensor,
    initial_state: torch.Tensor,
    boundaries: Sequence[int],
    *,
    lower_bound: float,
    scale: float | None = None,
) -> tuple[torch.Tensor, torch.Tensor]:
    """Return ``(o [T,H,V] in v.dtype, final_state [N,H,V,K] fp32)``.

    ``q``/``k``/``raw_g`` are ``[T,H,K]``, ``v`` is ``[T,H,V]``, ``beta`` is the
    already-sigmoided ``[T,H]``, ``initial_state`` is ``[N,H,V,K]`` and
    ``boundaries`` holds the N+1 cumulative sequence offsets.
    """
    head_dim = q.shape[-1]
    if scale is None:
        scale = head_dim**-0.5
    qn = l2norm_like_fla(q).float() * scale
    kn = l2norm_like_fla(k).float()
    vf = v.float()
    decay = kda_safe_gate(raw_g, a_log, dt_bias, lower_bound).exp()
    betaf = beta.float()

    output = torch.empty(vf.shape, dtype=torch.float32, device=v.device)
    final_states = []
    bounds = [int(b) for b in boundaries]
    for sequence, (start, end) in enumerate(zip(bounds[:-1], bounds[1:], strict=True)):
        state = initial_state[sequence].float().transpose(-1, -2).clone()  # [H, K, V]
        for t in range(start, end):
            state = state * decay[t].unsqueeze(-1)
            predicted = torch.einsum("hk,hkv->hv", kn[t], state)
            correction = betaf[t].unsqueeze(-1) * (vf[t] - predicted)
            state = state + kn[t].unsqueeze(-1) * correction.unsqueeze(-2)
            output[t] = torch.einsum("hk,hkv->hv", qn[t], state)
        final_states.append(state.transpose(-1, -2))
    return output.to(v.dtype), torch.stack(final_states)


def rmse_ratio(reference: torch.Tensor, candidate: torch.Tensor) -> float:
    """RMSE of the difference over RMS of the reference (the fork's KDA metric)."""
    ref = reference.detach().float()
    diff = ref - candidate.detach().float()
    return float(diff.square().mean().sqrt() / (ref.square().mean().sqrt() + 1e-8))
```

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/kda_chunk_prefill.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
