# KDA recurrent decode (`kda_recurrent_decode`) | ServingStudio Kernel Library

Update each request's KDA state, gated per key channel, and read one output token.

The core of KDA decode. KDA is a gated delta rule like Gated DeltaNet, but its decay gate has one value per key channel instead of one per head. vLLM's fused_recurrent_kda call first copies q, k, v and beta out of the merged input projection, as four copy launches. One Triton kernel then computes the gate from the raw projection, applies the beta sigmoid and the q and k L2 norms, updates each request's FP32 state in place in a slot-indexed state pool, and writes the output. Activations are BF16.

Category: Attention; group: Gated DeltaNet. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=kda_recurrent_decode. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `decay = exp(−5·sigmoid(exp(A_log)·(raw_g + dt_bias))), per head and key channel`
- `q, k = L2Norm(q, k); q = q·head_dim⁻¹ᐟ²; beta = sigmoid(b)`
- `S′ = diag(decay)·S + k·(beta·(v − kᵀ(diag(decay)·S)))ᵀ; y = qᵀS′`
- `FLOPs = 7·batch_size·num_heads·head_dim²`
- `bytes = 10·batch_size·num_heads·head_dim + 2·batch_size·num_heads + 8·batch_size·num_heads·head_dim²`
- `TFLOPS = FLOPs / time`
- `GB/s = bytes / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. Five warm-up calls run first; the four copies and the recurrent kernel are all counted. Every call advances the states in place, as vLLM does. Before timing, one call is checked against a single-step PyTorch reference and the states are restored.

## Caveats

- The kernel takes any head_dim.
- Only a plain decode step is measured, one token per request and no speculative decoding.
- The short causal convolution before the call is a separate kind, gdn_causal_conv_decode.
- GB/s counts each input once and leaves out the copies' extra reads and writes.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `batch_size` | number | requests | Requests decoded together, one token each. |
| `num_heads` | number | heads | Heads, each with its own query, key, value and state. |
| `head_dim` | number | elements | Features in each query, key and value head. |
| `dtype` | dtype |  | Element type of the activations, gate input and output. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_triton`: vLLM's fused_recurrent_kda: four input copies, then one Triton kernel that gates, normalizes and updates the indexed FP32 states. dtypes `bf16`, any CUDA GPU. Source: https://github.com/vllm-project/vllm/blob/main/vllm/models/glm5next/nvidia/ops/third_party/kda/kernels.py

## Measured rows

36 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `vllm_triton` | `bf16` | 36 |

## Used by these deployments

- [GLM-5.3-Flash NVFP4: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe`)
- [GLM-5.3-Flash: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe`)
- [GLM-5.3-Flash: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe`)
- [GLM-5.3-Flash: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/kda_recurrent_decode
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/kda_recurrent_decode/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/kda_recurrent_decode/configs

## Torch reference

What the kind computes, as Torch code (`profiling/runners/attention/kda_recurrent_decode_reference.py` in ServingStudio Sim):

```python
"""Naive single-step Torch oracle for ``kda_recurrent_decode``.

Recomputes one decode token per sequence the way ``fused_recurrent_kda``
computes it on the GLM-5.3-Flash plain-decode path. It uses no FLA kernel:

1. l2-normalize q and k per head in fp32 (``eps=1e-6``). The recurrent kernel
   normalizes in registers, so unlike the chunk oracle nothing is rounded back
   to bf16.
2. Compute the per-channel safe gate
   ``g = lower_bound * sigmoid(exp(A_log[h]) * (raw_g + dt_bias[h, :]))`` and
   ``beta = sigmoid(raw_beta)`` in fp32.
3. Apply one gated-delta-rule step to each sequence's fp32 state ``S[h, k, v]``:
   ``S *= exp(g)``, ``S += beta * k (v - k^T S)``, ``o = (scale * q)^T S``.

States are read from and written to the ``[slots, H, V, K]`` pool through
``state_indices``, the production layout. Only torch is imported, so the unit
tests run this module on CPU.
"""

from __future__ import annotations

import torch

from profiling.runners.attention.kda_chunk_prefill_reference import kda_safe_gate

L2NORM_EPS = 1e-6


def kda_recurrent_decode_reference(
    q: torch.Tensor,
    k: torch.Tensor,
    v: torch.Tensor,
    raw_g: torch.Tensor,
    raw_beta: torch.Tensor,
    a_log: torch.Tensor,
    dt_bias: torch.Tensor,
    state_pool: torch.Tensor,
    state_indices: torch.Tensor,
    *,
    lower_bound: float,
    scale: float | None = None,
) -> tuple[torch.Tensor, torch.Tensor]:
    """Return ``(o [B,H,V] in v.dtype, updated copy of state_pool)``.

    ``q``/``k``/``raw_g`` are ``[B,H,K]``, ``v`` is ``[B,H,V]``, ``raw_beta``
    is the un-sigmoided ``[B,H]``, ``state_pool`` is ``[slots,H,V,K]`` and
    ``state_indices`` holds each sequence's slot. The input pool is not mutated.
    """
    head_dim = q.shape[-1]
    if scale is None:
        scale = head_dim**-0.5
    qf, kf = q.float(), k.float()
    qn = qf * torch.rsqrt(qf.square().sum(-1, keepdim=True) + L2NORM_EPS) * scale
    kn = kf * torch.rsqrt(kf.square().sum(-1, keepdim=True) + L2NORM_EPS)
    decay = kda_safe_gate(raw_g, a_log, dt_bias, lower_bound).exp()  # [B, H, K]
    beta = raw_beta.float().sigmoid()

    slots = state_indices.long()
    state = state_pool[slots].float().transpose(-1, -2)  # [B, H, K, V]
    state = state * decay.unsqueeze(-1)
    predicted = torch.einsum("bhk,bhkv->bhv", kn, state)
    correction = beta.unsqueeze(-1) * (v.float() - predicted)
    state = state + kn.unsqueeze(-1) * correction.unsqueeze(-2)
    output = torch.einsum("bhk,bhkv->bhv", qn, state)

    updated = state_pool.clone()
    updated[slots] = state.transpose(-1, -2).to(state_pool.dtype)
    return output.to(v.dtype), updated
```

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/kda_recurrent_decode.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
