# Recurrent decode (`gdn_recurrent_decode`) | ServingStudio Kernel Library

Update the gated delta-rule state and read one output token per request.

The core of Gated DeltaNet decode. Query and key heads are L2-normalized and repeated to match the value heads. For each value head, a decay gate shrinks the matrix state, a delta-rule update writes the new key-value pair with strength beta, and the query reads the updated state. Each request has its own FP32 state; activations are BF16.

Category: Attention; group: Gated DeltaNet. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=gdn_recurrent_decode. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `q, k = L2Norm(q, k); q = q·key_head_dim⁻¹ᐟ²; decay = exp(−exp(A_log)·softplus(a + dt_bias))`
- `beta = sigmoid(b) in FP32`
- `S′ = decay·S + k·(beta·(v − kᵀ(decay·S)))ᵀ; y = qᵀS′`
- `T = num_tokens (batch_size when absent); E₁ = num_value_heads·key_head_dim·value_head_dim`
- `FLOPs = 2·T·num_qk_heads·(3·key_head_dim + 1) + T·num_value_heads·key_head_dim + 4·T·num_value_heads + num_value_heads + 7·T·E₁`
- `bytes = (2·T·num_qk_heads·key_head_dim + 2·T·num_value_heads·value_head_dim + 2·T·num_value_heads)·bytes(dtype) + (2·num_value_heads + (batch_size + T)·E₁)·bytes(state_dtype)`
- `TFLOPS = FLOPs / time`
- `GB/s = bytes / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. torch counts every launch of its recurrent reference; vllm_triton counts only fused_recurrent_gated_delta_rule_packed_decode_kernel and vllm_triton_sigmoid_gating only fused_sigmoid_gating_delta_rule_update_kernel. Input packing and the vLLM output check run before timing.

## Caveats

- num_tokens \> batch_size is a speculative verify step, priced by vllm_triton_sigmoid_gating only (the call vLLM makes for the verify requests of a step that also carries a prefill): each request starts from the checkpoint of its last accepted token and writes one checkpoint per token (gdn_spec_decode_reference).
- torch and vllm_triton_sigmoid_gating take separate Q, K and V tensors; vllm_triton takes packed QKV. Both vLLM backends index into a state pool.
- vLLM uses vllm_triton for decode-only steps and vllm_triton_sigmoid_gating for the decode requests of a step that also carries a prefill.
- All backends update the states in place on every timed call.
- FLOPs and bytes count the operation itself, not the torch backend's intermediates.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `batch_size` | number | requests | Requests decoded together, one token each. |
| `num_qk_heads` | number | heads | Query and key heads per token. |
| `num_value_heads` | number | heads | Value heads and recurrent states per token. |
| `key_head_dim` | number | elements | Features in each query or key head. |
| `value_head_dim` | number | elements | Features in each value head. |
| `dtype` | dtype |  | Element type of activations and output. |
| `state_dtype` | dtype |  | Element type of the recurrent state and gate parameters. |
| `num_tokens` | label | tokens | Tokens across the batch, an equal number per request: a speculative (MTP verify) step's accepted token plus drafts. Absent when each request decodes one token. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `torch`: The PyTorch reference, with separate normalization, gate and state-update launches. dtypes `bf16`, any CUDA GPU.
- `vllm_triton`: vLLM's fused_recurrent_gated_delta_rule_packed_decode Triton call uses packed QKV and indexed FP32 state. dtypes `bf16`, any CUDA GPU. Source: https://github.com/vllm-project/vllm/blob/main/vllm/third_party/flash_linear_attention/ops/fused_recurrent.py
- `vllm_triton_sigmoid_gating`: vLLM's fused_sigmoid_gating_delta_rule_update Triton call, which updates the decode requests of a step that also carries a prefill: separate Q/K/V, one-token varlen batch, indexed FP32 state. dtypes `bf16`, any CUDA GPU. Source: https://github.com/vllm-project/vllm/blob/main/vllm/third_party/flash_linear_attention/ops/fused_sigmoid_gating.py

## Measured rows

75 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `vllm_triton` | `bf16` | 9 |
| NVIDIA B200 | `vllm_triton_sigmoid_gating` | `bf16` | 48 |
| NVIDIA H200 | `vllm_triton` | `bf16` | 18 |

## Used by these deployments

- [Qwen3.6 35B-A3B FP8: Single GPU](https://servingstudio.cs.washington.edu/models/Qwen3.6-35B-A3B-FP8/qwen36_local.md) (`Qwen3.6-35B-A3B-FP8/qwen36_local`)
- [Qwen3.8 27B NVFP4: vLLM, one GPU](https://servingstudio.cs.washington.edu/models/Qwen3.8-27B-NVFP4/qwen35_vllm_nvfp4_gdn_dense.md) (`Qwen3.8-27B-NVFP4/qwen35_vllm_nvfp4_gdn_dense`)
- [Qwen3.8 Flash Next NVFP4: vLLM, one GPU, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative.md) (`Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/gdn_recurrent_decode
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/gdn_recurrent_decode/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/gdn_recurrent_decode/configs

## Torch reference

What the kind computes, as Torch code (`profiling/runners/attention/gdn_recurrent_decode_reference.py` in ServingStudio Sim):

```python
"""Torch semantics for Qwen3.5/3.6 Gated DeltaNet recurrent decode."""

from __future__ import annotations

import torch
import torch.nn.functional as F

__all__ = ["gdn_recurrent_decode_reference"]

_L2_NORM_EPS = 1e-6


def gdn_recurrent_decode_reference(
    query: torch.Tensor,
    key: torch.Tensor,
    value: torch.Tensor,
    a: torch.Tensor,
    b: torch.Tensor,
    A_log: torch.Tensor,
    dt_bias: torch.Tensor,
    state: torch.Tensor,
) -> tuple[torch.Tensor, torch.Tensor]:
    """Run one non-speculative Gated DeltaNet decode step.

    Inputs are semantic tensors, not vLLM's packed QKV or slot-indexed state:
    Q/K are ``[B, Hq, Dk]``, V is ``[B, Hv, Dv]``, raw ``a``/``b`` are
    ``[B, Hv]``, learned ``A_log``/``dt_bias`` are ``[Hv]``, and state is
    ``[B, Hv, Dk, Dv]``. Q/K/V/a/b and the returned core output are BF16;
    parameters and recurrent state are FP32.

    For ``Qwen/Qwen3.6-35B-A3B-FP8``, ``Hq=16``, ``Hv=32``, and
    ``Dk=Dv=128``, so state is ``[B, 32, 128, 128]``. The implementation stays
    dimension-parameterized for smaller correctness tests and compatible GDNs.

    Q/K heads are repeated consecutively to value heads, matching Qwen's
    ``repeat_interleave(Hv // Hq)``. The recurrent state is updated in place and
    returned by identity, as in vLLM's packed decode kernel. All validation is
    completed before mutation.

    This first reference intentionally excludes multi-token/speculative decode,
    interleaved or packed QKV layouts, non-FP32 state/parameters, non-BF16
    activations, and head layouts where ``Hv`` is not divisible by ``Hq``.

    Semantics follow Transformers 5.13 ``torch_recurrent_gated_delta_rule`` and
    vLLM's ``fused_recurrent_gated_delta_rule_packed_decode_kernel`` and
    ``fused_sigmoid_gating_delta_rule_update_kernel``: both compute the gates
    and beta in FP32 and apply the recurrence in FP32.
    """
    _validate(query, key, value, a, b, A_log, dt_bias, state)

    num_value_heads = value.shape[1]
    repeats = num_value_heads // query.shape[1]

    query_fp32 = query.float()
    key_fp32 = key.float()
    query_fp32 = query_fp32 / torch.sqrt(
        torch.sum(query_fp32 * query_fp32, dim=-1, keepdim=True) + _L2_NORM_EPS
    )
    key_fp32 = key_fp32 / torch.sqrt(
        torch.sum(key_fp32 * key_fp32, dim=-1, keepdim=True) + _L2_NORM_EPS
    )
    query_fp32 = query_fp32.repeat_interleave(repeats, dim=1)
    key_fp32 = key_fp32.repeat_interleave(repeats, dim=1)
    query_fp32 = query_fp32 * (query.shape[-1] ** -0.5)

    decay_log = -torch.exp(A_log) * F.softplus(a.float() + dt_bias)
    decay = torch.exp(decay_log).unsqueeze(-1).unsqueeze(-1)
    beta = torch.sigmoid(b.float()).unsqueeze(-1)

    decayed_state = state * decay
    memory = torch.sum(decayed_state * key_fp32.unsqueeze(-1), dim=-2)
    delta = (value.float() - memory) * beta
    updated_state = decayed_state + key_fp32.unsqueeze(-1) * delta.unsqueeze(-2)
    output = torch.sum(updated_state * query_fp32.unsqueeze(-1), dim=-2)

    state.copy_(updated_state)
    return output.to(value.dtype), state


def _validate(
    query: object,
    key: object,
    value: object,
    a: object,
    b: object,
    A_log: object,
    dt_bias: object,
    state: object,
) -> None:
    tensors = {
        "query": query,
        "key": key,
        "value": value,
        "a": a,
        "b": b,
        "A_log": A_log,
        "dt_bias": dt_bias,
        "state": state,
    }
    for name, tensor in tensors.items():
        if not isinstance(tensor, torch.Tensor):
            raise TypeError(f"{name} must be a torch.Tensor")

    assert all(isinstance(tensor, torch.Tensor) for tensor in tensors.values())
    query = tensors["query"]
    key = tensors["key"]
    value = tensors["value"]
    a = tensors["a"]
    b = tensors["b"]
    A_log = tensors["A_log"]
    dt_bias = tensors["dt_bias"]
    state = tensors["state"]

    expected_ranks = {
        "query": 3,
        "key": 3,
        "value": 3,
        "a": 2,
        "b": 2,
        "A_log": 1,
        "dt_bias": 1,
        "state": 4,
    }
    for name, tensor in tensors.items():
        expected_rank = expected_ranks[name]
        if tensor.ndim != expected_rank:
            raise ValueError(f"{name} must be rank {expected_rank}, got rank {tensor.ndim}")
        if any(dimension <= 0 for dimension in tensor.shape):
            raise ValueError(f"{name} dimensions must be positive")

    for name in ("query", "key", "value", "a", "b"):
        tensor = tensors[name]
        if tensor.dtype is not torch.bfloat16:
            raise TypeError(f"{name} dtype must be torch.bfloat16, got {tensor.dtype}")
    for name in ("A_log", "dt_bias", "state"):
        tensor = tensors[name]
        if tensor.dtype is not torch.float32:
            raise TypeError(f"{name} dtype must be torch.float32, got {tensor.dtype}")

    for name, tensor in tensors.items():
        if tensor.layout is not torch.strided:
            raise ValueError(f"{name} must have torch.strided layout")
        if tensor.device != query.device:
            raise ValueError(
                f"{name} must be on the same device as query, "
                f"got {tensor.device} and {query.device}"
            )
    if query.device.type == "meta":
        raise ValueError("meta tensors are not supported")

    if key.shape != query.shape:
        raise ValueError(
            f"key shape must match query shape {tuple(query.shape)}, got {tuple(key.shape)}"
        )

    batch, num_query_heads, key_dim = query.shape
    num_value_heads, value_dim = value.shape[1:]
    if value.shape[0] != batch:
        raise ValueError("value batch dimension must match query")
    if num_value_heads % num_query_heads != 0:
        raise ValueError(
            "value head count must be divisible by query/key head count, "
            f"got {num_value_heads} and {num_query_heads}"
        )

    expected_gate_shape = (batch, num_value_heads)
    for name, tensor in (("a", a), ("b", b)):
        if tensor.shape != expected_gate_shape:
            raise ValueError(
                f"{name} must have shape {expected_gate_shape}, got {tuple(tensor.shape)}"
            )
    expected_parameter_shape = (num_value_heads,)
    for name, tensor in (("A_log", A_log), ("dt_bias", dt_bias)):
        if tensor.shape != expected_parameter_shape:
            raise ValueError(
                f"{name} must have shape {expected_parameter_shape}, got {tuple(tensor.shape)}"
            )

    expected_state_shape = (batch, num_value_heads, key_dim, value_dim)
    if state.shape != expected_state_shape:
        raise ValueError(f"state must have shape {expected_state_shape}, got {tuple(state.shape)}")
```

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/gdn_recurrent_decode.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
