# Verify recurrence with output norm (`gdn_recurrent_decode_gated_norm`) | ServingStudio Kernel Library

Run the gated delta rule over each request's verify tokens and apply the gated RMSNorm, in one launch.

A speculative (MTP) verify step carries q = num_tokens / batch_size tokens per request. Each request starts from the FP32 state checkpoint of its last accepted token, runs the Gated DeltaNet decode recurrence over its q tokens (L2-normalized Q/K, decay and beta gates in FP32), and writes one checkpoint per token. Each value head's BF16 output row is then RMS-normalized, scaled by a learned weight and multiplied by the output gate.

Category: Attention; group: Gated DeltaNet. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=gdn_recurrent_decode_gated_norm. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `per token: S′ = decay·S + k·(beta·(v − kᵀ(decay·S)))ᵀ; o = BF16(qᵀS′)`
- `y = BF16(o / √(mean(o²) + eps) · weight · gate(z)), gate = sigmoid or SiLU`
- `T = num_tokens; E₁ = num_value_heads·key_head_dim·value_head_dim`
- `FLOPs = 2·T·num_qk_heads·(3·key_head_dim + 1) + T·num_value_heads·key_head_dim + 4·T·num_value_heads + num_value_heads + 7·T·E₁ + 7·T·num_value_heads·value_head_dim`
- `bytes = (2·T·num_qk_heads·key_head_dim + 3·T·num_value_heads·value_head_dim + 2·T·num_value_heads + value_head_dim)·bytes(dtype) + (2·num_value_heads + (batch_size + T)·E₁)·bytes(state_dtype)`
- `TFLOPS = FLOPs / time`
- `GB/s = bytes / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. vllm_cuda counts only gdn_decode_post_conv_mtp_kernel. Operands come from one strided qkvz and one ba projection as in vLLM; the reference check and its state restoration run before timing.

## Caveats

- The gate activation is not a key: the kernel's sigmoid and SiLU builds differ by one multiply per element. Rows are measured with sigmoid.
- Requests' accepted counts cycle through 1..q; the kernel reads one checkpoint whatever the count.
- Every timed call advances the checkpoints in place.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `batch_size` | number | requests | Requests verified together. |
| `num_qk_heads` | number | heads | Query and key heads per token. |
| `num_value_heads` | number | heads | Value heads and recurrent states per token. |
| `key_head_dim` | number | elements | Features in each query or key head. |
| `value_head_dim` | number | elements | Features in each value head. |
| `dtype` | dtype |  | Element type of activations, gate, norm weight and output. |
| `state_dtype` | dtype |  | Element type of the recurrent state and gate parameters. |
| `num_tokens` | number | tokens | Tokens across the batch, an equal number per request: the accepted token plus the drafts. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_cuda`: vLLM's fused_gdn_decode_post_conv_mtp CUDA op: verify recurrence, per-token FP32 checkpoints and gated RMSNorm in one kernel (K = V = 128, num_value_heads / num_qk_heads in 1, 2, 3, 4, 8, at most 8 tokens per request). dtypes `bf16`, compute capability 8.0 or newer, GPUs A100-SXM4-40GB, A100-SXM4-80GB, A100-PCIe-40GB, A100-PCIe-80GB, H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip, H800-SXM5-80GB, H20-SXM-96GB, A40, L40, L40S, L4. Source: https://github.com/vllm-project/vllm/blob/main/csrc/libtorch_stable/gdn/fused_gdn_decode_kernel.cu

## Measured rows

104 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `vllm_cuda` | `bf16` | 104 |

## Used by these deployments

- [Qwen3.8 Flash Next NVFP4: vLLM, one GPU, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative.md) (`Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/gdn_recurrent_decode_gated_norm
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/gdn_recurrent_decode_gated_norm/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/gdn_recurrent_decode_gated_norm/configs

## Torch reference

What the kind computes, as Torch code (`profiling/runners/attention/gdn_spec_decode_reference.py` in ServingStudio Sim):

```python
"""Torch semantics for Gated DeltaNet speculative (MTP verify) decode.

A verify step carries ``q`` tokens per request: the accepted token plus the
drafts. vLLM keeps one checkpoint slot per token position (``state_indices``
is ``[B, q]``); the request's history starts from the checkpoint of its last
accepted token (column ``num_accepted - 1``), and every token writes its own
checkpoint. These references take that slot-indexed pool layout directly, so
the runners can compare whole pools.

The convolution keeps one raw-history row per request (``state_indices[:, 0]``)
of width ``kernel_size - 2 + q``; the recurrence keeps FP32 ``[Hv, Dv, Dk]``
slot contents (vLLM's transposed state layout).
"""

from __future__ import annotations

import torch
import torch.nn.functional as F

from profiling.runners.attention.gdn_recurrent_decode_reference import (
    gdn_recurrent_decode_reference,
)

__all__ = [
    "gdn_causal_conv_spec_decode_reference",
    "gdn_recurrent_spec_decode_reference",
    "gdn_recurrent_spec_decode_gated_norm_reference",
]


def _check_uniform(num_tokens: int, state_indices: torch.Tensor, num_accepted: torch.Tensor) -> int:
    if state_indices.ndim != 2:
        raise ValueError("state_indices must be [requests, tokens_per_request]")
    batch, per_request = state_indices.shape
    if num_tokens != batch * per_request:
        raise ValueError(
            f"{num_tokens} tokens do not fill {batch} requests of {per_request} tokens each"
        )
    if num_accepted.shape != (batch,):
        raise ValueError("num_accepted must hold one count per request")
    if torch.any(num_accepted < 1).item() or torch.any(num_accepted > per_request).item():
        raise ValueError("num_accepted must lie in [1, tokens_per_request]")
    if torch.any(state_indices <= 0).item():
        raise ValueError("state_indices must not name reserved slot zero")
    return per_request


def gdn_causal_conv_spec_decode_reference(
    x: torch.Tensor,
    weight: torch.Tensor,
    state: torch.Tensor,
    state_indices: torch.Tensor,
    num_accepted: torch.Tensor,
) -> tuple[torch.Tensor, torch.Tensor]:
    """Depthwise causal convolution + SiLU over ``q`` verify tokens per request.

    ``x`` is ``[B*q, C]`` (request-major), ``weight`` ``[C, W]``, ``state``
    ``[S, C, W - 2 + q]`` (dim-first), ``state_indices`` ``[B, q]`` int32 and
    ``num_accepted`` ``[B]`` int32. Request ``b`` reads history columns
    ``a-1 .. a+W-3`` of slot ``state_indices[b, 0]`` (``a`` its accepted
    count), convolves its ``q`` tokens, and rewrites that slot as history
    columns ``a .. a+W-3`` followed by its raw tokens. Returns the output (in
    ``x``'s dtype) and ``state`` (mutated in place).
    """
    per_request = _check_uniform(x.shape[0], state_indices, num_accepted)
    channels, width = weight.shape
    if state.shape[1:] != (channels, width - 2 + per_request):
        raise ValueError(f"state slots must be [{channels}, {width - 2 + per_request}]")
    output = torch.empty_like(x)
    weight_fp32 = weight.float()
    for request in range(state_indices.shape[0]):
        slot = int(state_indices[request, 0])
        accepted = int(num_accepted[request])
        tokens = x[request * per_request : (request + 1) * per_request]
        old = state[slot].clone()
        history = old[:, accepted - 1 : accepted - 1 + width - 1]
        sequence = torch.cat((history, tokens.transpose(0, 1)), dim=-1).float()
        for token in range(per_request):
            window = sequence[:, token : token + width]
            output[request * per_request + token] = F.silu(
                torch.sum(window * weight_fp32, dim=-1)
            ).to(x.dtype)
        state[slot] = torch.cat(
            (old[:, accepted : accepted + width - 2], tokens.transpose(0, 1)), dim=-1
        )
    return output, state


def gdn_recurrent_spec_decode_reference(
    query: torch.Tensor,
    key: torch.Tensor,
    value: torch.Tensor,
    a: torch.Tensor,
    b: torch.Tensor,
    A_log: torch.Tensor,
    dt_bias: torch.Tensor,
    state: torch.Tensor,
    state_indices: torch.Tensor,
    num_accepted: torch.Tensor,
) -> tuple[torch.Tensor, torch.Tensor]:
    """Gated delta-rule recurrence over ``q`` verify tokens per request.

    Q/K are ``[B*q, Hq, Dk]``, V ``[B*q, Hv, Dv]``, raw gates ``[B*q, Hv]``,
    ``state`` an FP32 slot pool ``[S, Hv, Dv, Dk]``. Each request starts from
    slot ``state_indices[b, a-1]`` and writes its state after token ``t`` to
    slot ``state_indices[b, t]``. Returns the ``[B*q, Hv, Dv]`` output (in the
    input dtype) and the pool (mutated in place).
    """
    per_request = _check_uniform(query.shape[0], state_indices, num_accepted)
    batch = state_indices.shape[0]
    rows = torch.arange(batch, device=state_indices.device)
    sources = state_indices[rows, num_accepted.to(torch.int64) - 1].to(torch.int64)
    running = state[sources].transpose(-1, -2).contiguous()
    output = torch.empty_like(value)

    def per_token(tensor: torch.Tensor, token: int) -> torch.Tensor:
        return tensor.view(batch, per_request, *tensor.shape[1:])[:, token].contiguous()

    for token in range(per_request):
        token_output, running = gdn_recurrent_decode_reference(
            per_token(query, token),
            per_token(key, token),
            per_token(value, token),
            per_token(a, token),
            per_token(b, token),
            A_log,
            dt_bias,
            running,
        )
        output.view(batch, per_request, *value.shape[1:])[:, token] = token_output
        state[state_indices[:, token].to(torch.int64)] = running.transpose(-1, -2)
    return output, state


def gdn_recurrent_spec_decode_gated_norm_reference(
    query: torch.Tensor,
    key: torch.Tensor,
    value: torch.Tensor,
    a: torch.Tensor,
    b: torch.Tensor,
    A_log: torch.Tensor,
    dt_bias: torch.Tensor,
    state: torch.Tensor,
    state_indices: torch.Tensor,
    num_accepted: torch.Tensor,
    output_gate: torch.Tensor,
    norm_weight: torch.Tensor,
    norm_eps: float,
) -> tuple[torch.Tensor, torch.Tensor]:
    """:func:`gdn_recurrent_spec_decode_reference` followed by the output norm.

    The BF16 recurrent output of each value head is RMS-normalized in FP32,
    scaled by ``norm_weight`` and multiplied by ``sigmoid(output_gate)``
    (``output_gate`` is ``[B*q, Hv, Dv]``), then rounded to BF16.
    """
    core, state = gdn_recurrent_spec_decode_reference(
        query, key, value, a, b, A_log, dt_bias, state, state_indices, num_accepted
    )
    core_fp32 = core.float()
    rstd = torch.rsqrt(torch.mean(core_fp32 * core_fp32, dim=-1, keepdim=True) + norm_eps)
    gated = core_fp32 * rstd * norm_weight.float() * torch.sigmoid(output_gate.float())
    return gated.to(torch.bfloat16), state
```

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/gdn_recurrent_decode_gated_norm.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
