# Gated DeltaNet output RMSNorm (`gdn_gated_rms_norm`) | ServingStudio Kernel Library

RMS-normalize each recurrent output row and multiply it by a SiLU gate.

The last step of a Gated DeltaNet block before the output projection: each value head's output row is RMS-normalized, scaled by a learned weight and multiplied by a SiLU gate. m counts rows, one per token per value head, and hidden is the row width. Statistics and gating are computed in FP32 and the output is rounded to BF16.

Category: Normalization; group: RMSNorm. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=gdn_gated_rms_norm. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `y = (x / √(mean(x²) + 1e−6) · weight) · (z · sigmoid(z))`
- `FLOPs = 7·m·hidden + 2·m`
- `bytes = (3·m·hidden + hidden)·bytes(dtype)`
- `TFLOPS = FLOPs / time`
- `GB/s = bytes / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. torch counts every launch of the reference; vllm_triton counts only layer_norm_fwd_kernel. Allocation and the vLLM output check run before timing.

## Caveats

- torch uses several launches; vllm_triton fuses normalization and gating in one.
- GB/s counts logical input and output bytes, not the torch backend's FP32 intermediates.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `m` | number | rows | Value-head rows across the token batch. |
| `hidden` | number | elements | Features normalized within each row. |
| `dtype` | dtype |  | Element type of the activation, gate, weight and output. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `torch`: The PyTorch reference, timed across its separate RMSNorm and SiLU-gate launches. dtypes `bf16`, any CUDA GPU.
- `vllm_triton`: vLLM's rmsnorm_fn Triton call fuses rowwise RMSNorm and the SiLU output gate. dtypes `bf16`, any CUDA GPU. Source: https://github.com/vllm-project/vllm/blob/main/vllm/third_party/flash_linear_attention/ops/layernorm_guard.py

## Measured rows

44 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `vllm_triton` | `bf16` | 15 |
| NVIDIA H200 | `vllm_triton` | `bf16` | 29 |

## Used by these deployments

- [Qwen3.6 35B-A3B FP8: Single GPU](https://servingstudio.cs.washington.edu/models/Qwen3.6-35B-A3B-FP8/qwen36_local.md) (`Qwen3.6-35B-A3B-FP8/qwen36_local`)
- [Qwen3.8 27B NVFP4: vLLM, one GPU](https://servingstudio.cs.washington.edu/models/Qwen3.8-27B-NVFP4/qwen35_vllm_nvfp4_gdn_dense.md) (`Qwen3.8-27B-NVFP4/qwen35_vllm_nvfp4_gdn_dense`)
- [Qwen3.8 Flash Next NVFP4: vLLM, one GPU, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative.md) (`Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/gdn_gated_rms_norm
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/gdn_gated_rms_norm/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/gdn_gated_rms_norm/configs

## Torch reference

What the kind computes, as Torch code (`profiling/runners/attention/gdn_gated_rms_norm_reference.py` in ServingStudio Sim):

```python
"""Torch semantics for Qwen3.5/3.6 GDN gated RMS normalization."""

from __future__ import annotations

import torch

__all__ = ["gdn_gated_rms_norm_reference"]

_EPSILON = 1e-6


def gdn_gated_rms_norm_reference(
    x: torch.Tensor,
    z: torch.Tensor,
    weight: torch.Tensor,
) -> torch.Tensor:
    """Apply Qwen's norm-before-SiLU-gate output normalization.

    ``x`` and output gate ``z`` are ``[M, hidden]`` and ``weight`` is
    ``[hidden]``. Qwen3.6 reshapes its GDN output to ``hidden=128`` and
    ``M=num_tokens * 32``. All semantic inputs and the returned output are
    BF16. RMS statistics, affine multiplication, SiLU, and gating are computed
    in FP32; only the final output is rounded to BF16.

    This reference freezes epsilon to ``1e-6``, has no bias or groups, and
    always normalizes before applying a SiLU gate. It intentionally excludes
    non-BF16 data, sigmoid-only or alternate activations, norm-after-gate, and
    alternate epsilon. Non-contiguous strided tensors are supported by the
    semantic math; contiguous feature storage is a production Triton backend
    constraint rather than a semantic axis. Inputs are not mutated, and the
    returned tensor has fresh storage.
    """
    _validate(x, z, weight)

    x_fp32 = x.float()
    z_fp32 = z.float()
    weight_fp32 = weight.float()
    rstd = torch.rsqrt(x_fp32.square().mean(dim=-1, keepdim=True) + _EPSILON)
    normalized = x_fp32 * rstd * weight_fp32
    gate = z_fp32 * torch.sigmoid(z_fp32)
    return (normalized * gate).to(torch.bfloat16)


def _validate(x: object, z: object, weight: object) -> None:
    tensors = {"x": x, "z": z, "weight": weight}
    for name, tensor in tensors.items():
        if not isinstance(tensor, torch.Tensor):
            raise TypeError(f"{name} must be a torch.Tensor")

    assert all(isinstance(tensor, torch.Tensor) for tensor in tensors.values())
    x = tensors["x"]
    z = tensors["z"]
    weight = tensors["weight"]

    expected_ranks = {"x": 2, "z": 2, "weight": 1}
    for name, tensor in tensors.items():
        expected_rank = expected_ranks[name]
        if tensor.ndim != expected_rank:
            raise ValueError(f"{name} must be rank {expected_rank}, got rank {tensor.ndim}")
        if any(dimension <= 0 for dimension in tensor.shape):
            raise ValueError(f"{name} dimensions must be positive")
        if tensor.dtype is not torch.bfloat16:
            raise TypeError(f"{name} dtype must be torch.bfloat16, got {tensor.dtype}")
        if tensor.layout is not torch.strided:
            raise ValueError(f"{name} must have torch.strided layout")
        if tensor.device != x.device:
            raise ValueError(
                f"{name} must be on the same device as x, got {tensor.device} and {x.device}"
            )

    if x.device.type == "meta":
        raise ValueError("meta tensors are not supported")
    if z.shape != x.shape:
        raise ValueError(f"z shape must match x shape {tuple(x.shape)}, got {tuple(z.shape)}")
    expected_weight_shape = (x.shape[1],)
    if weight.shape != expected_weight_shape:
        raise ValueError(
            f"weight must have shape {expected_weight_shape}, got {tuple(weight.shape)}"
        )
```

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/gdn_gated_rms_norm.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
