# Grouped RMSNorm (`grouped_rms_norm`) | ServingStudio Kernel Library

RMS-normalize each contiguous group of a row separately.

A row of num_groups · hidden_size features is split into num_groups contiguous groups, and each group is normalized by its own RMS and scaled by a Gemma (1 + w) weight that covers the whole row. Gated hyper-connections run it on the stream-outer residual (one group per stream) when an HC module has no pending combine to fuse, such as the first layer. One Triton program per (token, group). Inputs are seeded random bf16 rows.

Category: Normalization; group: RMSNorm. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=grouped_rms_norm. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `y_g = x_g / √(mean(x_g²) + ε) · (1 + w_g), per token and group`
- `GB/s = 2 · (2·num_tokens·num_groups·hidden_size + num_groups·hidden_size) / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. Three warm-up calls and a comparison with an independent PyTorch reference precede capture; every launch of the call is counted.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_tokens` | number | tokens | Token rows. |
| `hidden_size` | number | elements | Features in each group, normalized together. |
| `num_groups` | number | groups | Contiguous groups per row. |
| `hidden_dtype` | dtype |  | Element type of the input, weight and output. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_triton`: The fork's grouped_gemma_rmsnorm Triton op. dtypes `bf16`, compute capability 8.0 or newer, GPUs A100-SXM4-40GB, A100-SXM4-80GB, A100-PCIe-40GB, A100-PCIe-80GB, H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip, H800-SXM5-80GB, H20-SXM-96GB, A40, L40, L40S, L4. Source: https://github.com/serendipity-zk/vllm/blob/892da0822f7e866cec8e519727f014c45a5ad792/vllm/models/qwen4_exp/nvidia/ops/hc.py

## Measured rows

68 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `vllm_triton` | `bf16` | 68 |

## Used by these deployments

- [Qwen3.8 Flash Next NVFP4: vLLM, one GPU, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative.md) (`Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/grouped_rms_norm
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/grouped_rms_norm/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/grouped_rms_norm/configs

## Torch reference

What the kind computes, as Torch code (`profiling/runners/mhc/hc_reference.py` in ServingStudio Sim):

```python
"""Independent value references for the gated hyper-connection (HC) kernels.

The residual holds ``hc_mult`` streams of ``hidden_size`` features per token,
stream-outer (``[tokens, hc_mult * hidden_size]``). The norms are Gemma-style:
``x / rms(x) * (1 + w)``, one RMS per stream, with a ``[hc_mult * hidden_size]``
weight.
"""

from typing import Any

from profiling.runners.norm.gemma_rms_norm_reference import gemma_rms_norm_reference


def hc_combine_reference(
    torch: Any, residual: Any, block: Any, injection: Any, hc_mult: int
) -> Any:
    """``residual + coeff * block`` per stream.

    coeff = 2*sigmoid(logit/hc_mult), or 1 without logits.
    """
    streams = residual.float().unflatten(-1, (hc_mult, -1))
    if injection is None:
        coeff = torch.ones(1, hc_mult, 1, device=residual.device)
    else:
        coeff = (2.0 * torch.sigmoid(injection.float() / hc_mult))[:, :, None]
    return (streams + coeff * block.float()[:, None, :]).flatten(-2).to(residual.dtype)


def hc_combine_norm_reference(
    torch: Any, residual: Any, block: Any, injection: Any, weight: Any, eps: float, hc_mult: int
) -> tuple[Any, Any]:
    combined = hc_combine_reference(torch, residual, block, injection, hc_mult)
    return combined, gemma_rms_norm_reference(torch, combined, weight, eps, hc_mult)


def hc_gate_mix_reference(torch: Any, x: Any, gate: Any, hc_mult: int) -> Any:
    """Mean over the streams of ``sigmoid(gate) * x``."""
    mixed = torch.sigmoid(gate.float()) * x.float()
    return mixed.unflatten(-1, (hc_mult, -1)).mean(-2).to(x.dtype)


def scaled_silu_reference(torch: Any, x: Any, divisor: float) -> Any:
    return torch.nn.functional.silu(x.float() / divisor).to(x.dtype)


__all__ = [
    "hc_combine_norm_reference",
    "hc_combine_reference",
    "hc_gate_mix_reference",
    "scaled_silu_reference",
]
```

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/grouped_rms_norm.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
