# mHC pre-block with RMSNorm (`mhc_pre_rms_norm`) | ServingStudio Kernel Library

From a token's parallel residual streams, compute the mHC mixing weights and the RMS-normalized input of the next block.

mHC keeps hc_mult residual streams per token instead of one. Before each block, one vLLM TileLang call projects the streams to three sets of mixing weights, sums the streams with the pre-mix weights into the block input, and RMS-normalizes it. TileLang compiles the call for each hidden_size and hc_mult; the measurement uses random residuals.

Category: Normalization; group: Hyper-connections. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=mhc_pre_rms_norm. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `x = the hc_mult streams concatenated; mixes = x · fnᵀ / √(mean(x²) + 1e-6)`
- `pre = σ(s₀·mixes_pre + b_pre) + 1e-6; post = 2·σ(s₁·mixes_post + b_post)`
- `comb = Sinkhorn₂₀(softmax(s₂·mixes_comb + b_comb)), an hc_mult × hc_mult matrix`
- `input = RMSNorm(Σⱼ preⱼ · streamⱼ), ε = 1e-6; returns (post, comb, input)`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. Three warm-up calls run first, and every launch of the TileLang call is counted. The outputs are checked against vLLM's PyTorch implementation before timing.

## Caveats

- The runner uses ε = 1e-6; another ε runs the same launches.
- TFLOPS is not computed. GB/s counts the streams and weights read once and the mixes and block input written once.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_tokens` | number | tokens | Token rows processed by the MHC block. |
| `hidden_size` | number | elements | Features in each residual stream. |
| `hc_mult` | number | streams | Parallel residual streams per token. |
| `hidden_dtype` | dtype |  | Element type of the residual streams. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_tilelang`: vLLM's mhc_pre_tilelang with the RMSNorm fused in. dtypes `bf16`, any CUDA GPU. Source: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/kernels/mhc/tilelang.py

## Measured rows

204 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `vllm_tilelang` | `bf16` | 136 |
| NVIDIA H200 | `vllm_tilelang` | `bf16` | 68 |

## Used by these deployments

- [DeepSeek V4 Flash 0731: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm.md) (`DeepSeek-V4-Flash-0731/deepseek_v4_vllm`)
- [DeepSeek V4 Flash 0731: vLLM, DP attention, EP, parallel streams serialized](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams.md) (`DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams`)
- [DeepSeek V4.1 Flash 1017: vLLM, DP attention, EP MoE, DSpark speculative decoding](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dp_attn_ep_moe_dspark.md) (`DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dp_attn_ep_moe_dspark`)
- [DeepSeek V4.1 Flash 1017: vLLM, TP attention, EP MoE, DSpark speculative decoding](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dspark.md) (`DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dspark`)
- [DeepSeek V4.1 Flash: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash/deepseek_v41_vllm.md) (`DeepSeek-V4.1-Flash/deepseek_v41_vllm`)
- [DeepSeek V4.1 Flash: vLLM, TP attention, EP MoE, side streams serialized](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash/deepseek_v41_vllm_serial_streams.md) (`DeepSeek-V4.1-Flash/deepseek_v41_vllm_serial_streams`)
- [GLM-5.3-Flash NVFP4: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe`)
- [GLM-5.3-Flash: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe`)
- [GLM-5.3-Flash: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe`)
- [GLM-5.3-Flash: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/mhc_pre_rms_norm
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/mhc_pre_rms_norm/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/mhc_pre_rms_norm/configs

## Torch reference

What the kind computes, as Torch code (`profiling/runners/mhc/_common.py` in ServingStudio Sim):

```python
"""Shared production identity and correctness helpers for DeepSeek V4 MHC."""

from dataclasses import dataclass
from typing import Any

from profiling.db.args import DType
from profiling.runners.exceptions import ProfilerNotImplemented

# Every model with mHC boundaries launches these same post/pre callables, some
# with an rms_norm_eps other than RMS_EPS below; the eps is a scalar and does
# not change the launch sequence or the timing.
# The TileLang kernels are JIT-compiled for the current GPU and take hidden_size
# and hc_mult from the tensor shapes, so neither the GPU nor the stream geometry
# is a launch constraint; every runner checks its outputs against vLLM's Torch
# implementation before timing, which rejects a geometry a kernel mishandles.
RMS_EPS = 1e-6
HC_EPS = 1e-6
POST_MULTIPLIER = 2.0
SINKHORN_ITERATIONS = 20


def mix_width(hc_mult: int) -> int:
    """Rows of fn: hc_mult pre + hc_mult post + hc_mult² comb mixes."""
    return hc_mult * (hc_mult + 2)


_BF16_BYTES = 2
_FP32_BYTES = 4


@dataclass(frozen=True)
class Shape:
    num_tokens: int
    hidden_size: int
    hc_mult: int


@dataclass(frozen=True)
class CommonInputs:
    residual: Any
    fn: Any
    hc_scale: Any
    hc_base: Any
    norm_weight: Any


def validate_args(
    kind: str,
    num_tokens: int,
    hidden_size: int,
    hc_mult: int,
    hidden_dtype: DType | str,
) -> Shape:
    for name, value in (
        ("num_tokens", num_tokens),
        ("hidden_size", hidden_size),
        ("hc_mult", hc_mult),
    ):
        if type(value) is not int or value <= 0:
            raise ValueError(f"{name} must be a positive integer")
    # The vLLM wrappers assert bf16 residual streams (mhc/tilelang.py).
    if DType.from_value(hidden_dtype) is not DType.BF16:
        raise ProfilerNotImplemented(f"{kind} requires hidden_dtype=bf16")
    return Shape(num_tokens, hidden_size, hc_mult)


def prepare_common(torch: Any, shape: Shape) -> CommonInputs:
    generator = torch.Generator().manual_seed(41)
    hidden, streams = shape.hidden_size, shape.hc_mult
    residual = torch.randn(
        (shape.num_tokens, streams, hidden),
        dtype=torch.bfloat16,
        generator=generator,
    ).cuda()
    width = mix_width(streams)
    fn = (torch.randn((width, streams * hidden), generator=generator) * 0.01).cuda()
    hc_scale = torch.tensor([0.5, 0.5, 0.5], dtype=torch.float32).cuda()
    hc_base = torch.zeros(width, dtype=torch.float32).cuda()
    norm_weight = torch.ones(hidden, dtype=torch.bfloat16).cuda()
    return CommonInputs(residual, fn, hc_scale, hc_base, norm_weight)


def reference_pre(torch: Any, inputs: CommonInputs) -> tuple[Any, Any, Any]:
    from vllm.model_executor.kernels.mhc.torch import mhc_pre_torch

    post_mix, comb_mix, layer_input = mhc_pre_torch(
        inputs.residual,
        inputs.fn,
        inputs.hc_scale,
        inputs.hc_base,
        RMS_EPS,
        HC_EPS,
        HC_EPS,
        POST_MULTIPLIER,
        SINKHORN_ITERATIONS,
    )
    normalized = torch.nn.functional.rms_norm(
        layer_input,
        (inputs.residual.shape[-1],),
        inputs.norm_weight,
        RMS_EPS,
    )
    return post_mix, comb_mix, normalized


def residual_bytes(shape: Shape) -> int:
    """The bf16 residual streams, (num_tokens, hc_mult, hidden_size)."""
    return _BF16_BYTES * shape.num_tokens * shape.hc_mult * shape.hidden_size


def hidden_bytes(shape: Shape) -> int:
    """One bf16 hidden state per token, (num_tokens, hidden_size)."""
    return _BF16_BYTES * shape.num_tokens * shape.hidden_size


def mix_bytes(shape: Shape) -> int:
    """The fp32 post-mix (num_tokens, hc_mult, 1) and comb-mix (num_tokens, hc_mult, hc_mult)."""
    return _FP32_BYTES * shape.num_tokens * (shape.hc_mult + shape.hc_mult * shape.hc_mult)


def pre_weight_bytes(shape: Shape) -> int:
    """fn, hc_scale, hc_base and the RMSNorm weight, read once per call."""
    width = mix_width(shape.hc_mult)
    fn = width * shape.hc_mult * shape.hidden_size
    return _FP32_BYTES * (fn + 3 + width) + _BF16_BYTES * shape.hidden_size


def head_weight_bytes(shape: Shape) -> int:
    """The terminal head's fn rows, scale and base, and the final RMSNorm weight."""
    head_fn = shape.hc_mult * shape.hc_mult * shape.hidden_size
    return _FP32_BYTES * (head_fn + 1 + shape.hc_mult) + _BF16_BYTES * shape.hidden_size


def post_pre_one_pass_bytes(shape: Shape) -> int:
    """The fused post/pre call's traffic if it touched each tensor once.

    Reads the layer output, the streams, the previous mixes and the weights,
    and writes the updated streams, the next mixes and the next block input.
    """
    return (
        2 * residual_bytes(shape)
        + 2 * mix_bytes(shape)
        + 2 * hidden_bytes(shape)
        + pre_weight_bytes(shape)
    )


def bandwidth_gbps(logical_bytes: int, time_ms: float) -> float:
    return logical_bytes / (time_ms / 1000.0) / 1e9


def assert_outputs_close(torch: Any, actual: tuple[Any, ...], expected: tuple[Any, ...]) -> None:
    if len(actual) != len(expected):
        raise AssertionError(f"expected {len(expected)} MHC outputs, got {len(actual)}")
    for actual_tensor, expected_tensor in zip(actual, expected, strict=True):
        torch.testing.assert_close(actual_tensor, expected_tensor, atol=0.05, rtol=0.02)


__all__ = [
    "CommonInputs",
    "HC_EPS",
    "POST_MULTIPLIER",
    "RMS_EPS",
    "SINKHORN_ITERATIONS",
    "Shape",
    "assert_outputs_close",
    "bandwidth_gbps",
    "head_weight_bytes",
    "hidden_bytes",
    "mix_bytes",
    "post_pre_one_pass_bytes",
    "pre_weight_bytes",
    "prepare_common",
    "mix_width",
    "reference_pre",
    "residual_bytes",
    "validate_args",
]
```

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/mhc_pre_rms_norm.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
