# HC combine + per-stream norm (`hc_combine_norm`) | ServingStudio Kernel Library

Add a block's output into every residual stream with a learned gate, then RMS-normalize each stream, in one launch.

Gated hyper-connections keep hc_mult residual streams of hidden_size features per token (stream-outer). The combine of a finished block is delayed to the next HC module, where one Triton program per (token, stream) adds coeff·block to the stream, stores the bf16 result, and normalizes it with a Gemma (1 + w) per-stream weight. coeff = 2·sigmoid(logit / hc_mult) from the previous module's injection logits. Inputs are seeded random bf16 rows; the logits are a contiguous \[tokens, hc_mult\] tensor.

Category: Normalization; group: Hyper-connections. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=hc_combine_norm. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `s_j ← s_j + 2·σ(logit_j / hc_mult) · block; y_j = s_j / √(mean(s_j²) + ε) · (1 + w_j)`
- `GB/s = 2 · (num_tokens · (3·hc_mult·hidden_size + hidden_size + hc_mult) + hc_mult·hidden_size) / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. Three warm-up calls and a comparison with an independent PyTorch reference precede capture; every launch of the call is counted.

## Caveats

- Without injection logits (unit coefficient, the MTP layer's first combine) the op compiles a variant that has no separate row: on B200 it is about 0.3-0.4 µs faster below 256 rows and within 3% above 1k rows.
- The production logits are a strided slice of the down-projection output; that ran within 1% of the contiguous logits used here.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_tokens` | number | tokens | Token rows. |
| `hidden_size` | number | elements | Features in each residual stream. |
| `hc_mult` | number | streams | Parallel residual streams per token. |
| `hidden_dtype` | dtype |  | Element type of the streams, block output and outputs. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_triton`: The fork's hc_combine_norm Triton op (learned injection logits). dtypes `bf16`, compute capability 8.0 or newer, GPUs A100-SXM4-40GB, A100-SXM4-80GB, A100-PCIe-40GB, A100-PCIe-80GB, H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip, H800-SXM5-80GB, H20-SXM-96GB, A40, L40, L40S, L4. Source: https://github.com/serendipity-zk/vllm/blob/892da0822f7e866cec8e519727f014c45a5ad792/vllm/models/qwen4_exp/nvidia/ops/hc.py

## Measured rows

68 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `vllm_triton` | `bf16` | 68 |

## Used by these deployments

- [Qwen3.8 Flash Next NVFP4: vLLM, one GPU, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative.md) (`Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/hc_combine_norm
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/hc_combine_norm/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/hc_combine_norm/configs

## Torch reference

What the kind computes, as Torch code (`profiling/runners/mhc/hc_reference.py` in ServingStudio Sim):

```python
"""Independent value references for the gated hyper-connection (HC) kernels.

The residual holds ``hc_mult`` streams of ``hidden_size`` features per token,
stream-outer (``[tokens, hc_mult * hidden_size]``). The norms are Gemma-style:
``x / rms(x) * (1 + w)``, one RMS per stream, with a ``[hc_mult * hidden_size]``
weight.
"""

from typing import Any

from profiling.runners.norm.gemma_rms_norm_reference import gemma_rms_norm_reference


def hc_combine_reference(
    torch: Any, residual: Any, block: Any, injection: Any, hc_mult: int
) -> Any:
    """``residual + coeff * block`` per stream.

    coeff = 2*sigmoid(logit/hc_mult), or 1 without logits.
    """
    streams = residual.float().unflatten(-1, (hc_mult, -1))
    if injection is None:
        coeff = torch.ones(1, hc_mult, 1, device=residual.device)
    else:
        coeff = (2.0 * torch.sigmoid(injection.float() / hc_mult))[:, :, None]
    return (streams + coeff * block.float()[:, None, :]).flatten(-2).to(residual.dtype)


def hc_combine_norm_reference(
    torch: Any, residual: Any, block: Any, injection: Any, weight: Any, eps: float, hc_mult: int
) -> tuple[Any, Any]:
    combined = hc_combine_reference(torch, residual, block, injection, hc_mult)
    return combined, gemma_rms_norm_reference(torch, combined, weight, eps, hc_mult)


def hc_gate_mix_reference(torch: Any, x: Any, gate: Any, hc_mult: int) -> Any:
    """Mean over the streams of ``sigmoid(gate) * x``."""
    mixed = torch.sigmoid(gate.float()) * x.float()
    return mixed.unflatten(-1, (hc_mult, -1)).mean(-2).to(x.dtype)


def scaled_silu_reference(torch: Any, x: Any, divisor: float) -> Any:
    return torch.nn.functional.silu(x.float() / divisor).to(x.dtype)


__all__ = [
    "hc_combine_norm_reference",
    "hc_combine_reference",
    "hc_gate_mix_reference",
    "scaled_silu_reference",
]
```

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/hc_combine_norm.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
