# HC gate mix (`hc_gate_mix`) | ServingStudio Kernel Library

Collapse the normalized residual streams into one block input with a per-feature sigmoid gate.

The last step of a gated hyper-connection module: the low-rank up projection produces a gate for every stream feature, and the block input is the mean over the hc_mult streams of sigmoid(gate)·x, where x is the per-stream normalized residual. One Triton program per (token, 512-feature tile) loops over the streams. Inputs are seeded random bf16 rows.

Category: Normalization; group: Hyper-connections. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=hc_gate_mix. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `out = (1 / hc_mult) · Σ_j σ(gate_j) · x_j`
- `GB/s = 2 · num_tokens · (2·hc_mult·hidden_size + hidden_size) / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. Three warm-up calls and a comparison with an independent PyTorch reference precede capture; every launch of the call is counted.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_tokens` | number | tokens | Token rows. |
| `hidden_size` | number | elements | Features in each residual stream. |
| `hc_mult` | number | streams | Parallel residual streams per token. |
| `hidden_dtype` | dtype |  | Element type of the streams, the gate and the mixed output. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_triton`: The fork's hc_gate_mix Triton op. dtypes `bf16`, compute capability 8.0 or newer, GPUs A100-SXM4-40GB, A100-SXM4-80GB, A100-PCIe-40GB, A100-PCIe-80GB, H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip, H800-SXM5-80GB, H20-SXM-96GB, A40, L40, L40S, L4. Source: https://github.com/serendipity-zk/vllm/blob/892da0822f7e866cec8e519727f014c45a5ad792/vllm/models/qwen4_exp/nvidia/ops/hc.py

## Measured rows

68 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `vllm_triton` | `bf16` | 68 |

## Used by these deployments

- [Qwen3.8 Flash Next NVFP4: vLLM, one GPU, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative.md) (`Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/hc_gate_mix
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/hc_gate_mix/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/hc_gate_mix/configs

## Torch reference

What the kind computes, as Torch code (`profiling/runners/mhc/hc_reference.py` in ServingStudio Sim):

```python
"""Independent value references for the gated hyper-connection (HC) kernels.

The residual holds ``hc_mult`` streams of ``hidden_size`` features per token,
stream-outer (``[tokens, hc_mult * hidden_size]``). The norms are Gemma-style:
``x / rms(x) * (1 + w)``, one RMS per stream, with a ``[hc_mult * hidden_size]``
weight.
"""

from typing import Any

from profiling.runners.norm.gemma_rms_norm_reference import gemma_rms_norm_reference


def hc_combine_reference(
    torch: Any, residual: Any, block: Any, injection: Any, hc_mult: int
) -> Any:
    """``residual + coeff * block`` per stream.

    coeff = 2*sigmoid(logit/hc_mult), or 1 without logits.
    """
    streams = residual.float().unflatten(-1, (hc_mult, -1))
    if injection is None:
        coeff = torch.ones(1, hc_mult, 1, device=residual.device)
    else:
        coeff = (2.0 * torch.sigmoid(injection.float() / hc_mult))[:, :, None]
    return (streams + coeff * block.float()[:, None, :]).flatten(-2).to(residual.dtype)


def hc_combine_norm_reference(
    torch: Any, residual: Any, block: Any, injection: Any, weight: Any, eps: float, hc_mult: int
) -> tuple[Any, Any]:
    combined = hc_combine_reference(torch, residual, block, injection, hc_mult)
    return combined, gemma_rms_norm_reference(torch, combined, weight, eps, hc_mult)


def hc_gate_mix_reference(torch: Any, x: Any, gate: Any, hc_mult: int) -> Any:
    """Mean over the streams of ``sigmoid(gate) * x``."""
    mixed = torch.sigmoid(gate.float()) * x.float()
    return mixed.unflatten(-1, (hc_mult, -1)).mean(-2).to(x.dtype)


def scaled_silu_reference(torch: Any, x: Any, divisor: float) -> Any:
    return torch.nn.functional.silu(x.float() / divisor).to(x.dtype)


__all__ = [
    "hc_combine_norm_reference",
    "hc_combine_reference",
    "hc_gate_mix_reference",
    "scaled_silu_reference",
]
```

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/hc_gate_mix.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
