# Reduce-scatter + short conv + all-gather + residual + RMSNorm (`rs_sconv_ag_residual_rms_norm`) | ServingStudio Kernel Library

Sum sublayer partials across GPUs, run a causal short convolution on each GPU's hidden shard, gather it back, add the residual and RMS-normalize.

One fused call: reduce-scatter, short conv, all-gather, residual add and RMSNorm. Each GPU publishes its partial (plus the MoE sink partial when has_shared) to the shard owners through symmetric memory; each owner sums them, writes the result into the paged conv-state cache, applies the causal depthwise k=4 convolution with skip over that cache, adds the residual and publishes its shard to every GPU, which gathers the full row and applies RMSNorm. Inputs are random BF16 partials; the conv-state cache uses the fork's paged layout (block size 4) with one decode token per request and its prior taps already cached.

Category: Communication. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=rs_sconv_ag_residual_rms_norm. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `x = Σᵣ (partialᵣ + sharedᵣ)  (reduce-scatter over hidden_dim)`
- `y = x + Σᵢ conv_weightᵢ · x[pos − 3 + i]`
- `h = residual + y;  normed = h / √(mean(h²) + 1e-6) · weight`
- `algbw = num_tokens·hidden_dim·bytes per element / time`
- `busbw = algbw · 2(num_gpus − 1) / num_gpus`

## How it is measured

Host time of CUDA graph replays (vLLM runs the call inside a graph) of 30 back-to-back calls, divided by the call count, as the other graph-captured PDL collectives are timed. After a reference check and 50 direct calls, a host barrier aligns the GPUs; the slowest GPU is kept.

## Caveats

- Every GPU arrives together, so the time excludes the cross-GPU skew that the spin-waits absorb in serving (for example uneven expert load before an MoE call).
- algbw and busbw divide the reduce-scatter input size by the whole call's time, so they include the conv, residual and norm work.
- The 30 calls in a graph reuse the same buffers with no L2 clear between them, so the inputs and the conv-cache rows are L2-warm.
- Rows were measured on one conv-state layout that the args do not carry: 8 KV heads of head_dim 128 (per-head width 2·128 + 2·hidden_dim/8, padded to a power of two), so num_gpus is at most 8.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_gpus` | number | GPUs | GPUs in the tensor-parallel group. |
| `num_tokens` | number | tokens | Token rows on every GPU (the padded CUDA-graph batch). |
| `hidden_dim` | number | elements | Model hidden size; each GPU owns hidden_dim / num_gpus. |
| `conv_kernel_size` | number | tokens | Short-conv window; the kernel requires 4. |
| `has_shared` | label |  | Whether a second (MoE sink) partial is summed into the reduce-scatter input. |
| `dtype` | dtype |  | Element type of the partials, residual, cache and outputs. |
| `fabric` | label |  | Interconnect label used to identify the measurement; the run uses the reserved GPUs' links. |

## Metrics

- Time (`time_ms`, ms)
- Algorithm bandwidth (`algbw_gbps`, GB/s)
- Bus bandwidth (`busbw_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_lamport_triton`: The alignment fork's LamportRSConv.rs_sconv_ag_add_norm: four PDL-chained Triton launches over torch symmetric memory, run in a real vLLM TP group. dtypes `bf16`, compute capability 8.0 or newer, GPUs A100-SXM4-40GB, A100-SXM4-80GB, A100-PCIe-40GB, A100-PCIe-80GB, H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip, H800-SXM5-80GB, H20-SXM-96GB, A40, L40, L40S, L4.

## Measured rows

254 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `vllm_lamport_triton` | `bf16` | 254 |

## Used by these deployments

- [Inkling NVFP4: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Inkling-NVFP4/inkling_vllm_nvfp4_rel_attn_moe.md) (`Inkling-NVFP4/inkling_vllm_nvfp4_rel_attn_moe`)
- [Inkling Small NVFP4: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Inkling-Small-NVFP4/inkling_vllm_nvfp4_rel_attn_moe.md) (`Inkling-Small-NVFP4/inkling_vllm_nvfp4_rel_attn_moe`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/rs_sconv_ag_residual_rms_norm
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/rs_sconv_ag_residual_rms_norm/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/rs_sconv_ag_residual_rms_norm/configs

## Torch reference

What the kind computes, as Torch code (`profiling/runners/comm/rs_sconv_ag_residual_rms_norm_reference.py` in ServingStudio Sim):

```python
"""Torch semantics for the fused RS + short-conv + AG + residual + RMSNorm.

The production call (the alignment fork's ``LamportRSConv.rs_sconv_ag_add_norm``) is a
tensor-parallel collective. This reference computes the same result on one
device from every rank's inputs at once, with no communication and no paging:

    v_r    = bf16(partial_r + shared_r)                  # per-rank sum before the RS
    x      = bf16(Σ_r v_r)                               # reduce-scatter result (all shards)
    y      = bf16(Σ_i w_i · x[pos-K+1+i] + x[pos])       # causal depthwise conv + skip
    h      = bf16(residual + y)                          # residual add (gathered row)
    normed = bf16(h · rsqrt(mean(h²) + eps) · gamma)     # RMSNorm

Additions run in FP32 and round to BF16 at the same points as the production
kernels. The conv taps before the current token come from ``prior_taps`` (the
conv-state cache content held before the call) unless an earlier token of the
same request is in this batch: the production path inserts every token's ``x``
into the cache before any conv reads, so same-batch taps see this call's ``x``.
"""

from __future__ import annotations

import torch

__all__ = ["rs_sconv_ag_residual_rms_norm_reference"]


def rs_sconv_ag_residual_rms_norm_reference(
    partials: torch.Tensor,
    shared: torch.Tensor | None,
    residual: torch.Tensor,
    conv_weight: torch.Tensor,
    norm_weight: torch.Tensor,
    eps: float,
    prior_taps: torch.Tensor,
    positions: torch.Tensor,
    seq_idx: torch.Tensor,
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
    """Return ``(normed, residual_out, conv_input)``, each BF16 ``[T, H]``.

    ``partials`` and ``shared`` are BF16 ``[W, T, H]`` (one row block per TP
    rank); ``shared`` is the optional MoE sink partial. ``residual`` is BF16
    ``[T, H]``. ``conv_weight`` is the full BF16 depthwise filter ``[H, K]``
    (rank r applies rows ``r·H/W .. (r+1)·H/W``). ``norm_weight`` is BF16 ``[H]``.
    ``prior_taps`` is BF16 ``[T, K-1, H]``: the cached conv input at absolute
    positions ``pos-K+1 .. pos-1`` before this call; taps at negative positions
    are zero whatever it holds. ``positions`` is ``[T]`` absolute positions and
    ``seq_idx`` ``[T]`` the owning request of each token. ``conv_input`` is the
    reduced ``x`` that the call writes into the conv-state cache.

    Padded tokens (production ``slot_mapping == -1``) are not modeled.
    """
    if partials.dtype != torch.bfloat16 or partials.ndim != 3:
        raise ValueError("partials must be bf16 [W, T, H]")
    _, tokens, hidden = partials.shape
    if shared is not None and shared.shape != partials.shape:
        raise ValueError("shared must match partials")
    if residual.shape != (tokens, hidden):
        raise ValueError("residual must be [T, H]")
    kernel_size = conv_weight.shape[1]
    if conv_weight.shape != (hidden, kernel_size):
        raise ValueError("conv_weight must be [H, K]")
    if prior_taps.shape != (tokens, kernel_size - 1, hidden):
        raise ValueError("prior_taps must be [T, K-1, H]")

    published = partials.float()
    if shared is not None:
        published = published + shared.float()
    published = published.to(torch.bfloat16)
    conv_input = published.float().sum(dim=0).to(torch.bfloat16)

    taps = _resolve_taps(conv_input, prior_taps, positions, seq_idx)
    weight = conv_weight.float()
    conv = torch.zeros(tokens, hidden, dtype=torch.float32, device=partials.device)
    for tap in range(kernel_size):
        conv = conv + taps[:, tap].float() * weight[:, tap]
    short_conv = (conv + conv_input.float()).to(torch.bfloat16)
    residual_out = (residual.float() + short_conv.float()).to(torch.bfloat16)

    row = residual_out.float()
    inv_rms = torch.rsqrt(row.square().mean(dim=-1, keepdim=True) + eps)
    normed = (row * inv_rms * norm_weight.float()).to(torch.bfloat16)
    return normed, residual_out, conv_input


def _resolve_taps(
    conv_input: torch.Tensor,
    prior_taps: torch.Tensor,
    positions: torch.Tensor,
    seq_idx: torch.Tensor,
) -> torch.Tensor:
    """``[T, K, H]`` conv window per token, oldest tap first, current tap last."""
    tokens, history, _ = prior_taps.shape
    kernel_size = history + 1
    positions = positions.to(torch.int64)
    seq_idx = seq_idx.to(torch.int64)
    # One integer key per (request, position) of the tokens in this batch.
    span = int(positions.max().item()) + kernel_size + 1
    batch_keys = seq_idx * span + positions
    order = torch.argsort(batch_keys)
    sorted_keys = batch_keys[order]
    if tokens > 1 and bool((sorted_keys[1:] == sorted_keys[:-1]).any()):
        raise ValueError("two tokens share one (request, position)")

    window = []
    for tap in range(history):
        source_position = positions - (kernel_size - 1) + tap
        value = prior_taps[:, tap].clone()
        key = seq_idx * span + source_position
        found = torch.searchsorted(sorted_keys, key).clamp(max=tokens - 1)
        in_batch = (sorted_keys[found] == key) & (source_position >= 0)
        value[in_batch] = conv_input[order[found[in_batch]]]
        value[source_position < 0] = 0
        window.append(value)
    window.append(conv_input)
    return torch.stack(window, dim=1)
```

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/rs_sconv_ag_residual_rms_norm.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
