# Clamped SwiGLU (`clamped_swiglu`) | ServingStudio Kernel Library

Clamp the gate and up projections, then multiply SiLU of the gate by the up projection.

Routed and shared experts apply this activation to the output of their gate-up projection. Each input row holds a gate half and an up half, each hidden_dim wide; the gate is capped at 10 and the up half clamped to \[−10, 10\] before the SiLU product. The measurement uses seeded random bf16 rows.

Category: MoE; group: Expert compute. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=clamped_swiglu. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `y = SiLU(min(gate, 10)) · clamp(up, −10, 10)`
- `GB/s = 6 · num_rows · hidden_dim / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. Three untimed calls and a comparison with the PyTorch reference precede capture. Every launch of vLLM's callable is counted.

## Caveats

- GB/s counts the two bf16 input halves and one bf16 output, without counting any temporary traffic.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_rows` | number | rows | Rows entering the activation: token-expert pairs for routed experts, tokens for the shared expert. |
| `hidden_dim` | number | elements | Width of the gate half and of the up half of each row. |
| `dtype` | dtype |  | Element type of the input and output. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_inductor`: vLLM's swiglu_limit_func without topk_ids, which dispatches to its CUDA silu_and_mul_with_clamp op. dtypes `bf16`, any CUDA GPU. Source: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/fused_moe/utils.py

## Measured rows

68 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA H200 | `vllm_inductor` | `bf16` | 68 |

## Used by these deployments

- [DeepSeek V4 Flash 0731: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm.md) (`DeepSeek-V4-Flash-0731/deepseek_v4_vllm`)
- [DeepSeek V4 Flash 0731: vLLM, DP attention, EP, parallel streams serialized](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams.md) (`DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/clamped_swiglu
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/clamped_swiglu/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/clamped_swiglu/configs

## Torch reference

What the kind computes, as Torch code (`profiling/runners/moe/clamped_swiglu_reference.py` in ServingStudio Sim):

```python
"""Independent value reference for DeepSeek's clamped SwiGLU."""

from typing import Any

CLAMP_LIMIT = 10.0


def clamped_swiglu_reference(torch: Any, input_tensor: Any) -> Any:
    if input_tensor.ndim != 2 or input_tensor.shape[1] % 2:
        raise ValueError("input_tensor must have shape [rows, 2 * hidden_dim]")
    if input_tensor.dtype is not torch.bfloat16:
        raise TypeError("input_tensor must use bfloat16")
    if not input_tensor.is_contiguous() or any(size <= 0 for size in input_tensor.shape):
        raise ValueError("input_tensor must be non-empty contiguous storage")
    gate, up = input_tensor.chunk(2, dim=1)
    gate = torch.clamp(gate, max=CLAMP_LIMIT)
    up = torch.clamp(up, min=-CLAMP_LIMIT, max=CLAMP_LIMIT)
    return torch.nn.functional.silu(gate) * up


__all__ = ["CLAMP_LIMIT", "clamped_swiglu_reference"]
```

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/clamped_swiglu.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
