# SiLU and mul (`silu_and_mul`) | ServingStudio Kernel Library

Multiply SiLU of the gate half of each row by its up half.

A gated MLP applies this activation to the output of its fused gate-up projection: each input row holds a gate half and an up half, each hidden_dim wide, and the output row is hidden_dim wide. Under torch.compile vLLM's native SiluAndMul becomes one Inductor pointwise kernel; a following quantization of the down projection's input stays a separate kernel when it is a custom op. Inputs are seeded random bf16 rows.

Category: Other. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=silu_and_mul. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `y = SiLU(gate) · up, computed in fp32`
- `GB/s = 6 · num_tokens · hidden_dim / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. The block is compiled once with a symbolic token count, and its first call, at 8192 tokens or at the row's token count when larger, picks the launch config the row then uses, as vLLM's profile run at max_num_batched_tokens does. Every launch of the compiled block is counted. vllm_cuda: the CUDA custom op writes into a preallocated output and is one launch.

## Caveats

- GB/s counts the two input halves and the output once each.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_tokens` | number | tokens | Rows entering the activation. |
| `hidden_dim` | number | elements | Width of the gate half and of the up half of each row. |
| `dtype` | dtype |  | Element type of the input and output. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_inductor`: vLLM's SiluAndMul.forward_native compiled by Inductor into one Triton kernel. dtypes `bf16`, compute capability 8.0 or newer, GPUs A100-SXM4-40GB, A100-SXM4-80GB, A100-PCIe-40GB, A100-PCIe-80GB, H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip, H800-SXM5-80GB, H20-SXM-96GB, A40, L40, L40S, L4. Source: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/activation.py
- `vllm_cuda`: vLLM's SiluAndMul.forward_cuda: the torch.ops._C.silu_and_mul custom op. dtypes `bf16`, compute capability 8.0 or newer, GPUs A100-SXM4-40GB, A100-SXM4-80GB, A100-PCIe-40GB, A100-PCIe-80GB, H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip, H800-SXM5-80GB, H20-SXM-96GB, A40, L40, L40S, L4. Source: https://github.com/vllm-project/vllm/blob/main/csrc/libtorch_stable/activation_kernels.cu

## Measured rows

136 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `vllm_cuda` | `bf16` | 68 |
| NVIDIA B200 | `vllm_inductor` | `bf16` | 68 |

## Used by these deployments

- [Qwen3.8 27B NVFP4: vLLM, one GPU](https://servingstudio.cs.washington.edu/models/Qwen3.8-27B-NVFP4/qwen35_vllm_nvfp4_gdn_dense.md) (`Qwen3.8-27B-NVFP4/qwen35_vllm_nvfp4_gdn_dense`)
- [Qwen3.8 Flash Next NVFP4: vLLM, one GPU, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative.md) (`Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/silu_and_mul
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/silu_and_mul/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/silu_and_mul/configs

## Torch reference

What the kind computes, as Torch code (`profiling/runners/elementwise/silu_and_mul_reference.py` in ServingStudio Sim):

```python
"""Independent value reference for SiLU-and-mul."""

from typing import Any


def silu_and_mul_reference(torch: Any, input_tensor: Any) -> Any:
    gate, up = input_tensor.float().chunk(2, dim=-1)
    return (torch.nn.functional.silu(gate) * up).to(input_tensor.dtype)


__all__ = ["silu_and_mul_reference"]
```

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/silu_and_mul.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
