# Static FP8 quantization (`static_fp8_quant`) | ServingStudio Kernel Library

Scale an activation by a fixed per-tensor scale and round it to FP8 E4M3.

A linear layer with static per-tensor FP8 activation scales quantizes its input before the GEMM. Under torch.compile vLLM's native QuantFP8 becomes one Inductor pointwise kernel, and an attention output gate in front of the projection fuses into it. Inputs are seeded random bf16 rows; the scale is a scalar fp32 tensor read on the device.

Category: Quantization. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=static_fp8_quant. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `y = clamp(x / scale, ±448) as E4M3, or clamp(x · σ(g) / scale, ±448) when gated`
- `GB/s = (2 + 2·gated + 1) · num_tokens · hidden_dim / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. The block is compiled once with a symbolic token count, and its first call, at 8192 tokens or at the row's token count when larger, picks the launch config the row then uses, as vLLM's profile run at max_num_batched_tokens does. Every launch of the compiled block is counted.

## Caveats

- GB/s counts the input, the gate and the output once each.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_tokens` | number | tokens | Rows quantized together. |
| `hidden_dim` | number | elements | Width of each row. |
| `gated` | label |  | Whether each row is first multiplied by the sigmoid of a gate row of the same width, as an attention output gate does. |
| `dtype` | dtype |  | Element type of the input (and of the gate). |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_inductor`: vLLM's QuantFP8.forward_native (static, per-tensor) compiled by Inductor into one Triton kernel. dtypes `bf16`, compute capability 8.9 or newer, GPUs H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip, H800-SXM5-80GB, H20-SXM-96GB, L40, L40S, L4. Source: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/quantization/input_quant_fp8.py

## Measured rows

136 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `vllm_inductor` | `bf16` | 136 |

## Used by these deployments

- [Qwen3.8 27B NVFP4: vLLM, one GPU](https://servingstudio.cs.washington.edu/models/Qwen3.8-27B-NVFP4/qwen35_vllm_nvfp4_gdn_dense.md) (`Qwen3.8-27B-NVFP4/qwen35_vllm_nvfp4_gdn_dense`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/static_fp8_quant
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/static_fp8_quant/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/static_fp8_quant/configs

## Torch reference

What the kind computes, as Torch code (`profiling/runners/elementwise/static_fp8_quant_reference.py` in ServingStudio Sim):

```python
"""Independent value reference for static per-tensor FP8 quantization."""

from typing import Any

FP8_E4M3_MAX = 448.0


def static_fp8_quant_reference(torch: Any, values: Any, gate: Any | None, scale: Any) -> Any:
    values = values.float()
    if gate is not None:
        values = values * torch.sigmoid(gate.float())
    return (values / scale.float()).clamp(-FP8_E4M3_MAX, FP8_E4M3_MAX).to(torch.float8_e4m3fn)


__all__ = ["static_fp8_quant_reference"]
```

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/static_fp8_quant.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
