# FP8 block quantization (`fp8_block_quant`) | ServingStudio Kernel Library

Quantize BF16 expert inputs to FP8 E4M3 in 128-element blocks.

The input step of the FP8 blockscale grouped GEMM (fp8_blockscale_grouped_gemm): each routed row is split into 128-element blocks, and each block is scaled to FP8 E4M3 with one FP32 scale. The kernel walks the rows expert by expert, so the measurement splits num_tokens into num_problems contiguous segments of near-equal size.

Category: Quantization. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=fp8_block_quant. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `scale = max(abs(block)) / 448, or 1 for an all-zero block`
- `FP8 block = FP8_E4M3(block / scale)`
- `GB/s = num_tokens · (hidden_size · (BF16 bytes + FP8 bytes) + hidden_size / 128 · FP32 bytes) / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. Only the grouped scale_1x128_kernel launch is counted. FlashInfer does not expose this launch on its own, so a local binding launches the vendored TensorRT-LLM template directly.

## Caveats

- Segments are near-equal; real routing gives uneven segments for the same num_tokens and num_problems.
- GB/s counts logical input, output and scale bytes, without the padding of the grouped scale layout.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_tokens` | number | tokens | Routed token rows on this GPU. |
| `hidden_size` | number | elements | Elements in each token row. |
| `num_problems` | number | experts | Local expert segments in the grouped input. |
| `input_dtype` | dtype |  | Element type of the input rows. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `flashinfer_trtllm`: TensorRT-LLM's grouped scale_1x128_kernel from the FlashInfer wheel, launched alone. dtypes `bf16`, SM targets `sm_90a`, GPUs H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, H800-SXM5-80GB, H20-SXM-96GB.

## Measured rows

748 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA H200 | `flashinfer_trtllm` | `bf16` | 748 |

## Used by these deployments

- [GLM-5.2 FP8: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-FP8/glm52_vllm_dsa_moe.md) (`GLM-5.2-FP8/glm52_vllm_dsa_moe`)
- [Qwen3 235B-A22B FP8: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_vllm_moe_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-FP8/qwen3_vllm_moe_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B Thinking 2507 FP8: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_vllm_moe_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_vllm_moe_dp_attn_ep_ffn`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/fp8_block_quant
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/fp8_block_quant/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/fp8_block_quant/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/fp8_block_quant.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
