# FP8 per-token-group quantization (`fp8_per_token_group_quant`) | ServingStudio Kernel Library

Quantize BF16 activation rows to FP8 E4M3 with a scale per 128-element group.

vLLM's blockwise FP8 path quantizes activations before dense projections and before the MoE gate/up GEMM. Each row is split into group_size-element groups, and each group gets one scale rounded up to a power of two (UE8M0). scale_format picks the layout the consuming kernel reads: ue8m0_column_major stores FP32 scales in a column-major \[token, group\] layout for Hopper DeepGEMM, ue8m0_row_major stores them row-major for the Blackwell FP8 block-scale MoE, and ue8m0_packed_int32 packs four one-byte scales per int32 for Blackwell DeepGEMM.

Category: Quantization. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=fp8_per_token_group_quant. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `G = hidden_size / group_size; ue8m0_packed_int32: G = ⌈hidden_size / group_size / 4⌉`
- `GB/s = (num_tokens · hidden_size · (BF16 bytes + FP8 bytes) + num_tokens · G · 4 scale bytes) / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. Only the quantization kernel is counted: per_token_group_quant_8bit_packed_register_kernel for ue8m0_packed_int32, per_token_group_quant_8bit_kernel otherwise.

## Caveats

- ue8m0_packed_int32 needs group_size = 128: vLLM compiles its packed kernel for that width only. The other formats take any group_size that divides hidden_size.
- GB/s counts logical BF16 input, FP8 output and scale bytes, not physical memory transactions.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_tokens` | number | tokens | Activation rows to quantize. |
| `hidden_size` | number | elements | Elements in each activation row. |
| `group_size` | number | elements | Elements sharing one quantization scale. |
| `input_dtype` | dtype |  | Element type of the input activations. |
| `scale_format` | label |  | Encoding and layout of the output scales. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_cuda`: vLLM's per_token_group_fp8_quant CUDA op with UE8M0 scales, or per_token_group_fp8_quant_packed for the packed int32 layout. dtypes `bf16`, any CUDA GPU. Source: https://github.com/vllm-project/vllm/blob/main/csrc/libtorch_stable/quantization/w8a8/fp8/per_token_group_quant.cu

## Measured rows

1,224 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `vllm_cuda` | `bf16` | 612 |
| NVIDIA H200 | `vllm_cuda` | `bf16` | 612 |

## Used by these deployments

- [DeepSeek V4 Flash 0731: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm.md) (`DeepSeek-V4-Flash-0731/deepseek_v4_vllm`)
- [DeepSeek V4 Flash 0731: vLLM, DP attention, EP, parallel streams serialized](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams.md) (`DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams`)
- [GLM-5.2 FP8: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-FP8/glm52_vllm_dsa_moe.md) (`GLM-5.2-FP8/glm52_vllm_dsa_moe`)
- [GLM-5.3-Flash: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe`)
- [GLM-5.3-Flash: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe`)
- [GLM-5.3-Flash: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe`)
- [MiMo V2.6 Flash RL: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/MiMo-V2.6-Flash-RL/mimo_v2_vllm_fp8_swa_moe.md) (`MiMo-V2.6-Flash-RL/mimo_v2_vllm_fp8_swa_moe`)
- [Qwen3 235B-A22B FP8: FFN side of an attention–FFN split](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_fp8_ffn_moe.md) (`Qwen3-235B-A22B-FP8/qwen3_fp8_ffn_moe`)
- [Qwen3 235B-A22B FP8: DP attention, EP MoE, DeepGEMM experts](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_moe_fp8_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-FP8/qwen3_moe_fp8_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B FP8: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_vllm_moe_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-FP8/qwen3_vllm_moe_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B Thinking 2507 FP8: DP attention, EP MoE, DeepGEMM experts](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_moe_fp8_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_moe_fp8_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B Thinking 2507 FP8: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_vllm_moe_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_vllm_moe_dp_attn_ep_ffn`)
- [Qwen3 Coder 480B-A35B: FFN side of an attention–FFN split](https://servingstudio.cs.washington.edu/models/Qwen3-Coder-480B-A35B-Instruct/qwen3_fp8_ffn_moe.md) (`Qwen3-Coder-480B-A35B-Instruct/qwen3_fp8_ffn_moe`)
- [Qwen3.6 35B-A3B FP8: Single GPU](https://servingstudio.cs.washington.edu/models/Qwen3.6-35B-A3B-FP8/qwen36_local.md) (`Qwen3.6-35B-A3B-FP8/qwen36_local`)
- [Qwen3.8 Flash Next NVFP4: vLLM, one GPU, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative.md) (`Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/fp8_per_token_group_quant
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/fp8_per_token_group_quant/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/fp8_per_token_group_quant/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/fp8_per_token_group_quant.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
