# NVFP4 activation quantization (`nvfp4_quant`) | ServingStudio Kernel Library

Quantize BF16 activation rows to packed NVFP4 values and E4M3 group scales.

Before an NVFP4 GEMM on B200, each activation row is quantized in 16-element groups to packed FP4 values with one E4M3 scale per group. scale_format selects where the group scales go. linear_e4m3 writes them row-major, as the TensorRT-LLM NVFP4 MoE GEMM reads them; both backends support it. swizzled_e4m3, vllm_cuda only, writes them in the 128-row by 4-group tile layout that vLLM's default B200 NVFP4 linear GEMM reads, padded with zero scales to whole tiles. vLLM's MoE path uses a global scale of 1 and its linear path the calibrated input scale; FlashInfer, on SGLang's path, also writes a per-token FP32 scale.

Category: Quantization. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=nvfp4_quant. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `GB/s = (num_tokens · hidden_size · (2 input + 1/2 packed FP4) + scale bytes) / time`
- `scale bytes = num_tokens · hidden_size/16 for linear_e4m3; ceil(num_tokens/128)·128 · ceil(hidden_size/64)·4 for swizzled_e4m3`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. vllm_cuda counts cvt_fp16_to_fp4_sf_major for linear_e4m3 and cvt_fp16_to_fp4 (not _sf_major) for swizzled_e4m3; flashinfer_cutedsl counts nvfp4_quantize, after one call that builds the kernel.

## Caveats

- The two backends scale differently, so their outputs are not interchangeable.
- GB/s counts BF16 input, packed FP4 output and the E4M3 scale tensor, including the zero-filled tile padding of swizzled_e4m3; global and per-token scales are excluded.
- swizzled_e4m3 walks ceil(num_tokens/128)·128 rows, so below 128 tokens the padded rows still cost scale writes.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_tokens` | number | tokens | Activation rows to quantize. |
| `hidden_size` | number | elements | Elements in each activation row. |
| `group_size` | number | elements | Elements sharing one FP4 scale. |
| `input_dtype` | dtype |  | Element type of the input activations. |
| `scale_format` | label |  | Encoding and layout of the output scales: linear_e4m3 or swizzled_e4m3. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_cuda`: vLLM's scaled_fp4_quant CUDA op, with row-major E4M3 scales for the TensorRT-LLM MoE or swizzled ones for the NVFP4 linear GEMM. dtypes `bf16`, compute capability 10.0 or newer, GPUs B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip. Source: https://github.com/vllm-project/vllm/blob/main/csrc/libtorch_stable/quantization/fp4/nvfp4_quant_kernels.cu
- `flashinfer_cutedsl`: FlashInfer nvfp4_quantize, CuTe DSL backend with per-token activation scales, as SGLang calls it. dtypes `bf16`, compute capability 10.0 or newer, GPUs B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip.

## Measured rows

748 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `flashinfer_cutedsl` | `bf16` | 68 |
| NVIDIA B200 | `vllm_cuda` | `bf16` | 680 |

## Used by these deployments

- [GLM-5.2 NVFP4: SGLang, TP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, TP and EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, TP and EP, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative`)
- [GLM-5.2 NVFP4: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_pp_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_pp_dsa_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe`)
- [GLM-5.3 NVFP4: vLLM, TP and EP, DFlash speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2.md) (`GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2`)
- [Inkling NVFP4: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Inkling-NVFP4/inkling_vllm_nvfp4_rel_attn_moe.md) (`Inkling-NVFP4/inkling_vllm_nvfp4_rel_attn_moe`)
- [Inkling Small NVFP4: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Inkling-Small-NVFP4/inkling_vllm_nvfp4_rel_attn_moe.md) (`Inkling-Small-NVFP4/inkling_vllm_nvfp4_rel_attn_moe`)
- [Qwen3.8 27B NVFP4: vLLM, one GPU](https://servingstudio.cs.washington.edu/models/Qwen3.8-27B-NVFP4/qwen35_vllm_nvfp4_gdn_dense.md) (`Qwen3.8-27B-NVFP4/qwen35_vllm_nvfp4_gdn_dense`)
- [Qwen3.8 Flash Next NVFP4: vLLM, one GPU, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative.md) (`Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/nvfp4_quant
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/nvfp4_quant/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/nvfp4_quant/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/nvfp4_quant.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
