# Non-causal rectangular attention (`flashinfer_attn_rect`) | ServingStudio Kernel Library

Attend every query token over every key and value token in one request.

Attention without a causal mask: every query token sees every key and value token of its request. The DFlash2 draft layer uses it so each drafted position sees the whole context. The measurement is one request with q_len queries and kv_len contiguous keys and values; the two lengths are independent.

Category: Attention; group: MHA / GQA. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=flashinfer_attn_rect. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `O = softmax(Q · Kᵀ / √head_dim) · V, per query head`
- `TFLOPS = 4 · q_len · kv_len · num_qo_heads · head_dim / time`
- `GB/s = bytes of Q, K, V and O / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. Planning and input construction are not timed; every launch of the attention call is counted. fa2 first times the default plan and fixed KV split sizes of 1,024 to 16,384 tokens below kv_len, then measures the fastest.

## Caveats

- One request with contiguous K and V is measured, not a batch of requests of different lengths.
- The fp8 path needs fp8 queries and KV with a non-fp8 output, so bf16 queries over fp8 KV are not measured.
- cudnn measures no fp8.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_qo_heads` | number | heads | Query and output heads on this GPU. |
| `num_kv_heads` | number | heads | KV heads on this GPU. |
| `head_dim` | number | elements | Dimension of each query, key and value head. |
| `q_dtype` | dtype |  | Query element type. |
| `kv_dtype` | dtype |  | Key and value element type. |
| `o_dtype` | dtype |  | Output element type. |
| `q_len` | number | tokens | Query tokens in the measured request. |
| `kv_len` | number | tokens | Key and value tokens in the measured request. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `fa2`: FlashInfer FA2 non-causal ragged attention with a measured KV split search. dtypes `bf16`, KV dtypes `bf16`, `fp8_e4m3`, any CUDA GPU.
- `fa3`: FlashInfer's non-causal ragged wrapper with FA3 kernels. dtypes `bf16`, `fp8_e4m3`, KV dtypes `bf16`, `fp8_e4m3`, any CUDA GPU.
- `cudnn`: PyTorch scaled_dot_product_attention pinned to cuDNN for non-causal attention. dtypes `bf16`, KV dtypes `bf16`, any CUDA GPU.

## Measured rows

3,969 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `fa2` | `bf16` | 3,969 |

## Used by these deployments

- [GLM-5.3 NVFP4: vLLM, TP and EP, DFlash speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2.md) (`GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/flashinfer_attn_rect
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/flashinfer_attn_rect/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/flashinfer_attn_rect/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/flashinfer_attn_rect.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
