# Causal prefill attention (`flashinfer_attn_prefill`) | ServingStudio Kernel Library

Attend a block of new query tokens over its preceding context and itself.

The attention step of prefill and chunked prefill for MHA and GQA models. A request brings append_len new tokens; each attends causally to the prefix_len tokens already cached and to the new tokens before it. The measurement is one request with contiguous K and V, so q_len = append_len and kv_len = prefix_len + append_len.

Category: Attention; group: MHA / GQA. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=flashinfer_attn_prefill. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `O = softmax(Q · Kᵀ / √head_dim, causal mask) · V, per query head`
- `q_len = append_len, kv_len = prefix_len + append_len`
- `TFLOPS = 2 · append_len · (2·prefix_len + append_len) · num_qo_heads · head_dim / time`
- `GB/s = bytes of Q, K, V and O / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. Planning and input construction are not timed; every launch of the attention call is counted. fa2 first times the default plan and fixed KV split sizes of 1,024 to 16,384 tokens below kv_len, then measures the fastest.

## Caveats

- One request with contiguous K and V is measured, not a batch of requests of different lengths or a paged cache.
- The fp8 path needs fp8 queries and KV with a non-fp8 output, so bf16 queries over an fp8 cache are not measured.
- cudnn measures only prefix_len = 0 and no fp8.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_qo_heads` | number | heads | Query and output heads on this GPU. |
| `num_kv_heads` | number | heads | KV heads on this GPU. |
| `head_dim` | number | elements | Dimension of each query, key and value head. |
| `q_dtype` | dtype |  | Query element type. |
| `kv_dtype` | dtype |  | Key and value element type. |
| `o_dtype` | dtype |  | Output element type. |
| `prefix_len` | number | tokens | Cached tokens preceding the new query block. |
| `append_len` | number | tokens | New query and KV tokens in this append. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `fa2`: FlashInfer FA2 ragged prefill with a measured KV split search. dtypes `bf16`, KV dtypes `bf16`, `fp8_e4m3`, any CUDA GPU.
- `fa3`: FlashInfer's ragged prefill wrapper with FA3 kernels. dtypes `bf16`, `fp8_e4m3`, KV dtypes `bf16`, `fp8_e4m3`, any CUDA GPU.
- `cudnn`: PyTorch scaled_dot_product_attention pinned to cuDNN for fresh causal prefill. dtypes `bf16`, KV dtypes `bf16`, any CUDA GPU.

## Measured rows

2,835 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA H200 | `fa2` | `bf16` | 1,080 |
| NVIDIA H200 | `fa3` | `bf16` | 1,080 |
| NVIDIA H200 | `fa3` | `fp8_e4m3` | 675 |

## Used by these deployments

- [Llama 3.1 8B: Single GPU](https://servingstudio.cs.washington.edu/models/Llama-3.1-8B/llama3_dense.md) (`Llama-3.1-8B/llama3_dense`)
- [Llama 3.1 8B: TP](https://servingstudio.cs.washington.edu/models/Llama-3.1-8B/llama3_dense_tp.md) (`Llama-3.1-8B/llama3_dense_tp`)
- [Llama 3.1 8B: DP attention, TP FFN](https://servingstudio.cs.washington.edu/models/Llama-3.1-8B/llama3_dp_attn_tp_ffn.md) (`Llama-3.1-8B/llama3_dp_attn_tp_ffn`)
- [Qwen3 235B-A22B FP8: Attention side of an attention–FFN split](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_attn_tp.md) (`Qwen3-235B-A22B-FP8/qwen3_attn_tp`)
- [Qwen3 235B-A22B FP8: DP attention, EP MoE, DeepGEMM experts](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_moe_fp8_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-FP8/qwen3_moe_fp8_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B FP8: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-FP8/qwen3_vllm_moe_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-FP8/qwen3_vllm_moe_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B Thinking 2507 FP8: DP attention, EP MoE, DeepGEMM experts](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_moe_fp8_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_moe_fp8_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B Thinking 2507 FP8: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_vllm_moe_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B-Thinking-2507-FP8/qwen3_vllm_moe_dp_attn_ep_ffn`)
- [Qwen3 235B-A22B: Attention side of an attention–FFN split](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B/qwen3_attn_tp.md) (`Qwen3-235B-A22B/qwen3_attn_tp`)
- [Qwen3 235B-A22B: DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Qwen3-235B-A22B/qwen3_moe_dp_attn_ep_ffn.md) (`Qwen3-235B-A22B/qwen3_moe_dp_attn_ep_ffn`)
- [Qwen3 Coder 480B-A35B: Attention side of an attention–FFN split](https://servingstudio.cs.washington.edu/models/Qwen3-Coder-480B-A35B-Instruct/qwen3_attn_tp.md) (`Qwen3-Coder-480B-A35B-Instruct/qwen3_attn_tp`)
- [Qwen3.6 35B-A3B FP8: Single GPU](https://servingstudio.cs.washington.edu/models/Qwen3.6-35B-A3B-FP8/qwen36_local.md) (`Qwen3.6-35B-A3B-FP8/qwen36_local`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/flashinfer_attn_prefill
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/flashinfer_attn_prefill/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/flashinfer_attn_prefill/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/flashinfer_attn_prefill.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
