# Compressed sparse MLA prefill (`compressed_sparse_mla_prefill`) | ServingStudio Kernel Library

Attend from prefill queries to sliding-window and optional compressed keys.

Compressed sparse MLA in prefill: each new query token attends to a 128-token sliding window and, in compressed layers, to its selected compressed keys, all from a BF16 workspace gathered beforehand. The measurement builds request-local indices with the compressed keys spread evenly across the visible context, and runs FlashMLA on groups of up to four requests, as vLLM does.

Category: Attention; group: Compressed sparse MLA. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=compressed_sparse_mla_prefill. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `O = softmax(q · K_selectedᵀ · softmax_scale) · V_selected, per query head`
- `Q = total query tokens; V = sum of valid index lengths; P = total padded index entries`
- `TFLOPS = 2·64·V·(512 + 512) / time`
- `GB/s = [2·Q·64·512 + 4·P + 4·Q + 2·V·512 + 2·Q·64·512 + 8·Q·64] / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. One FlashMLA launch per group of up to four requests is counted. The output checks and cache construction are excluded.

## Caveats

- Selected keys are spread evenly rather than chosen by an indexer.
- Gathering the cache and combining the indices are separate kinds and are not included.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `query_context_pairs` | list |  | Per-request (query tokens, context tokens) in the prefill batch. |
| `max_model_len` | number | tokens | Maximum context length allowed by the model. |
| `max_num_batched_tokens` | number | tokens | Maximum query tokens allowed in the batch. |
| `prefill_chunk_size` | number | requests | Requests in each FlashMLA call. |
| `compress_ratio` | number | tokens | Source tokens represented by one compressed cache position. |
| `window_size` | number | tokens | Sliding-window keys available per query. |
| `selected_k` | number | tokens | Maximum selected compressed keys per query. |
| `selected_index_pattern` | label |  | How selected and sliding-window indices are combined. |
| `num_heads` | number | heads | Query heads on this GPU. |
| `num_kv_heads` | number | heads | KV heads in the gathered cache. |
| `head_dim` | number | elements | Elements in each query and key head. |
| `value_dim` | number | elements | Elements in each output head. |
| `softmax_scale` | number | multiplier | Multiplier applied to attention logits. |
| `q_dtype` | dtype |  | Query element type. |
| `cache_dtype` | dtype |  | Gathered cache element type. |
| `index_dtype` | label |  | Selected-index element type. |
| `output_dtype` | dtype |  | Output element type. |
| `cache_layout` | label |  | Arrangement of gathered cache rows. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_flashmla_bf16`: vLLM's flash_mla_sparse_fwd calls BF16 FlashMLA on combined request-local indices. dtypes `bf16`, KV dtypes `bf16`, SM targets `sm_100f`, `sm_90a`, GPUs H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip, H800-SXM5-80GB, H20-SXM-96GB. Source: https://github.com/vllm-project/vllm/blob/main/vllm/v1/attention/ops/flashmla.py

## Measured rows

2,259 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA H200 | `vllm_flashmla_bf16` | `bf16` | 2,259 |

## Used by these deployments

- [DeepSeek V4 Flash 0731: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm.md) (`DeepSeek-V4-Flash-0731/deepseek_v4_vllm`)
- [DeepSeek V4 Flash 0731: vLLM, DP attention, EP, parallel streams serialized](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams.md) (`DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/compressed_sparse_mla_prefill
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/compressed_sparse_mla_prefill/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/compressed_sparse_mla_prefill/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/compressed_sparse_mla_prefill.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
