# Compressed sparse MLA decode (`compressed_sparse_mla_decode`) | ServingStudio Kernel Library

Attend from each decode query to sliding-window and optional compressed keys.

Compressed sparse MLA in decode: each query row attends to its sliding-window keys and, in compressed layers, to the compressed keys the indexer selected, in one FlashMLA call. The two sources have separate FP8 page sets and padded index arrays; swa_valid_counts and extra_valid_counts give the valid entries per row. As in serving, the call is captured in a CUDA graph and replayed.

Category: Attention; group: Compressed sparse MLA. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=compressed_sparse_mla_decode. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `O = softmax(q · K_selectedᵀ / √head_dim) · V_selected, per query head`
- `S = Σ(swa_valid_counts) + Σ(extra_valid_counts); R = len(swa_valid_counts)`
- `TFLOPS = 2·num_heads·(head_dim + value_dim)·S / time`
- `GB/s = [R·num_heads·head_dim·2 + S·(584 + 4) + R·4·(2 if compress_ratio > 1 else 1) + R·num_heads·(value_dim·2 + 4) + num_heads·4] / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. The launches of each graph replay are counted, after three warm-up replays. Graph capture and metadata construction are excluded; in reused planner mode one sparse launch also runs before capture.

## Caveats

- Queries are zero and cache rows hold patterned values with contiguous valid indices; FlashMLA's work does not depend on the values.
- GB/s counts the valid selected rows and indices, not the padding of the index arrays.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `swa_valid_counts` | list | tokens | Valid sliding-window keys for each flattened decode query. |
| `extra_valid_counts` | list | tokens | Valid selected compressed keys for each decode query. |
| `num_heads` | number | heads | Query heads on this GPU. |
| `num_kv_heads` | number | heads | KV heads in the cache. |
| `head_dim` | number | elements | Elements in each query and key head. |
| `value_dim` | number | elements | Elements in each output head. |
| `swa_window` | number | tokens | Maximum sliding-window keys per query. |
| `extra_index_capacity` | number | tokens | Allocated selected-key indices per query. |
| `compress_ratio` | number | tokens | Source tokens represented by one compressed cache position. |
| `q_dtype` | dtype |  | Query element type. |
| `cache_dtype` | dtype |  | Cache element type. |
| `output_dtype` | dtype |  | Output element type. |
| `planner_mode` | label |  | Whether a sparse launch runs before graph capture. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_flashmla_fp8_cudagraph`: FlashMLA flash_mla_with_kvcache, as vLLM calls it, over FP8 sliding-window and selected-key caches, replayed from a CUDA graph. dtypes `bf16`, KV dtypes `fp8_e4m3`, SM targets `sm_100f`, `sm_90a`, GPUs H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip, H800-SXM5-80GB, H20-SXM-96GB. Source: https://github.com/deepseek-ai/FlashMLA

## Measured rows

386 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA H200 | `vllm_flashmla_fp8_cudagraph` | `bf16` | 386 |

## Used by these deployments

- [DeepSeek V4 Flash 0731: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm.md) (`DeepSeek-V4-Flash-0731/deepseek_v4_vllm`)
- [DeepSeek V4 Flash 0731: vLLM, DP attention, EP, parallel streams serialized](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams.md) (`DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/compressed_sparse_mla_decode
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/compressed_sparse_mla_decode/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/compressed_sparse_mla_decode/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/compressed_sparse_mla_decode.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
