# KV compression and store (`kv_compress_store`) | ServingStudio Kernel Library

Save partial states, then compress and store keys at compression boundaries.

A compressed KV cache is built incrementally: every token saves a partial state, and at every compress_ratio-th position a window of saved states is combined with softmax weights, RMS-normalized, rotated with RoPE, quantized and stored as one compressed key. row_positions and row_request_ids give each row's position and request; a row stores a key when position + 1 is divisible by compress_ratio. The 512-element attention cache uses a CuTe DSL kernel; the 128-element indexer cache uses Triton.

Category: Attention; group: Compressed sparse MLA. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=kv_compress_store. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `compressed = Σwindow state_key · softmax(state_score); key = RoPE(RMSNorm(compressed))`
- `N = len(row_positions); A = number of boundary rows; W = compress_ratio·(2 if compress_ratio = 4 else 1)`
- `TFLOPS = [A·(W·head_dim·5 + head_dim·8) + N·state_width] / time`
- `GB/s = [A·W·head_dim·2·4 + A·cache_row_bytes + N·(4 + 8 + 8 + 8) + N·(state_width·5·4 + 16)] / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. Every launch is counted: saving the partial states, then compressing and storing the boundary rows. The 512-element C128 path takes three launches; the C4 paths take two.

## Caveats

- state_width is 1024 for C4 with CuTe DSL and 512 otherwise; cache_row_bytes is 584 for CuTe DSL and 132 for Triton.
- Partial states and the RoPE table are generated values, not model activations.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `row_positions` | list | tokens | Absolute position of each input row. |
| `row_request_ids` | list | requests | Request index associated with each input row. |
| `state_block_table_width` | number | blocks | Partial-state block-table entries allocated per request. |
| `compress_ratio` | number | tokens | Source tokens represented by one compressed cache position. |
| `num_kv_heads` | number | heads | KV heads in the compressed cache. |
| `head_dim` | number | elements | Elements in each compressed key. |
| `rope_head_dim` | number | elements | Compressed-key elements rotated by RoPE. |
| `logical_block_size` | number | tokens | Source tokens in a logical cache block. |
| `rms_eps` | number | unitless | Stabilizer added to the RMS variance. |
| `state_dtype` | dtype |  | Element type of saved partial states. |
| `norm_dtype` | dtype |  | Element type of the normalization weight. |
| `cache_dtype` | label |  | Packed compressed-cache format. |
| `cache_layout` | label |  | Ordering of cache data and scales within a block. |
| `scale_format` | label |  | Encoding of the compressed-key scales. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_cutedsl`: vLLM's save_partial_states, then the CuTe DSL compress_norm_rope_store_cutedsl, writing the 512-element FP8 attention cache. dtypes `bf16`, `fp32`, KV dtypes `fp8_e4m3`, compute capability 8.9 or newer, GPUs H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip, H800-SXM5-80GB, H20-SXM-96GB, L40, L40S, L4. Source: https://github.com/vllm-project/vllm/blob/main/vllm/models/deepseek_v4/nvidia/ops/sparse_attn_compress_cutedsl.py
- `vllm_triton`: vLLM's save_partial_states, then the Triton compress_norm_rope_store_triton, writing the 128-element FP8 indexer cache. dtypes `bf16`, `fp32`, KV dtypes `fp8_e4m3`, compute capability 8.9 or newer, GPUs H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip, H800-SXM5-80GB, H20-SXM-96GB, L40, L40S, L4. Source: https://github.com/vllm-project/vllm/blob/main/vllm/models/deepseek_v4/common/ops/fused_compress_quant_cache.py

## Measured rows

361 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA H200 | `vllm_cutedsl` | `fp32` | 242 |
| NVIDIA H200 | `vllm_triton` | `fp32` | 119 |

## Used by these deployments

- [DeepSeek V4 Flash 0731: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm.md) (`DeepSeek-V4-Flash-0731/deepseek_v4_vllm`)
- [DeepSeek V4 Flash 0731: vLLM, DP attention, EP, parallel streams serialized](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams.md) (`DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/kv_compress_store
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/kv_compress_store/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/kv_compress_store/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/kv_compress_store.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
