# Packed KV cache gather (`packed_kv_cache_gather`) | ServingStudio Kernel Library

Gather and dequantize paged FP8 keys into a BF16 prefill workspace.

Before compressed sparse MLA prefill, vLLM gathers keys from the paged, packed FP8 cache into a contiguous BF16 workspace, dequantizing as it copies: either the compressed keys or a sliding-window suffix. seq_lens gives each request's cache length; gather_lens, when given, limits the copy to the trailing rows. The measurement reverses the physical block order and fills the cache with patterned FP8 and BF16 values, so page lookup and dequantization are both exercised.

Category: Attention; group: Compressed sparse MLA. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=packed_kv_cache_gather. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `G = Σi (gather_lens[i] if supplied else seq_lens[i]); R = len(seq_lens)`
- `TFLOPS = G·448 / time`
- `GB/s = [G·(584 + 4 + 512·2) + 4·R·(2 if gather_lens else 1)] / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. One call with an output check and three warm-up calls run before the capture; every launch of the gather call is counted.

## Caveats

- TFLOPS counts 448 dequantization operations per gathered row; there is no matrix multiply.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `seq_lens` | list | tokens | Cache sequence length per request. |
| `gather_lens` | list | tokens | Suffix length per request; an empty tuple gathers full sequences. |
| `workspace_rows` | number | rows | Output rows allocated per request. |
| `block_table_width` | number | blocks | Block-table entries allocated per request. |
| `block_size` | number | tokens | Cache tokens in each physical block. |
| `offset` | number | rows | First output row written within each request's workspace. |
| `num_kv_heads` | number | heads | KV heads in the packed cache. |
| `head_dim` | number | elements | Output elements per gathered key. |
| `fp8_dim` | number | elements | FP8 elements in each packed key. |
| `quant_group_size` | number | elements | FP8 elements sharing one scale. |
| `cache_dtype` | label |  | Packed cache storage format. |
| `output_dtype` | dtype |  | Element type of the gathered keys. |
| `cache_layout` | label |  | Ordering of packed cache data and scales within a block. |
| `scale_format` | label |  | Encoding of the packed FP8 scales. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_cutedsl`: vLLM's dequantize_and_gather_k_cache, dispatched to its CuTe DSL kernel for the packed FP8 cache. dtypes `bf16`, KV dtypes `fp8_e4m3`, compute capability 8.9 or newer, GPUs H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip, H800-SXM5-80GB, H20-SXM-96GB, L40, L40S, L4. Source: https://github.com/vllm-project/vllm/blob/main/vllm/models/deepseek_v4/nvidia/ops/dequant_gather_k_cutedsl.py

## Measured rows

2 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA H200 | `vllm_cutedsl` | `bf16` | 2 |

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/packed_kv_cache_gather
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/packed_kv_cache_gather/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/packed_kv_cache_gather/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/packed_kv_cache_gather.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
