# Indexer key cache append (`dsa_index_cache_append`) | ServingStudio Kernel Library

Quantize new DSA index keys and write them into a paged FP8 cache.

The DSA indexer stores each new index key in a paged FP8 cache for later scoring. Within a page, keys and their scales sit in separate planes, and a slot mapping scatters tokens across pages. torch and vllm_cuda quantize BF16 keys as given; SGLang's fused call also applies LayerNorm and RoPE to the keys before storing them.

Category: Attention; group: DSA. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=dsa_index_cache_append. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `cache[slot] = FP8(key / scale), one scale per quant_block_size elements`
- `groups = index_dim / quant_block_size`
- `B = num_tokens · (2 · index_dim + 8 + index_dim + 4 · groups)`
- `GB/s = B / time; SGLang adds 4 · 64 bytes per token of RoPE-table reads`

## How it is measured

torch is timed with CUDA events: five warm-up calls, then a loop of back-to-back calls, taking the median of three runs, so the gaps between its launches are included. vllm_cuda and sglang_fused_norm_rope_store use CUPTI kernel time with the L2 cache flushed before each launch: vllm_cuda counts only indexer_k_quant_and_cache_kernel, and SGLang counts every launch of its call after one setup call.

## Caveats

- torch and vllm_cuda round scales to powers of two (UE8M0); SGLang keeps FP32 scales and also runs LayerNorm and RoPE, so its rows cover more work.
- torch rows are warm-cache elapsed time and the others are cold-cache kernel time, so the backends are not directly comparable.
- GB/s counts logical key, slot, scale and cache bytes, not torch intermediates or physical transactions.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_tokens` | number | tokens | New index-key tokens written to the cache. |
| `index_dim` | number | elements | Elements in each index key. |
| `block_size` | number | tokens | Tokens stored in each cache page. |
| `quant_block_size` | number | elements | Key elements sharing one FP8 scale. |
| `input_dtype` | dtype |  | Element type of the input index keys. |
| `cache_dtype` | dtype |  | Element type of the cached index keys. |
| `scale_format` | label |  | Encoding of each quantization scale. |
| `cache_format` | label |  | Page-planar layout of cached keys and scales. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `torch`: PyTorch quantization and scattered page writes as a multi-launch composite. dtypes `bf16`, KV dtypes `fp8_e4m3`, any CUDA GPU.
- `vllm_cuda`: vLLM indexer_k_quant_and_cache: one CUDA launch for FP8 quantization and page writes. dtypes `bf16`, KV dtypes `fp8_e4m3`, any CUDA GPU. Source: https://github.com/vllm-project/vllm/blob/main/csrc/libtorch_stable/cache_kernels.cu
- `sglang_fused_norm_rope_store`: SGLang fused_k_indexer_norm_rope_store: LayerNorm, RoPE, FP8 quantization and page writes in one call. dtypes `bf16`, KV dtypes `fp8_e4m3`, any CUDA GPU. Source: https://github.com/sgl-project/sglang/blob/main/python/sglang/kernels/ops/attention/dsa/indexer_k.py

## Measured rows

279 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `sglang_fused_norm_rope_store` | `bf16` | 70 |
| NVIDIA B200 | `vllm_cuda` | `bf16` | 70 |
| NVIDIA H200 | `vllm_cuda` | `bf16` | 139 |

## Used by these deployments

- [GLM-5.2 FP8: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-FP8/glm52_vllm_dsa_moe.md) (`GLM-5.2-FP8/glm52_vllm_dsa_moe`)
- [GLM-5.2 NVFP4: SGLang, TP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, TP and EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, TP and EP, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative`)
- [GLM-5.2 NVFP4: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_pp_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_pp_dsa_moe`)
- [GLM-5.2: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2/glm52_vllm_dsa_moe.md) (`GLM-5.2/glm52_vllm_dsa_moe`)
- [GLM-5.3 NVFP4: vLLM, TP and EP, DFlash speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2.md) (`GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/dsa_index_cache_append
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/dsa_index_cache_append/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/dsa_index_cache_append/configs

## Torch reference

What the kind computes, as Torch code (`profiling/runners/attention/dsa_index_cache_append_reference.py` in ServingStudio Sim):

```python
"""Torch semantics for the DSA index-key quantization and cache append."""

from __future__ import annotations

import torch

_SUPPORTED_INPUT_DTYPES = (torch.bfloat16, torch.float16, torch.float32)
_SUPPORTED_SCALE_FORMAT = "ue8m0"
_SUPPORTED_CACHE_FORMAT = "page_planar_fp8_fp32_scale"
_FP8_E4M3_MAX = 448.0
_AMAX_FLOOR = 1e-4


def dsa_index_cache_append_reference(
    k: torch.Tensor,
    cache: torch.Tensor,
    slot_mapping: torch.Tensor,
    *,
    quant_block_size: int = 128,
    scale_format: str = _SUPPORTED_SCALE_FORMAT,
    cache_format: str = _SUPPORTED_CACHE_FORMAT,
) -> torch.Tensor:
    """Quantize index keys into a page-planar FP8-key/FP32-scale cache.

    Reduction and scale calculations use FP32. The cache stores all FP8 key
    bytes for a page first, followed by raw FP32 scale bytes. The function
    mutates and returns only ``cache``; padded K rows beyond ``slot_mapping``
    are ignored.

    Source boundary: vLLM v0.23.0 commit
    0fc695fc6d1d82e9a5ac6835ac8e4e1c83703665,
    ``indexer_k_quant_and_cache`` / ``indexer_k_quant_and_cache_kernel``.
    Later profiling backends construct production operands and time either a
    Torch composite or the fused vLLM CUDA launch. FP16/FP32 support here is
    for semantic validation only and does not imply profiler backend support.
    """
    _validate(
        k,
        cache,
        slot_mapping,
        quant_block_size=quant_block_size,
        scale_format=scale_format,
        cache_format=cache_format,
    )

    num_actual = slot_mapping.shape[0]
    index_dim = k.shape[1]
    block_size = cache.shape[1]
    num_groups = index_dim // quant_block_size
    page_bytes = block_size * cache.shape[2]
    scale_plane_offset = block_size * index_dim
    cache_bytes = cache.view(-1)

    for row in range(num_actual):
        slot = int(slot_mapping[row].item())
        if slot < 0:
            continue

        block, offset = divmod(slot, block_size)
        page_offset = block * page_bytes
        for group in range(num_groups):
            group_start = group * quant_block_size
            group_end = group_start + quant_block_size
            values_fp32 = k[row, group_start:group_end].float()
            amax = values_fp32.abs().amax()
            base_scale = torch.clamp_min(amax, _AMAX_FLOOR) / _FP8_E4M3_MAX
            scale = torch.exp2(torch.ceil(torch.log2(base_scale)))

            quantized_bytes = (values_fp32 / scale).to(torch.float8_e4m3fn).view(torch.uint8)
            key_offset = page_offset + offset * index_dim + group_start
            cache_bytes[key_offset : key_offset + quant_block_size].copy_(quantized_bytes)

            scale_offset = page_offset + scale_plane_offset + (offset * num_groups + group) * 4
            scale_bytes = scale.reshape(1).view(torch.uint8)
            cache_bytes[scale_offset : scale_offset + 4].copy_(scale_bytes)

    return cache


def _validate(
    k: object,
    cache: object,
    slot_mapping: object,
    *,
    quant_block_size: object,
    scale_format: object,
    cache_format: object,
) -> None:
    for name, tensor in (
        ("k", k),
        ("cache", cache),
        ("slot_mapping", slot_mapping),
    ):
        if not isinstance(tensor, torch.Tensor):
            raise TypeError(f"{name} must be a torch.Tensor")

    assert isinstance(k, torch.Tensor)
    assert isinstance(cache, torch.Tensor)
    assert isinstance(slot_mapping, torch.Tensor)

    if not isinstance(quant_block_size, int) or isinstance(quant_block_size, bool):
        raise TypeError("quant_block_size must be an integer")
    if quant_block_size <= 0:
        raise ValueError("quant_block_size must be positive")
    if scale_format != _SUPPORTED_SCALE_FORMAT:
        raise ValueError(f"scale_format must be {_SUPPORTED_SCALE_FORMAT!r}, got {scale_format!r}")
    if cache_format != _SUPPORTED_CACHE_FORMAT:
        raise ValueError(f"cache_format must be {_SUPPORTED_CACHE_FORMAT!r}, got {cache_format!r}")

    _validate_rank_and_shape(k, cache, slot_mapping)
    _validate_dtype_and_device(k, cache, slot_mapping)
    _validate_layout(k, cache, slot_mapping)

    index_dim = k.shape[1]
    if index_dim % quant_block_size != 0:
        raise ValueError(
            f"index_dim {index_dim} must be divisible by quant_block_size {quant_block_size}"
        )
    num_groups = index_dim // quant_block_size
    expected_width = index_dim + 4 * num_groups
    if cache.shape[2] != expected_width:
        raise ValueError(
            "cache final width must equal index_dim + 4*num_groups "
            f"({expected_width}), got {cache.shape[2]}"
        )
    if slot_mapping.shape[0] > k.shape[0]:
        raise ValueError(
            "slot_mapping length must not exceed the K backing-row count, "
            f"got {slot_mapping.shape[0]} and {k.shape[0]}"
        )

    if torch._C._overlaps(cache, k):
        raise ValueError("cache must not alias k")
    if torch._C._overlaps(cache, slot_mapping):
        raise ValueError("cache must not alias slot_mapping")

    participating_k = k[: slot_mapping.shape[0]]
    if not bool(torch.isfinite(participating_k).all().item()):
        raise ValueError("participating k rows must contain only finite values")

    nonnegative_slots = slot_mapping[slot_mapping >= 0]
    if nonnegative_slots.numel() == 0:
        return

    capacity = cache.shape[0] * cache.shape[1]
    if bool(torch.any(nonnegative_slots >= capacity).item()):
        raise ValueError(f"nonnegative slot values must be less than {capacity}")
    if torch.unique(nonnegative_slots).numel() != nonnegative_slots.numel():
        raise ValueError("duplicate nonnegative slots are not allowed")


def _validate_rank_and_shape(
    k: torch.Tensor,
    cache: torch.Tensor,
    slot_mapping: torch.Tensor,
) -> None:
    for name, tensor, expected_rank in (
        ("k", k, 2),
        ("cache", cache, 3),
        ("slot_mapping", slot_mapping, 1),
    ):
        if tensor.ndim != expected_rank:
            raise ValueError(f"{name} must be rank {expected_rank}, got rank {tensor.ndim}")
        if any(dimension <= 0 for dimension in tensor.shape):
            raise ValueError(f"{name} dimensions must be positive, got {tuple(tensor.shape)}")


def _validate_dtype_and_device(
    k: torch.Tensor,
    cache: torch.Tensor,
    slot_mapping: torch.Tensor,
) -> None:
    if k.dtype not in _SUPPORTED_INPUT_DTYPES:
        raise TypeError(
            f"k dtype must be torch.bfloat16, torch.float16, or torch.float32, got {k.dtype}"
        )
    if cache.dtype is not torch.uint8:
        raise TypeError(f"cache dtype must be torch.uint8, got {cache.dtype}")
    if slot_mapping.dtype is not torch.int64:
        raise TypeError(f"slot_mapping dtype must be torch.int64, got {slot_mapping.dtype}")

    for name, tensor in (("cache", cache), ("slot_mapping", slot_mapping)):
        if tensor.device != k.device:
            raise ValueError(
                f"{name} must be on the same device as k, got {tensor.device} and {k.device}"
            )
    if k.device.type == "meta":
        raise ValueError("meta tensors are not supported")


def _validate_layout(
    k: torch.Tensor,
    cache: torch.Tensor,
    slot_mapping: torch.Tensor,
) -> None:
    for name, tensor in (
        ("k", k),
        ("cache", cache),
        ("slot_mapping", slot_mapping),
    ):
        if tensor.layout is not torch.strided:
            raise ValueError(f"{name} must have torch.strided layout")
        if int(torch._debug_has_internal_overlap(tensor)) == 1:
            raise ValueError(f"{name} must not have internal overlap")
        if not tensor.is_contiguous():
            raise ValueError(f"{name} must be contiguous")
```

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/dsa_index_cache_append.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
