# MLA cache append (`mla_cache_append`) | ServingStudio Kernel Library

Write compressed KV latents and rotary keys into paged MLA cache entries.

After the MLA projections and RoPE, each token writes a kv_lora_rank-wide latent and a rope_dim-wide rotary key into one paged cache entry. The measurement scatters the rows to distinct random slots. The plain cache format takes any kv_lora_rank, rope_dim and page size; vllm_cuda also accepts rope_dim = 0, a latent without a rotary key, which it writes alone.

Category: Attention; group: MLA. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=mla_cache_append. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `cache[position] = [latent, rotary key]`
- `GB/s = num_tokens · ((kv_lora_rank + rope_dim) · (input bytes + cache bytes) + 8) / time`

## How it is measured

torch is timed with CUDA events: five warm-up calls, then a loop of back-to-back calls, taking the median of three runs, so the gap between its two launches is included. vllm_cuda and sglang_cuda use CUPTI kernel time with the L2 cache flushed before each launch: vllm_cuda counts only concat_and_cache_mla_kernel, after checking that every slot holds its row, and sglang_cuda counts every launch of its call after one setup call.

## Caveats

- torch rows are warm-cache elapsed time and CUPTI rows are cold-cache kernel time, so the backends are not directly comparable.
- sglang_cuda scatters bytes that are already FP8; vllm_cuda can convert BF16 input to an FP8 cache as it writes.
- GB/s counts logical source bytes, cache bytes and one 8-byte slot index per token; cache initialization is excluded.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_tokens` | number | tokens | Token rows appended to the cache. |
| `kv_lora_rank` | number | elements | Width of each compressed KV latent. |
| `rope_dim` | number | elements | Width of each rotary key part; 0 when there is none. |
| `block_size` | number | tokens | Token positions in each cache page. |
| `input_dtype` | dtype |  | Element type of the source rows. |
| `kv_dtype` | dtype |  | Element type of the cache entries. |
| `cache_format` | label |  | Layout of the MLA cache entries. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `torch`: PyTorch indexed assignment in two separate cache writes. dtypes `bf16`, KV dtypes `bf16`, any CUDA GPU.
- `vllm_cuda`: vLLM concat_and_cache_mla, a fused CUDA append that can convert BF16 rows to FP8 cache entries. dtypes `bf16`, KV dtypes `bf16`, `fp8_e4m3`, any CUDA GPU. Source: https://github.com/vllm-project/vllm/blob/main/csrc/libtorch_stable/cache_kernels.cu
- `sglang_cuda`: SGLang set_mla_kv_buffer_triton, a Triton kernel that scatters pre-quantized FP8 rows. dtypes `fp8_e4m3`, KV dtypes `fp8_e4m3`, any CUDA GPU.

## Measured rows

419 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `sglang_cuda` | `fp8_e4m3` | 70 |
| NVIDIA B200 | `vllm_cuda` | `bf16` | 210 |
| NVIDIA H200 | `vllm_cuda` | `bf16` | 139 |

## Used by these deployments

- [GLM-5.2 FP8: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-FP8/glm52_vllm_dsa_moe.md) (`GLM-5.2-FP8/glm52_vllm_dsa_moe`)
- [GLM-5.2 NVFP4: SGLang, TP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_sglang_nvfp4_tp_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dp_attn_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, TP and EP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe`)
- [GLM-5.2 NVFP4: vLLM, TP and EP, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_dsa_moe_speculative`)
- [GLM-5.2 NVFP4: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.2-NVFP4/glm52_vllm_nvfp4_pp_dsa_moe.md) (`GLM-5.2-NVFP4/glm52_vllm_nvfp4_pp_dsa_moe`)
- [GLM-5.2: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/GLM-5.2/glm52_vllm_dsa_moe.md) (`GLM-5.2/glm52_vllm_dsa_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_dp_attn_ep_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_kda_dsa_moe`)
- [GLM-5.3-Flash NVFP4: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe.md) (`GLM-5.3-Flash-NVFP4/glm53_flash_vllm_nvfp4_pp_kda_dsa_moe`)
- [GLM-5.3-Flash: vLLM, DP attention, EP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_dp_attn_ep_moe`)
- [GLM-5.3-Flash: vLLM, TP attention, EP or TP MoE](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_kda_dsa_moe`)
- [GLM-5.3-Flash: vLLM, PP](https://servingstudio.cs.washington.edu/models/GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe.md) (`GLM-5.3-Flash/glm53_flash_vllm_fp8_pp_kda_dsa_moe`)
- [GLM-5.3 NVFP4: vLLM, TP and EP, DFlash speculative decoding](https://servingstudio.cs.washington.edu/models/GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2.md) (`GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/mla_cache_append
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/mla_cache_append/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/mla_cache_append/configs

## Torch reference

What the kind computes, as Torch code (`profiling/runners/attention/mla_cache_append_reference.py` in ServingStudio Sim):

```python
"""Torch semantic reference for the plain MLA paged-cache append."""

from __future__ import annotations

import torch

_SUPPORTED_DTYPES = (torch.bfloat16, torch.float16, torch.float32)


def mla_cache_append_reference(
    kv_c: torch.Tensor,
    k_pe: torch.Tensor,
    cache: torch.Tensor,
    slot_mapping: torch.Tensor,
) -> torch.Tensor:
    """Append latent and RoPE rows to one paged cache and return that cache.

    Validation and slot addressing are completed before the cache is mutated.
    Negative slots are padding sentinels and do not write anything.

    Source boundary: vLLM v0.23.0 commit
    0fc695fc6d1d82e9a5ac6835ac8e4e1c83703665,
    ``concat_and_cache_mla`` / ``concat_and_cache_mla_kernel``. This reference
    covers the plain same-dtype format; mixed ``fp8_ds_mla`` is deferred.
    Profiling backends construct the exact GLM layouts and time either this
    Torch composite or vLLM's one fused CUDA launch.
    """
    _validate_tensors(kv_c, k_pe, cache, slot_mapping)

    valid_rows = torch.nonzero(slot_mapping >= 0, as_tuple=False).flatten()
    if valid_rows.numel() == 0:
        return cache

    slots = slot_mapping[valid_rows]
    block_size = cache.shape[1]
    block_indices = torch.div(slots, block_size, rounding_mode="floor")
    block_offsets = slots % block_size
    kv_lora_rank = kv_c.shape[1]

    cache[block_indices, block_offsets, :kv_lora_rank] = kv_c[valid_rows]
    cache[block_indices, block_offsets, kv_lora_rank:] = k_pe[valid_rows]
    return cache


def _validate_tensors(
    kv_c: object,
    k_pe: object,
    cache: object,
    slot_mapping: object,
) -> None:
    tensors = {
        "kv_c": kv_c,
        "k_pe": k_pe,
        "cache": cache,
        "slot_mapping": slot_mapping,
    }
    for name, tensor in tensors.items():
        if not isinstance(tensor, torch.Tensor):
            raise TypeError(f"{name} must be a torch.Tensor")

    assert isinstance(kv_c, torch.Tensor)
    assert isinstance(k_pe, torch.Tensor)
    assert isinstance(cache, torch.Tensor)
    assert isinstance(slot_mapping, torch.Tensor)

    _validate_rank_and_shape(kv_c, k_pe, cache, slot_mapping)
    _validate_dtype_and_device(kv_c, k_pe, cache, slot_mapping)
    _validate_layout(kv_c, k_pe, cache, slot_mapping)

    if k_pe.shape[0] != kv_c.shape[0]:
        raise ValueError(
            "kv_c and k_pe must have the same backing-row count, "
            f"got {kv_c.shape[0]} and {k_pe.shape[0]}"
        )
    if slot_mapping.shape[0] > kv_c.shape[0]:
        raise ValueError(
            "slot_mapping length must not exceed the input backing-row count, "
            f"got {slot_mapping.shape[0]} and {kv_c.shape[0]}"
        )

    expected_width = kv_c.shape[1] + k_pe.shape[1]
    if cache.shape[2] != expected_width:
        raise ValueError(
            f"cache final width must equal R + P ({expected_width}), "
            f"got {cache.shape[2]}"
        )

    if torch._C._overlaps(cache, kv_c) or torch._C._overlaps(cache, k_pe):
        raise ValueError("cache must not alias kv_c or k_pe")

    positive_slots = slot_mapping[slot_mapping >= 0]
    if positive_slots.numel() == 0:
        return

    num_cache_slots = cache.shape[0] * cache.shape[1]
    if bool(torch.any(positive_slots >= num_cache_slots).item()):
        raise ValueError(
            f"nonnegative slot values must be less than {num_cache_slots}"
        )
    if torch.unique(positive_slots).numel() != positive_slots.numel():
        raise ValueError("duplicate nonnegative slots are not allowed")


def _validate_rank_and_shape(
    kv_c: torch.Tensor,
    k_pe: torch.Tensor,
    cache: torch.Tensor,
    slot_mapping: torch.Tensor,
) -> None:
    expected_ranks = {"kv_c": 2, "k_pe": 2, "cache": 3, "slot_mapping": 1}
    for name, tensor in (
        ("kv_c", kv_c),
        ("k_pe", k_pe),
        ("cache", cache),
        ("slot_mapping", slot_mapping),
    ):
        expected_rank = expected_ranks[name]
        if tensor.ndim != expected_rank:
            raise ValueError(
                f"{name} must be rank {expected_rank}, got rank {tensor.ndim}"
            )
        if any(dimension <= 0 for dimension in tensor.shape):
            raise ValueError(
                f"{name} dimensions must be positive, got {tuple(tensor.shape)}"
            )


def _validate_dtype_and_device(
    kv_c: torch.Tensor,
    k_pe: torch.Tensor,
    cache: torch.Tensor,
    slot_mapping: torch.Tensor,
) -> None:
    if kv_c.dtype not in _SUPPORTED_DTYPES:
        raise TypeError(
            "kv_c dtype must be torch.bfloat16, torch.float16, or torch.float32, "
            f"got {kv_c.dtype}"
        )
    if k_pe.dtype != kv_c.dtype:
        raise TypeError(
            f"k_pe dtype must match kv_c dtype {kv_c.dtype}, got {k_pe.dtype}"
        )
    if cache.dtype != kv_c.dtype:
        raise TypeError(
            f"cache dtype must match kv_c dtype {kv_c.dtype}, got {cache.dtype}"
        )
    if slot_mapping.dtype is not torch.int64:
        raise TypeError(
            f"slot_mapping dtype must be torch.int64, got {slot_mapping.dtype}"
        )

    for name, tensor in (
        ("k_pe", k_pe),
        ("cache", cache),
        ("slot_mapping", slot_mapping),
    ):
        if tensor.device != kv_c.device:
            raise ValueError(
                f"{name} must be on the same device as kv_c, "
                f"got {tensor.device} and {kv_c.device}"
            )


def _validate_layout(
    kv_c: torch.Tensor,
    k_pe: torch.Tensor,
    cache: torch.Tensor,
    slot_mapping: torch.Tensor,
) -> None:
    for name, tensor in (("kv_c", kv_c), ("k_pe", k_pe), ("cache", cache)):
        if tensor.layout is not torch.strided:
            raise ValueError(f"{name} must have torch.strided layout")
        if tensor.stride(-1) != 1:
            raise ValueError(f"{name} innermost stride must be 1")
        # Status 1 is definite overlap. Status 2 means Torch cannot prove the
        # layout either way and must remain accepted for valid row-strided views.
        if int(torch._debug_has_internal_overlap(tensor)) == 1:
            raise ValueError(f"{name} must not have internal overlap")

    if slot_mapping.layout is not torch.strided:
        raise ValueError("slot_mapping must have torch.strided layout")
    if not slot_mapping.is_contiguous():
        raise ValueError("slot_mapping must be contiguous")
```

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/mla_cache_append.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
