# Indexer query RoPE and FP8 quantization, scales in weights (`dsa_indexer_q_rope_quant_weight_fold`) | ServingStudio Kernel Library

Rotate and quantize indexer queries while folding their FP8 scales into head weights.

A first step of the DSA indexer: each query head has RoPE applied to its trailing rope_dim elements and is quantized to FP8 with a power-of-two scale per token and head. That scale, together with both indexer scaling factors, is folded into the head's output weight, so the logits come out right without dequantizing. Positions are spread across max_model_len; queries and weights are deterministic.

Category: Attention; group: DSA. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=dsa_indexer_q_rope_quant_weight_fold. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `q_rot = RoPE(q), over the trailing rope_dim elements`
- `scale = 2^⌈log₂(max(max|q_rot|, scale_epsilon) / fp8_max)⌉, per token and head`
- `q_fp8 = q_rot / scale; weight_out = weight · scale · index_weights_softmax_scale · index_weights_head_scale`
- `GB/s = (8 · num_tokens + 2 · num_tokens · 64 · 128 + 4 · num_tokens · 64 + 2 · num_tokens · 64 + num_tokens · 64 · 128 + 4 · num_tokens · 64) / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. The output is checked against a PyTorch computation before timing; every launch of the CuTe DSL call is counted.

## Caveats

- GB/s counts logical query, position, weight and output bytes, not RoPE-table reads.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_tokens` | number | tokens | Indexer query rows processed together. |
| `num_heads` | number | heads | Indexer query heads per token. |
| `head_dim` | number | elements | Elements in each indexer query head. |
| `rope_dim` | number | elements | Trailing query elements rotated by RoPE. |
| `max_model_len` | number | tokens | Maximum position used to size the RoPE table. |
| `max_num_batched_tokens` | number | tokens | Maximum query rows admitted to one batch. |
| `index_weights_softmax_scale` | number | multiplier | Scale folded into each output head weight for query width. |
| `index_weights_head_scale` | number | multiplier | Scale folded into each output head weight for head count. |
| `fp8_max` | number | unitless | Largest magnitude used when quantizing each query head. |
| `scale_epsilon` | number | unitless | Minimum head magnitude used to calculate the FP8 scale. |
| `positions_dtype` | label |  | Element type of token positions. |
| `q_dtype` | dtype |  | Element type of the unrotated indexer queries. |
| `rope_dtype` | dtype |  | Element type of the RoPE table. |
| `weight_dtype` | dtype |  | Element type of input indexer head weights. |
| `q_output_dtype` | dtype |  | Element type of rotated, quantized indexer queries. |
| `weight_output_dtype` | dtype |  | Element type of output indexer head weights. |
| `rope_style` | label |  | Position and feature layout used by RoPE. |
| `quant_mode` | label |  | FP8 scaling scheme, including how scales enter the output weights. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_cutedsl_fp8`: vLLM's fused_indexer_q_rope_quant_fp8_cutedsl call combines RoPE, per-head FP8 quantization and weight scaling. dtypes `bf16`, compute capability 8.9 or newer, GPUs H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip, H800-SXM5-80GB, H20-SXM-96GB, L40, L40S, L4. Source: https://github.com/vllm-project/vllm/blob/main/vllm/models/deepseek_v4/nvidia/ops/fused_indexer_q_cutedsl.py

## Measured rows

40 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA H200 | `vllm_cutedsl_fp8` | `bf16` | 40 |

## Used by these deployments

- [DeepSeek V4 Flash 0731: vLLM, DP attention, EP](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm.md) (`DeepSeek-V4-Flash-0731/deepseek_v4_vllm`)
- [DeepSeek V4 Flash 0731: vLLM, DP attention, EP, parallel streams serialized](https://servingstudio.cs.washington.edu/models/DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams.md) (`DeepSeek-V4-Flash-0731/deepseek_v4_vllm_serial_streams`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/dsa_indexer_q_rope_quant_weight_fold
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/dsa_indexer_q_rope_quant_weight_fold/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/dsa_indexer_q_rope_quant_weight_fold/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/dsa_indexer_q_rope_quant_weight_fold.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
