# Indexed row copy (`indexed_row_copy`) | ServingStudio Kernel Library

Gather, scatter or zero whole rows by index with eager PyTorch.

The slot-pool bookkeeping around a recurrent-state prefill. vLLM gathers each prefilling request's state row from the pool by slot index, zeroes the rows of requests that start without a prior state, and scatters the final rows back to their slots, all as eager tensor indexing. Indices are int32 slots spread across a larger pool. The index_select and index_copy operations are the token-row split and merge of a step mixing speculative and plain rows, with sorted int64 token indices.

Category: Other. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=indexed_row_copy. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `bytes(gather, scatter, index_select, index_copy) = 2·num_rows·row_elements·bytes(dtype)`
- `bytes(masked_zero) = num_rows·row_elements·bytes(dtype)`
- `GB/s = bytes / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. Every launch of the indexing statement is counted, including the mask inversion and nonzero pass of masked_zero. Pools, rows and indices are allocated before the capture.

## Caveats

- masked_zero zeroes all num_rows rows; a step whose mask keeps some rows is priced at the number it zeroes.
- Pool rows are contiguous; a pool whose slot stride pads the row is not modeled.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_rows` | number | rows | Rows read, written or zeroed. |
| `row_elements` | number | elements | Elements in each row. |
| `dtype` | dtype |  | Element type of the rows. |
| `operation` | label |  | gather reads indexed pool rows into a new tensor; scatter writes rows into indexed pool slots; masked_zero zeroes the rows a boolean mask leaves unset; index_select reads sorted token rows of a tensor; index_copy writes rows to sorted token rows. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `torch`: Eager PyTorch advanced indexing: index for gather, index_put_ for scatter and for the masked zero, index_select and index_copy_ for the token-row operations. dtypes `bf16`, `fp32`, any CUDA GPU.
- `torch_vllm`: Eager PyTorch advanced indexing: index for gather, index_put_ for scatter and for the masked zero, index_select and index_copy_ for the token-row operations. dtypes `bf16`, `fp32`, any CUDA GPU.

## Measured rows

165 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `torch` | `fp32` | 42 |
| NVIDIA B200 | `torch_vllm` | `bf16` | 81 |
| NVIDIA B200 | `torch_vllm` | `fp32` | 42 |

## Used by these deployments

- [Qwen3.8 27B NVFP4: vLLM, one GPU](https://servingstudio.cs.washington.edu/models/Qwen3.8-27B-NVFP4/qwen35_vllm_nvfp4_gdn_dense.md) (`Qwen3.8-27B-NVFP4/qwen35_vllm_nvfp4_gdn_dense`)
- [Qwen3.8 Flash Next NVFP4: vLLM, one GPU, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative.md) (`Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/indexed_row_copy
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/indexed_row_copy/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/indexed_row_copy/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/indexed_row_copy.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
