# Embedding row gather (`embedding_row_gather`) | ServingStudio Kernel Library

Copy num_tokens × num_heads randomly addressed rows of a large table, kept in pinned host memory or HBM, into a device buffer.

Per-layer n-gram embeddings hash each token's preceding n-grams into num_heads ids, one per disjoint range of a very large table. The gather copies those rows unchanged (no dequantization) into a device staging buffer. With the table in pinned host memory the rows cross the host link one small read at a time, so the time follows the link's random-read rate and the table footprint, not HBM bandwidth.

Category: Other. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=embedding_row_gather. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `out[t, h, :] = table[id[t, h], :]`
- `GB/s = num_tokens · num_heads · (8 + 2 · head_dim · bytes(weight_dtype)) / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. Each timed call reads the next of 4 to 4096 id sets, as many as bring the pool to about four million rows (fewer for calls under 1024 ids), so consecutive calls do not reuse rows left in L2 or the TLBs. Before timing, the output must equal a PyTorch index_select bit for bit.

## Caveats

- Ids are uniform within each head's equal share of the table; real n-gram hashes repeat frequent n-grams and may hit fewer distinct pages.
- The table is pinned in the profiling process's host memory; NUMA placement relative to the GPU follows the worker's CPU binding.
- TFLOPS is not computed.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_tokens` | number | tokens | Rows of the \[tokens, heads\] id tensor. |
| `num_heads` | number | heads | Ids per token, each in its own table range. |
| `head_dim` | number | elements | Table elements in one row. |
| `table_rows` | number | rows | Rows of the table (sets the gather footprint). |
| `residency` | label |  | Where the table lives: host_uva (pinned host memory over UVA) or device (HBM). |
| `weight_dtype` | dtype |  | Element type of the table rows. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_triton`: The fork's Qwen4ExpPLEPinnedHostEmbedding._lookup (_lookup_ple_embedding_from_pinned_kernel) on a pinned host table through its UVA view; host_uva only. dtypes `fp8_e4m3`, compute capability 8.9 or newer, GPUs H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip, H800-SXM5-80GB, H20-SXM-96GB, L40, L40S, L4.
- `torch`: torch.nn.functional.embedding on an HBM table, the cpu_offload=False path (Qwen4ExpPLEDeviceEmbedding); device only. dtypes `bf16`, `fp8_e4m3`, any CUDA GPU.

## Measured rows

78 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `torch` | `bf16` | 26 |
| NVIDIA B200 | `torch` | `fp8_e4m3` | 26 |
| NVIDIA B200 | `vllm_triton` | `fp8_e4m3` | 26 |

## Used by these deployments

- [Qwen3.8 Flash Next NVFP4: vLLM, one GPU, MTP speculative decoding](https://servingstudio.cs.washington.edu/models/Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative.md) (`Qwen3.8-Flash-Next-NVFP4/qwen4_exp_vllm_nvfp4_gdn_qsa_moe_speculative`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/embedding_row_gather
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/embedding_row_gather/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/embedding_row_gather/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/embedding_row_gather.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
