# Engram n-gram embedding lookup (`engram_lookup`) | ServingStudio Kernel Library

Gather hashed n-gram rows from a large FP8 table, dequantize them and write BF16 rows for the layers that consume them.

An Engram layer adds embeddings looked up by hashed token n-grams. One Triton call gathers num_tokens × local_heads rows from this GPU's slice of the table. Each row is head_dim FP8 E4M3 values plus one UE8M0 scale per quant_block_size values in a separate scale table. The call dequantizes the row to BF16 and writes it to a device staging buffer. Tables usually stay in pinned host memory and are read over UVA on a side stream with half the SMs; table_rows sets the TLB footprint of the random gather.

Category: Other. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=engram_lookup. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `out[t, h] = bf16(table[id[t, h]] · 2^(scale[id[t, h]] − 127)), per 32-value block`
- `GB/s = num_tokens · local_heads · [4 + head_dim + head_dim / quant_block_size + 2 · head_dim] / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. Only the lookup kernel's launches are counted. Each timed launch reads another id set from a pool of at least about a million distinct rows, so repeats find no rows left in L2 or the TLBs by an earlier launch. Before timing, the output must equal a PyTorch gather and dequantize bit for bit.

## Caveats

- The runner takes head_dim 256, quant_block_size 32 and FP8 tables, with at most 24 local heads, 65,536 tokens and 400 million table rows.
- Ids are uniform within each head's slice of the table; real n-gram hashes may hit some rows more often and cache better.
- TFLOPS is not computed.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_tokens` | number | tokens | Rows of hashed n-gram ids handed to one call. |
| `local_heads` | number | heads | Hash columns (heads) this GPU owns. |
| `head_dim` | number | elements | FP8 values in one table row. |
| `quant_block_size` | number | elements | Row values sharing one UE8M0 scale. |
| `table_rows` | number | rows | Rows of this GPU's table slice. |
| `residency` | label |  | Where the table lives: host_uva (pinned host memory read over UVA, half-SM background grid) or device (HBM, full-SM grid). |
| `weight_dtype` | dtype |  | Element type of the table rows. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `vllm_triton`: vLLM's Triton _engram_lookup_kernel through ParallelEngramEmbedding.lookup, with the module's own grid rule. dtypes `fp8_e4m3`, compute capability 8.9 or newer, GPUs H100-SXM5-80GB, H100-PCIe-80GB, H200-SXM-141GB, B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip, H800-SXM5-80GB, H20-SXM-96GB, L40, L40S, L4.

## Measured rows

146 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `vllm_triton` | `fp8_e4m3` | 146 |

## Used by these deployments

- [DeepSeek V4.1 Flash 1017: vLLM, DP attention, EP MoE, DSpark speculative decoding](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dp_attn_ep_moe_dspark.md) (`DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dp_attn_ep_moe_dspark`)
- [DeepSeek V4.1 Flash 1017: vLLM, TP attention, EP MoE, DSpark speculative decoding](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dspark.md) (`DeepSeek-V4.1-Flash-1017/deepseek_v41_vllm_dspark`)
- [DeepSeek V4.1 Flash: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash/deepseek_v41_vllm.md) (`DeepSeek-V4.1-Flash/deepseek_v41_vllm`)
- [DeepSeek V4.1 Flash: vLLM, TP attention, EP MoE, side streams serialized](https://servingstudio.cs.washington.edu/models/DeepSeek-V4.1-Flash/deepseek_v41_vllm_serial_streams.md) (`DeepSeek-V4.1-Flash/deepseek_v41_vllm_serial_streams`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/engram_lookup
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/engram_lookup/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/engram_lookup/configs

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/engram_lookup.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
