# Relative-position attention (`rel_bias_attention`) | ServingStudio Kernel Library

Paged causal or sliding-window GQA attention with a per-query relative-position logit added to every score, over one mixed decode and prefill batch.

No RoPE is applied: each query row carries rel_extent learned logits, one per distance, added to its scores against the last rel_extent keys. Local layers also restrict keys to that window; global layers are full causal with no bias beyond it. One call covers the whole varlen batch over a paged LBNHC KV cache (K and V of one token and head packed side by side). On Blackwell the call first shears the logits into a KV-tile-aligned bias tensor, then runs FlashAttention 4 with bias tiles; global layers split the KV range by the fork's inkling_fa4_num_splits rule and combine the partials.

Category: Attention; group: MHA / GQA. Interactive page: https://servingstudio.cs.washington.edu/kernels.html?kind=rel_bias_attention. All kernel kinds: https://servingstudio.cs.washington.edu/kernels.md.

## Formula

- `O = softmax(Q · Kᵀ / head_dim + B_rel) · V, B_rel[i, j] = rel_logits[i, h, i − j] for 0 ≤ i − j < rel_extent, else 0 (global) or masked (local)`
- `n_i = keys query i sees: i + 1 (global) or min(i + 1, rel_extent) (local)`
- `TFLOPS = 4 · num_qo_heads · head_dim · Σ n_i / time`
- `GB/s = (Q + O + rel_logits + each request's visible K and V once) bytes / time`

## How it is measured

GPU kernel time from CUPTI activity records, averaged over repeated launches in one capture. The L2 cache is flushed before each launch, so the numbers are cold-cache times. Every launch of the call is counted: the cu-blocks metadata pair, ShearingBias, the split scheduler, FlashAttention forward and the combine, whichever the batch issues. num_splits follows the fork's inkling_fa4_num_splits. Before timing, the output is checked against a PyTorch reference on sampled query rows.

## Caveats

- Queries, keys, values and logits are random; q is drawn with standard deviation 4 so both the QK and the bias terms move the softmax.
- The cache insert and the logits projection belong to the QKV-prep kind, not to this call.
- KV pages are shuffled across a pool sized to the batch; the page table is max_model_len / page_size wide, as vLLM's.

## Arguments

| Argument | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `num_qo_heads` | number | heads | Query and output heads on this GPU. |
| `num_kv_heads` | number | heads | KV heads on this GPU. |
| `head_dim` | number | elements | Dimension of each query, key and value head. |
| `is_local` | label |  | Sliding-window layer (window = rel_extent tokens) rather than full causal. |
| `rel_extent` | number | tokens | Distances that carry a relative-position logit; the window if local. |
| `page_size` | number | tokens | Tokens per KV-cache page. |
| `kv_layout` | label |  | KV-cache page layout; lbnhc packs K and V of one (token, head) side by side. |
| `q_dtype` | dtype |  | Query, relative-logit and output element type. |
| `kv_dtype` | dtype |  | Key and value cache element type. |
| `max_model_len` | number | tokens | Model context limit; sizes the page table and the global split count. |
| `query_context_pairs` | list |  | (query tokens, context tokens) per request in the one varlen call; the context includes the query. |

## Metrics

- Time (`time_ms`, ms)
- Throughput (`tflops`, TFLOPS)
- Memory bandwidth (`memory_bandwidth_gbps`, GB/s)
- Energy (`energy_j`, J)

## Backends

- `tml_fa4`: The fork's InklingFA4RelAttentionKernel.kernel: tml_fa4 flash_attn_varlen_func with the sheared relative bias, split by inkling_fa4_num_splits. dtypes `bf16`, KV dtypes `bf16`, SM targets `sm_100f`, GPUs B200-SXM-180GB, B300-SXM-288GB, GB200-Superchip.

## Measured rows

6,472 rows in all.

| GPU | Backend | Precision | Rows |
| --- | --- | --- | --- |
| NVIDIA B200 | `tml_fa4` | `bf16` | 6,472 |

## Used by these deployments

- [Inkling NVFP4: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Inkling-NVFP4/inkling_vllm_nvfp4_rel_attn_moe.md) (`Inkling-NVFP4/inkling_vllm_nvfp4_rel_attn_moe`)
- [Inkling Small NVFP4: vLLM, TP attention, EP MoE](https://servingstudio.cs.washington.edu/models/Inkling-Small-NVFP4/inkling_vllm_nvfp4_rel_attn_moe.md) (`Inkling-Small-NVFP4/inkling_vllm_nvfp4_rel_attn_moe`)

## API

- This kind: https://servingstudio.cs.washington.edu/api/public/v1/kernels/rel_bias_attention
- Measured rows: https://servingstudio.cs.washington.edu/api/public/v1/kernels/rel_bias_attention/rows (filter by any column, such as `?gpu=NVIDIA%20H200`; `format=csv` for CSV)
- Configs the public deployments ask of it: https://servingstudio.cs.washington.edu/api/public/v1/kernels/rel_bias_attention/configs

## Torch reference

What the kind computes, as Torch code (`profiling/runners/attention/rel_bias_attention_reference.py` in ServingStudio Sim):

```python
"""PyTorch reference for relative-position attention.

Independent of tml_fa4: it reads keys and values through the page table and
applies the relative logits by distance, the way the model defines them
(``fa4_rel_attention.py`` score_mod and ``tml_fa4/testing.py attention_ref``):

    s[h, i, j] = (q[i, h] . k[j, h // G]) * softmax_scale + B[i, h, j]
    B[i, h, j] = rel_logits[i, h, pos_i - j]   if 0 <= pos_i - j < rel_extent
               = 0                            if pos_i - j >= rel_extent (global)
    keys j > pos_i are masked; local layers also mask pos_i - j >= rel_extent.

Everything runs in float32 on whatever device the inputs live on.
"""

from __future__ import annotations

from typing import Any


def gather_request_kv(key_cache: Any, value_cache: Any, block_row: Any, context: int) -> tuple:
    """Keys and values ``[context, kv_heads, head_dim]`` of one request.

    ``key_cache`` / ``value_cache`` are ``[num_pages, page_size, kv_heads, head_dim]``
    views (any strides); ``block_row`` maps a request page index to a physical page.
    """
    import torch

    page_size = key_cache.shape[1]
    tokens = torch.arange(context, device=key_cache.device)
    pages = block_row.to(key_cache.device).long()[tokens // page_size]
    offsets = tokens % page_size
    return key_cache[pages, offsets].float(), value_cache[pages, offsets].float()


def rel_attention_reference(
    q: Any,
    positions: Any,
    keys: Any,
    values: Any,
    rel_logits: Any,
    *,
    softmax_scale: float,
    is_local: bool,
) -> Any:
    """Attention output ``[rows, heads, head_dim]`` (float32) for one request.

    ``q`` is ``[rows, heads, head_dim]`` at absolute ``positions`` ``[rows]``;
    ``keys`` / ``values`` hold the request's first ``context`` tokens
    ``[context, kv_heads, head_dim]``; ``rel_logits`` is ``[rows, heads, rel_extent]``.
    """
    import torch

    rows, heads, _ = q.shape
    kv_heads = keys.shape[1]
    rel_extent = rel_logits.shape[-1]
    group = heads // kv_heads
    k = keys.float().repeat_interleave(group, dim=1)  # [S, H, D]
    v = values.float().repeat_interleave(group, dim=1)
    scores = torch.einsum("thd,shd->hts", q.float(), k) * softmax_scale
    distance = positions.long()[:, None] - torch.arange(keys.shape[0], device=q.device)[None, :]
    in_extent = (distance >= 0) & (distance < rel_extent)
    index = distance.clamp(0, rel_extent - 1)  # [rows, S]
    bias = torch.gather(
        rel_logits.float(), -1, index[:, None, :].expand(rows, heads, index.shape[1])
    )  # [rows, H, S]
    bias = bias.masked_fill(~in_extent[:, None, :], 0.0).permute(1, 0, 2)
    scores = scores + bias
    visible = in_extent if is_local else distance >= 0
    scores = scores.masked_fill(~visible[None], float("-inf"))
    probs = torch.softmax(scores, dim=-1)
    return torch.einsum("hts,shd->thd", probs, v)
```

---

This page as Markdown: https://servingstudio.cs.washington.edu/kernels/rel_bias_attention.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
