# Inkling Small NVFP4: vLLM, TP attention, EP MoE | ServingStudio Models

Deployment `Inkling-Small-NVFP4/inkling_vllm_nvfp4_rel_attn_moe` of Inkling Small NVFP4 (`thinkingmachines/Inkling-Small-NVFP4`, family Inkling). Arch `inkling_vllm_nvfp4_rel_attn_moe`; Whole iteration; GPU NVIDIA B200. Interactive page, with its cost tree and live prediction: https://servingstudio.cs.washington.edu/models.html?preset=Inkling-Small-NVFP4%2Finkling_vllm_nvfp4_rel_attn_moe. All models: https://servingstudio.cs.washington.edu/models.md.

## Configurations

The preset sweeps `workload`; each configuration (member) is one value of each.

| Parameters | GPUs per replica | Case form | Groups | max_model_len | Can be timed |
| --- | --- | --- | --- | --- | --- |
| workload diverse_100 | 4 | `iter` | 1 | 131,072 | yes |

## Timing a batch

`POST https://servingstudio.cs.washington.edu/api/public/v1/predict` with the preset id, one configuration's `params` and 1 to 64 `cases`. A case is `{"groups": [...]}`, one object per group (the table's Groups column), each with the fields of the configuration's case form. The answer gives each case's time per section and per node of its cost tree. The servingstudio-timing-predict skill walks through it: https://servingstudio.cs.washington.edu/skills/servingstudio-timing-predict.md.

```json
{"preset":"Inkling-Small-NVFP4/inkling_vllm_nvfp4_rel_attn_moe","params":{"workload":"diverse_100"},"cases":["…"]}
```

Fields of a group for case form `iter`:

| Field | Type | Required | Meaning |
| --- | --- | --- | --- |
| `prefill_chunk_pairs` | int_pair_list | no | Prefill requests, one `[prefix_len, append_len]` each: the tokens already in the KV cache and the tokens this iteration computes (`prefix_len` 0 for a fresh prompt). |
| `decode_kv_lens` | int_list | no | Decode requests, one KV length each: the context its new token attends to. Give this or `decode_count`, not both. |
| `decode_count` | int | no | Shorthand for `decode_kv_lens`: this many decode requests, each `average_decode_length` long. |
| `average_decode_length` | int | no | The KV length of each `decode_count` request; required with it. |

## Kernels of configuration workload diverse_100

Each section's leaf slots, in order: the kernel kind each calls and the backends measured for it. Other configurations share the shape but not every size.

### Section `iter`

| Slot | Kernel | Backends |
| --- | --- | --- |
| `unified.embed` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.log_scaling_tau` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `torch` |
| `unified.local_dense.attn.qkvr_proj` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_dense.attn.qkvr_prep` | [sconv_qkvr_prep](https://servingstudio.cs.washington.edu/kernels/sconv_qkvr_prep.md) | `vllm_triton` |
| `unified.local_dense.attn.rel_attention` | [rel_bias_attention](https://servingstudio.cs.washington.edu/kernels/rel_bias_attention.md) | `tml_fa4` |
| `unified.local_dense.attn.o_proj` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_dense.attn.rs_sconv_ag_norm` | [rs_sconv_ag_residual_rms_norm](https://servingstudio.cs.washington.edu/kernels/rs_sconv_ag_residual_rms_norm.md) | `vllm_lamport_triton` |
| `unified.local_dense.ffn.gate_up` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_dense.ffn.act_and_mul` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.local_dense.ffn.down` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_dense.ffn.global_scale` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `torch` |
| `unified.local_dense.ffn.rs_sconv_ag_norm` | [rs_sconv_ag_residual_rms_norm](https://servingstudio.cs.washington.edu/kernels/rs_sconv_ag_residual_rms_norm.md) | `vllm_lamport_triton` |
| `unified.local_bf16_moe.attn.qkvr_proj` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_bf16_moe.attn.qkvr_prep` | [sconv_qkvr_prep](https://servingstudio.cs.washington.edu/kernels/sconv_qkvr_prep.md) | `vllm_triton` |
| `unified.local_bf16_moe.attn.rel_attention` | [rel_bias_attention](https://servingstudio.cs.washington.edu/kernels/rel_bias_attention.md) | `tml_fa4` |
| `unified.local_bf16_moe.attn.o_proj` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_bf16_moe.attn.rs_sconv_ag_norm` | [rs_sconv_ag_residual_rms_norm](https://servingstudio.cs.washington.edu/kernels/rs_sconv_ag_residual_rms_norm.md) | `vllm_lamport_triton` |
| `unified.local_bf16_moe.moe.router` | [gemm_fp32_output](https://servingstudio.cs.washington.edu/kernels/gemm_fp32_output.md) | `vllm_ll_bf16_auto` |
| `unified.local_bf16_moe.moe.gate_select` | [moe_sink_gate_select](https://servingstudio.cs.washington.edu/kernels/moe_sink_gate_select.md) | `vllm_triton` |
| `unified.local_bf16_moe.moe.topk_copy` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `torch` |
| `unified.local_bf16_moe.moe.sink.gate_up` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_bf16_moe.moe.sink.act_and_mul` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.local_bf16_moe.moe.sink.down` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_bf16_moe.moe.routed_rank0.fused_moe` | [bf16_fused_moe](https://servingstudio.cs.washington.edu/kernels/bf16_fused_moe.md) | `flashinfer_trtllm_sm100` |
| `unified.local_bf16_moe.moe.routed_rank0.fused_moe` | [bf16_fused_moe](https://servingstudio.cs.washington.edu/kernels/bf16_fused_moe.md) | `flashinfer_trtllm_sm100` |
| `unified.local_bf16_moe.moe.sink.gate_up` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_bf16_moe.moe.sink.act_and_mul` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.local_bf16_moe.moe.sink.down` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_bf16_moe.moe.sink.gate_up` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_bf16_moe.moe.sink.act_and_mul` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.local_bf16_moe.moe.sink.down` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_bf16_moe.moe.routed_rank1.fused_moe` | [bf16_fused_moe](https://servingstudio.cs.washington.edu/kernels/bf16_fused_moe.md) | `flashinfer_trtllm_sm100` |
| `unified.local_bf16_moe.moe.routed_rank1.fused_moe` | [bf16_fused_moe](https://servingstudio.cs.washington.edu/kernels/bf16_fused_moe.md) | `flashinfer_trtllm_sm100` |
| `unified.local_bf16_moe.moe.sink.gate_up` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_bf16_moe.moe.sink.act_and_mul` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.local_bf16_moe.moe.sink.down` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_bf16_moe.moe.sink.gate_up` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_bf16_moe.moe.sink.act_and_mul` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.local_bf16_moe.moe.sink.down` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_bf16_moe.moe.routed_rank2.fused_moe` | [bf16_fused_moe](https://servingstudio.cs.washington.edu/kernels/bf16_fused_moe.md) | `flashinfer_trtllm_sm100` |
| `unified.local_bf16_moe.moe.routed_rank2.fused_moe` | [bf16_fused_moe](https://servingstudio.cs.washington.edu/kernels/bf16_fused_moe.md) | `flashinfer_trtllm_sm100` |
| `unified.local_bf16_moe.moe.sink.gate_up` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_bf16_moe.moe.sink.act_and_mul` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.local_bf16_moe.moe.sink.down` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_bf16_moe.moe.sink.gate_up` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_bf16_moe.moe.sink.act_and_mul` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.local_bf16_moe.moe.sink.down` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_bf16_moe.moe.routed_rank3.fused_moe` | [bf16_fused_moe](https://servingstudio.cs.washington.edu/kernels/bf16_fused_moe.md) | `flashinfer_trtllm_sm100` |
| `unified.local_bf16_moe.moe.routed_rank3.fused_moe` | [bf16_fused_moe](https://servingstudio.cs.washington.edu/kernels/bf16_fused_moe.md) | `flashinfer_trtllm_sm100` |
| `unified.local_bf16_moe.moe.sink.gate_up` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_bf16_moe.moe.sink.act_and_mul` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.local_bf16_moe.moe.sink.down` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_bf16_moe.moe.rs_sconv_ag_norm` | [rs_sconv_ag_residual_rms_norm](https://servingstudio.cs.washington.edu/kernels/rs_sconv_ag_residual_rms_norm.md) | `vllm_lamport_triton` |
| `unified.local_nvfp4_moe.attn.qkvr_proj` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_nvfp4_moe.attn.qkvr_prep` | [sconv_qkvr_prep](https://servingstudio.cs.washington.edu/kernels/sconv_qkvr_prep.md) | `vllm_triton` |
| `unified.local_nvfp4_moe.attn.rel_attention` | [rel_bias_attention](https://servingstudio.cs.washington.edu/kernels/rel_bias_attention.md) | `tml_fa4` |
| `unified.local_nvfp4_moe.attn.o_proj` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_nvfp4_moe.attn.rs_sconv_ag_norm` | [rs_sconv_ag_residual_rms_norm](https://servingstudio.cs.washington.edu/kernels/rs_sconv_ag_residual_rms_norm.md) | `vllm_lamport_triton` |
| `unified.local_nvfp4_moe.moe.router` | [gemm_fp32_output](https://servingstudio.cs.washington.edu/kernels/gemm_fp32_output.md) | `vllm_ll_bf16_auto` |
| `unified.local_nvfp4_moe.moe.gate_select` | [moe_sink_gate_select](https://servingstudio.cs.washington.edu/kernels/moe_sink_gate_select.md) | `vllm_triton` |
| `unified.local_nvfp4_moe.moe.topk_copy` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `torch` |
| `unified.local_nvfp4_moe.moe.sink.gate_up` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_nvfp4_moe.moe.sink.act_and_mul` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.local_nvfp4_moe.moe.sink.down` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_nvfp4_moe.moe.routed_rank0.input_quant` | [nvfp4_quant](https://servingstudio.cs.washington.edu/kernels/nvfp4_quant.md) | `vllm_cuda` |
| `unified.local_nvfp4_moe.moe.routed_rank0.fused_moe` | [nvfp4_fused_moe](https://servingstudio.cs.washington.edu/kernels/nvfp4_fused_moe.md) | `flashinfer_trtllm_routed_sm100_fork` |
| `unified.local_nvfp4_moe.moe.routed_rank0.input_quant` | [nvfp4_quant](https://servingstudio.cs.washington.edu/kernels/nvfp4_quant.md) | `vllm_cuda` |
| `unified.local_nvfp4_moe.moe.routed_rank0.fused_moe` | [nvfp4_fused_moe](https://servingstudio.cs.washington.edu/kernels/nvfp4_fused_moe.md) | `flashinfer_trtllm_routed_sm100_fork` |
| `unified.local_nvfp4_moe.moe.sink.gate_up` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_nvfp4_moe.moe.sink.act_and_mul` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.local_nvfp4_moe.moe.sink.down` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_nvfp4_moe.moe.sink.gate_up` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_nvfp4_moe.moe.sink.act_and_mul` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.local_nvfp4_moe.moe.sink.down` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_nvfp4_moe.moe.routed_rank1.input_quant` | [nvfp4_quant](https://servingstudio.cs.washington.edu/kernels/nvfp4_quant.md) | `vllm_cuda` |
| `unified.local_nvfp4_moe.moe.routed_rank1.fused_moe` | [nvfp4_fused_moe](https://servingstudio.cs.washington.edu/kernels/nvfp4_fused_moe.md) | `flashinfer_trtllm_routed_sm100_fork` |
| `unified.local_nvfp4_moe.moe.routed_rank1.input_quant` | [nvfp4_quant](https://servingstudio.cs.washington.edu/kernels/nvfp4_quant.md) | `vllm_cuda` |
| `unified.local_nvfp4_moe.moe.routed_rank1.fused_moe` | [nvfp4_fused_moe](https://servingstudio.cs.washington.edu/kernels/nvfp4_fused_moe.md) | `flashinfer_trtllm_routed_sm100_fork` |
| `unified.local_nvfp4_moe.moe.sink.gate_up` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_nvfp4_moe.moe.sink.act_and_mul` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.local_nvfp4_moe.moe.sink.down` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_nvfp4_moe.moe.sink.gate_up` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_nvfp4_moe.moe.sink.act_and_mul` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.local_nvfp4_moe.moe.sink.down` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_nvfp4_moe.moe.routed_rank2.input_quant` | [nvfp4_quant](https://servingstudio.cs.washington.edu/kernels/nvfp4_quant.md) | `vllm_cuda` |
| `unified.local_nvfp4_moe.moe.routed_rank2.fused_moe` | [nvfp4_fused_moe](https://servingstudio.cs.washington.edu/kernels/nvfp4_fused_moe.md) | `flashinfer_trtllm_routed_sm100_fork` |
| `unified.local_nvfp4_moe.moe.routed_rank2.input_quant` | [nvfp4_quant](https://servingstudio.cs.washington.edu/kernels/nvfp4_quant.md) | `vllm_cuda` |
| `unified.local_nvfp4_moe.moe.routed_rank2.fused_moe` | [nvfp4_fused_moe](https://servingstudio.cs.washington.edu/kernels/nvfp4_fused_moe.md) | `flashinfer_trtllm_routed_sm100_fork` |
| `unified.local_nvfp4_moe.moe.sink.gate_up` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_nvfp4_moe.moe.sink.act_and_mul` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.local_nvfp4_moe.moe.sink.down` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_nvfp4_moe.moe.sink.gate_up` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_nvfp4_moe.moe.sink.act_and_mul` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.local_nvfp4_moe.moe.sink.down` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_nvfp4_moe.moe.routed_rank3.input_quant` | [nvfp4_quant](https://servingstudio.cs.washington.edu/kernels/nvfp4_quant.md) | `vllm_cuda` |
| `unified.local_nvfp4_moe.moe.routed_rank3.fused_moe` | [nvfp4_fused_moe](https://servingstudio.cs.washington.edu/kernels/nvfp4_fused_moe.md) | `flashinfer_trtllm_routed_sm100_fork` |
| `unified.local_nvfp4_moe.moe.routed_rank3.input_quant` | [nvfp4_quant](https://servingstudio.cs.washington.edu/kernels/nvfp4_quant.md) | `vllm_cuda` |
| `unified.local_nvfp4_moe.moe.routed_rank3.fused_moe` | [nvfp4_fused_moe](https://servingstudio.cs.washington.edu/kernels/nvfp4_fused_moe.md) | `flashinfer_trtllm_routed_sm100_fork` |
| `unified.local_nvfp4_moe.moe.sink.gate_up` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_nvfp4_moe.moe.sink.act_and_mul` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.local_nvfp4_moe.moe.sink.down` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.local_nvfp4_moe.moe.rs_sconv_ag_norm` | [rs_sconv_ag_residual_rms_norm](https://servingstudio.cs.washington.edu/kernels/rs_sconv_ag_residual_rms_norm.md) | `vllm_lamport_triton` |
| `unified.global_nvfp4_moe.attn.qkvr_proj` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.global_nvfp4_moe.attn.qkvr_prep` | [sconv_qkvr_prep](https://servingstudio.cs.washington.edu/kernels/sconv_qkvr_prep.md) | `vllm_triton` |
| `unified.global_nvfp4_moe.attn.rel_attention` | [rel_bias_attention](https://servingstudio.cs.washington.edu/kernels/rel_bias_attention.md) | `tml_fa4` |
| `unified.global_nvfp4_moe.attn.o_proj` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.global_nvfp4_moe.attn.rs_sconv_ag_norm` | [rs_sconv_ag_residual_rms_norm](https://servingstudio.cs.washington.edu/kernels/rs_sconv_ag_residual_rms_norm.md) | `vllm_lamport_triton` |
| `unified.global_nvfp4_moe.moe.router` | [gemm_fp32_output](https://servingstudio.cs.washington.edu/kernels/gemm_fp32_output.md) | `vllm_ll_bf16_auto` |
| `unified.global_nvfp4_moe.moe.gate_select` | [moe_sink_gate_select](https://servingstudio.cs.washington.edu/kernels/moe_sink_gate_select.md) | `vllm_triton` |
| `unified.global_nvfp4_moe.moe.topk_copy` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `torch` |
| `unified.global_nvfp4_moe.moe.sink.gate_up` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.global_nvfp4_moe.moe.sink.act_and_mul` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.global_nvfp4_moe.moe.sink.down` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.global_nvfp4_moe.moe.routed_rank0.input_quant` | [nvfp4_quant](https://servingstudio.cs.washington.edu/kernels/nvfp4_quant.md) | `vllm_cuda` |
| `unified.global_nvfp4_moe.moe.routed_rank0.fused_moe` | [nvfp4_fused_moe](https://servingstudio.cs.washington.edu/kernels/nvfp4_fused_moe.md) | `flashinfer_trtllm_routed_sm100_fork` |
| `unified.global_nvfp4_moe.moe.routed_rank0.input_quant` | [nvfp4_quant](https://servingstudio.cs.washington.edu/kernels/nvfp4_quant.md) | `vllm_cuda` |
| `unified.global_nvfp4_moe.moe.routed_rank0.fused_moe` | [nvfp4_fused_moe](https://servingstudio.cs.washington.edu/kernels/nvfp4_fused_moe.md) | `flashinfer_trtllm_routed_sm100_fork` |
| `unified.global_nvfp4_moe.moe.sink.gate_up` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.global_nvfp4_moe.moe.sink.act_and_mul` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.global_nvfp4_moe.moe.sink.down` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.global_nvfp4_moe.moe.sink.gate_up` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.global_nvfp4_moe.moe.sink.act_and_mul` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.global_nvfp4_moe.moe.sink.down` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.global_nvfp4_moe.moe.routed_rank1.input_quant` | [nvfp4_quant](https://servingstudio.cs.washington.edu/kernels/nvfp4_quant.md) | `vllm_cuda` |
| `unified.global_nvfp4_moe.moe.routed_rank1.fused_moe` | [nvfp4_fused_moe](https://servingstudio.cs.washington.edu/kernels/nvfp4_fused_moe.md) | `flashinfer_trtllm_routed_sm100_fork` |
| `unified.global_nvfp4_moe.moe.routed_rank1.input_quant` | [nvfp4_quant](https://servingstudio.cs.washington.edu/kernels/nvfp4_quant.md) | `vllm_cuda` |
| `unified.global_nvfp4_moe.moe.routed_rank1.fused_moe` | [nvfp4_fused_moe](https://servingstudio.cs.washington.edu/kernels/nvfp4_fused_moe.md) | `flashinfer_trtllm_routed_sm100_fork` |
| `unified.global_nvfp4_moe.moe.sink.gate_up` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.global_nvfp4_moe.moe.sink.act_and_mul` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.global_nvfp4_moe.moe.sink.down` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.global_nvfp4_moe.moe.sink.gate_up` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.global_nvfp4_moe.moe.sink.act_and_mul` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.global_nvfp4_moe.moe.sink.down` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.global_nvfp4_moe.moe.routed_rank2.input_quant` | [nvfp4_quant](https://servingstudio.cs.washington.edu/kernels/nvfp4_quant.md) | `vllm_cuda` |
| `unified.global_nvfp4_moe.moe.routed_rank2.fused_moe` | [nvfp4_fused_moe](https://servingstudio.cs.washington.edu/kernels/nvfp4_fused_moe.md) | `flashinfer_trtllm_routed_sm100_fork` |
| `unified.global_nvfp4_moe.moe.routed_rank2.input_quant` | [nvfp4_quant](https://servingstudio.cs.washington.edu/kernels/nvfp4_quant.md) | `vllm_cuda` |
| `unified.global_nvfp4_moe.moe.routed_rank2.fused_moe` | [nvfp4_fused_moe](https://servingstudio.cs.washington.edu/kernels/nvfp4_fused_moe.md) | `flashinfer_trtllm_routed_sm100_fork` |
| `unified.global_nvfp4_moe.moe.sink.gate_up` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.global_nvfp4_moe.moe.sink.act_and_mul` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.global_nvfp4_moe.moe.sink.down` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.global_nvfp4_moe.moe.sink.gate_up` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.global_nvfp4_moe.moe.sink.act_and_mul` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.global_nvfp4_moe.moe.sink.down` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.global_nvfp4_moe.moe.routed_rank3.input_quant` | [nvfp4_quant](https://servingstudio.cs.washington.edu/kernels/nvfp4_quant.md) | `vllm_cuda` |
| `unified.global_nvfp4_moe.moe.routed_rank3.fused_moe` | [nvfp4_fused_moe](https://servingstudio.cs.washington.edu/kernels/nvfp4_fused_moe.md) | `flashinfer_trtllm_routed_sm100_fork` |
| `unified.global_nvfp4_moe.moe.routed_rank3.input_quant` | [nvfp4_quant](https://servingstudio.cs.washington.edu/kernels/nvfp4_quant.md) | `vllm_cuda` |
| `unified.global_nvfp4_moe.moe.routed_rank3.fused_moe` | [nvfp4_fused_moe](https://servingstudio.cs.washington.edu/kernels/nvfp4_fused_moe.md) | `flashinfer_trtllm_routed_sm100_fork` |
| `unified.global_nvfp4_moe.moe.sink.gate_up` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.global_nvfp4_moe.moe.sink.act_and_mul` | [elementwise](https://servingstudio.cs.washington.edu/kernels/elementwise.md) | `triton` |
| `unified.global_nvfp4_moe.moe.sink.down` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.global_nvfp4_moe.moe.rs_sconv_ag_norm` | [rs_sconv_ag_residual_rms_norm](https://servingstudio.cs.washington.edu/kernels/rs_sconv_ag_residual_rms_norm.md) | `vllm_lamport_triton` |
| `unified.lm_head` | [single_gemm](https://servingstudio.cs.washington.edu/kernels/single_gemm.md) | `torch_linear_vllm` |
| `unified.logits_all_gather` | [all_reduce](https://servingstudio.cs.washington.edu/kernels/all_reduce.md) | `nccl` |

## Simulations that run it

- [`Inkling-Small-NVFP4/inkling_vllm_nvfp4_rel_attn_moe_chunked_prefill`](https://servingstudio.cs.washington.edu/simulate.html?preset=Inkling-Small-NVFP4%2Finkling_vllm_nvfp4_rel_attn_moe_chunked_prefill): unified deployment, 2 configurations

## API

- This deployment in the model list: https://servingstudio.cs.washington.edu/api/public/v1/models
- The kernels of one configuration: https://servingstudio.cs.washington.edu/api/public/v1/models/Inkling-Small-NVFP4/inkling_vllm_nvfp4_rel_attn_moe/tree?workload=diverse_100

---

This page as Markdown: https://servingstudio.cs.washington.edu/models/Inkling-Small-NVFP4/inkling_vllm_nvfp4_rel_attn_moe.md. A guide to the whole site for AI agents: [llms.txt](https://servingstudio.cs.washington.edu/llms.txt).
