All BPE, but implementations diverge wildly. llama.cpp's src/llama-vocab.cpp enumerates LLAMA_VOCAB_TYPE_BPE / SPM / WPM / UGM, each taking a slightly different path:

• GPT-2 BPE (Llama-3 / Qwen2 / Mistral family): byte-level, pre-tokenize with a long regex, greedy merge by rank. Most common.
• SentencePiece BPE (Llama-1/2 / Gemma): no pre-tokenize; BPE runs over the raw text with whitespace mapped to a special ▁. "Language-agnostic out of the box" is its pitch.
• Tiktoken (GPT-3.5+ / GPT-4): same family as GPT-2 BPE, but OpenAI rewrote it in Rust with a faster algorithm that replaces the priority queue with direct rank-table lookups, several × faster. llama.cpp doesn't link Tiktoken but reads its vocab.
• Tekken (next-gen Mistral): Tiktoken derivative; tighter vocab, better Chinese / code efficiency.

For an inference engine, tokenizer speed is almost never the bottleneck — compute lives on the GPU. But tokenizer correctness is a minefield: case, whitespace, emoji, rare unicode, chat-template specials like <|im_start|>. Mess one up and the model literally can't see how it was prompted. llama.cpp's BPE has been bug-fixed in dozens of PRs over the years — this is the easiest place in the whole stack to silently regress.

The pre-tokenize regex from GPT-2's 2019 release is famously cryptic-looking. What it does: slice any utf-8 string into "chunks" — whitespace, punctuation, digit runs, letter runs, each their own segment. Take a look:

llama.cpp uses pcre2 to run this regex (regex_split in src/llama-vocab.cpp). Notice: it uses \p{L} (any Unicode letter) and \p{N} (any Unicode digit) — it's Unicode-aware. So Chinese, Arabic, emoji all chunk correctly. This is why Llama-3 handles Chinese far better than Llama-2 — same regex but with a byte-level vocab; Chinese doesn't fragment into "weird 0xE4-type byte slices".

The "digits chunked at most 3 at a time" rule matters a lot: prevents the model from treating "1234567" as one token. Instead it splits into "123","456","7" — way better for arithmetic accuracy. GPT-4's infamous "can't compute 17+28" issues are noticeably better in Llama-3, partly because of this pre-tokenize improvement.

In the two weeks after Llama-3's April 2024 release, llama.cpp / Hugging Face / Ollama all had tokenizer bugs. Three classics:

1. Double BOS (issue #7088 et al.): Llama-3 chat templates already include <|begin_of_text|> (BOS) at the start; but llama.cpp defaulted to auto-adding BOS when loading new models — first token becomes BOS BOS. The model sees "two start markers", behaves oddly, quality drops visibly. Fix: model metadata add_bos_token = false.
2. Special tokens treated as normal BPE (issue #6920): users sometimes paste prompts containing literal <|eot_id|> (copied from logs). Early tokenizer treated it as a regular BPE string — not as the special token id 128009. The model interpreted "user mentioning eot_id in text" as "this turn ended" and stopped generating immediately. Fix: scan for known special tokens before BPE.
3. Chat template mismatch: Llama-3 base and instruct share the same tokenizer but different chat templates. Devs downloaded base, loaded the instruct template — model produced "systemwide" gibberish. Fix: GGUF metadata gained a chat_template field; llama.cpp picks up the model's own template at load.

None of these are algorithm bugs; they're "seam-layer" human errors. But the user-perceived impact is "this model got dumber". So tokenizer is the most overlooked, highest-damage piece of the inference stack — one byte wrong and the whole model visibly degrades.

"Embedding turns a token id into a 4096-dim vector" is the entrance; "LM head turns a 4096-dim vector back into a token id" is the exit (Ch.13). Shape-wise, the two are literal transposes: both are [n_vocab, n_embd]; only the matmul direction differs.

So in 2016 Inan et al. proposed "weight tying": have embedding and LM head share the same weights. Implementation: LM head uses embedding.T instead of an independent matrix. Benefits:

• Saves ~1 GB: Llama-3-8B's embedding is 1 GB; tying shrinks the full model by 1 GB (12% of params).
• Input and output learn together: a single token's input semantics and output prediction stay aligned. Papers measure perplexity improvements.

Who uses tying: GPT-2, Llama-1, Llama-2, Mistral, Qwen2, Gemma all do. Llama-3 doesn't — Meta's paper doesn't explain; the industry guess: once vocab grows to 128k, untied gives the model more expressive freedom; also untied lets prompt-side embedding (retrieval vectors) and generation-side embedding (semantic generation) each specialize.

llama.cpp auto-detects via GGUF metadata: if the output tensor is absent, it falls back to token_embd.T. So inference code doesn't need to know whether weights are tied — but training frameworks must explicitly set tied_weights=True, otherwise the two weights train independently and the tying benefit is lost.

4096 sounds like a lot, but LLMs actually use far fewer dimensions. Intrinsic dimension research: SVD on LLM hidden states; the top 90% variance is typically captured by ~500-1000 principal components. The remaining 3000+ dims are redundant — they exist mainly to provide optimization room during training; most are near-zero at inference.

This fact has directly driven several inference optimization directions:

• LoRA (Low-Rank Adaptation): finetune only a low-rank ΔW = A·B; both A and B are ~16-64 dim. A 7B model's LoRA adapter is just tens of MB — because the model's "actual information capacity" is low-dimensional.
• Pruning: drop dimensions with near-zero activation. Wanda (2024) shows 50% pruning barely affects accuracy — but engineering complexity keeps it rare in production.
• SVD-based quantization: SVD the weight matrix; keep top-k singular values; quantize the low-rank approximation. GPTQ is partly based on this idea.

So 4096 isn't "exactly enough" — it's more like "training-time slack for optimization". The "effective dimension" at inference is much smaller — which is the root reason every low-rank compression trick works so well on LLMs.

A user sends an OpenAI-compatible HTTP request to a vLLM server. Before becoming a ubatch, it passes through several stages:

1. HTTP receive (fastapi async handler): JSON parse + schema validate · 1-2 ms
2. Tokenize (C03, on CPU): split prompt into token ids · 1-5 ms depending on length
3. Sequence enqueue: create a SequenceGroup, push to scheduler's WAITING queue · μs
4. Wait for scheduling: until Scheduler.schedule() picks it · 0-100 ms depending on load
5. Become ubatch: scheduler packs this seq + others into a ubatch · μs
6. forward + sample: walk the graph · ms to hundreds of ms
7. Detokenize + stream: convert tokens back to strings, send to client · ~0.5 ms per token

Note step 4 — "wait for scheduling" is a non-negligible source of latency. Under high load a new request can wait tens of ms before being batched — what the user sees is high TTFT. "Scheduling latency" is a TTFT component just as much as prefill time.

Priority scheduling: introduced after vLLM 1.0, lets requests have priority levels (high-pri jumps the queue). Production use: distinguishing free vs paid tier, interactive chat vs async batch jobs. OS-scheduler thinking reused for LLM serving.

RoPE was trained seeing positions up to n_ctx_orig (8192 for Llama-3). Feeding a 128K prompt straight in doesn't work — angles mθ for unseen m have wrapped around several times and the model has no idea.

YaRN (and its predecessor NTK-aware) compresses the high-frequency dims' frequencies so they "stretch longer". In ggml_rope_ext this is the rope_factors param — a d/2 vector, each freq multiplied by a factor ≤ 1. Low-freq dims (long range) barely change; high-freq dims (short range) get compressed.

The trick lets you push an 8K Llama-3 to 32K context without retraining. The cost is reduced precision at long distance — fine for "retrieval" tasks, weak for "logical reasoning spanning 128K" workloads. Those need actual long-context training.

RoPE looks magical — why not just add a position embedding? Why rotate? Its real beauty: Q·K^T naturally becomes a "function of position difference" — and there's a mathematically unique construction for that. Step by step:

View each pair of Q / K dims as a point in the complex plane:
q_m = (q_a, q_b) → complex q = q_a + i·q_b
k_n = (k_a, k_b) → complex k = k_a + i·k_b

RoPE rotates each position m by mθ:
q'_m = q · e^(imθ)
k'_n = k · e^(inθ)

The magic moment: look at the inner product Q · K (which is attention's score):

⟨q'_m, k'_n⟩ = Re(q'_m · k'_n*)
= Re(q · e^(imθ) · k* · e^(-inθ))
= Re(q · k* · e^(i(m-n)θ))

That e^(i(m-n)θ) — the result depends only on m-n, no longer on m and n individually. Absolute position vanishes from the inner product; only relative position remains. This is the entirety of RoPE's mathematical beauty: a plain rotation makes attention automatically see only relative distance.

And this property cannot be achieved by simple "add a position embedding" — addition leaves three crossterms (q·pos_m + pos_n·k + pos_m·pos_n) in the inner product expansion, irreducible. When Su Jianlin discovered this construction in 2021, it was a stroke of pure math brilliance.

In practice: d_head dims are paired into d_head/2 complex pairs; each pair rotates by a different frequency. Frequencies θ_i = 10000^(-2i/d_head) decay from 1 down to 1/10000, so "wavelengths" span ~6 tokens to ~62800 tokens — covering every relative-distance scale. This is why d_head must be even — an odd dim would leave one unpaired axis with no complex partner.

The 10000 in 10000^(-2i/d) isn't a physical constant. It's a design parameter controlling "how far the model can perceive position differences". Intuition:

• base too small (e.g. 100): frequencies decay fast, low-freq dims quickly wrap — long-range info lost.
• base too large (e.g. 1000000): frequencies decay slowly, all dims care about short range — model can't distinguish "5K away" from "50K away".
• base = 10000 (original Transformer pick): empirical compromise — max wavelength ≈ 2π × 10000 ≈ 62800, roughly "a few times the training seq len" — leaving some extrapolation room at inference.

Llama-3 bumped base from Llama-2's 10000 to 500000 — to support 128K context. Larger base stretches all wavelengths so the model never "wraps" within 128K. This is the key knob (paired with long-context training data) enabling Llama-3's native 128K.

DeepSeek-V3 goes further: base 1,000,000, supporting 128K-256K. Often paired with YaRN — base sets the length budget, YaRN dictates how to extrapolate beyond training length at inference.

RMSNorm isn't the only normalizer in Transformers. Three mainstream variants differ in subtle ways that affect training stability; at inference the shape is the same but some properties matter:

Key differences:

• LayerNorm → RMSNorm: half the params (no beta), one less op (no mean), ~20% faster. Theoretically RMSNorm assumes input is already centered (an assumption that empirically holds in residual nets).
• QK-Norm: not in the Llama paper, but SOTA for long-context training — at large d_head, Q·K^T magnitudes blow up, softmax saturates, gradients vanish. RMSNorm on Q and K each pins them to a fixed-norm regime, stabilizing training.

At inference, RMSNorm is ggml_rms_norm; QK-Norm is added manually in build_attn. Either way inference cost is tiny (~5K FLOPs per token), but training impact is significant — pick the wrong norm and a large model can suddenly diverge at step 10000.

Looking at llama.cpp's KV tensor declarations reveals a curious detail: K and V have different shapes.

Not a typo. QKᵀ and P·V in attention have opposite memory access patterns:

• QKᵀ: Q is [n_tokens, d_head], K is [n_ctx, d_head]. For Q × K.T, K must be organized "row = token" — K cache is laid out this way, each row a token, sequential reads are fast.
• P·V: P is [n_tokens, n_ctx], V is [n_ctx, d_head]. For this we want V organized "column = token" — so P's i-th column times V's i-th row accumulates sequentially.

So V is physically stored transposed — ggml_cpy at write time does a transpose; reads stream sequentially. Looks minor, measurably 30% faster attention — GPU memory access is brutally sensitive to "sequential vs strided".

This is also why fp8 KV cache (Ch.18) is easier on K than V: after V's transpose, fp8 outliers spread across tokens, requiring per-head per-block scales to preserve precision. vLLM has a lot of CUDA kernel code dedicated to exactly this.

llama.cpp's find_slot returns -1 when all slots are occupied — the caller must do something. Three mainstream strategies:

1. Hard cutoff: error out with "context full", refuse to continue. Old llama.cpp behavior.
2. Sliding window (llama_kv_self_seq_rm): drop the earliest N tokens' KV, write new tokens into the freed slots. Problem: model totally forgets the beginning — and in chat "the beginning is the system prompt"; lose it and behavior breaks.
3. StreamingLLM (Han Song's group, 2023): keep the first 4 tokens' KV (attention sinks) + the most recent N tokens' KV; discard the middle. Research found attention puts anomalously high scores on the first few tokens (the "sink" phenomenon). Keep those and the model still "holds the conversational context". A 4K-context model can stream 4M tokens this way.

In real production, the more common move is to actively evict whole conversations before the cache fills — sessions inactive for hours get their entire KV freed. vLLM's BlockSpaceManager.swap_out goes further: park inactive sessions' KV in CPU memory, swap_in when the user comes back. OS swap, transplanted into LLM inference.

Take Llama-3-70B (GQA-8 · 80 layers · n_embd_head_v = 128). Weights: 140 GB at fp16, ~40 GB at Q4_K_M. Per-user, per-8K KV cache is ~2.5 GB.

Serving on 8 × H100 (80 GB):

• Weights 40 GB (quantized) · ~1 card's worth
• Activation buffers + CUDA contexts ~20 GB
• Remaining 640 − 40 − 20 = 580 GB for KV cache
• Per user 2.5 GB · theoretical max 232 concurrent users at 8K context
• Reserve ~25% for fragmentation and peaks → ~170 concurrent in practice

One H100 node (8 cards) on AWS lists at ~$32/h. Fully utilized at 170 concurrent users, 1 query/min each, that's ~10000 queries/hour · ~$0.0032 per query. But KV cache also holds memory when the user is typing; real utilization is much lower. This is why OpenAI / Anthropic post 50-70% gross margins that look high but are mostly eaten by idle KV cache.

This is the actual commercial meaning of MLA (Ch.11), PagedAttention (Ch.17), prefix caching (Ch.19), KV swap-out, every "save KV / share KV / move KV" technique: cut KV usage in half and service density doubles, gross margin gains 20 points.

StreamingLLM (Han Song team, 2023) found a counterintuitive result: always keep the first 4 KV cache tokens and a model handles arbitrarily long streams; drop them and the model breaks instantly. These 4 tokens aren't a "system prompt" — could be <|begin_of_text|> + any three early text tokens. Why so magical?

Investigation revealed a universal phenomenon: in Transformer training, attention tends to assign large amounts of score to the first few tokens of the sequence — even when semantically irrelevant. Reason: the softmax formula exp(QK)/Σexp requires all tokens' probabilities sum to 1 — but in some queries' view, "nothing in the history is relevant". The model has to dump probability somewhere; early tokens, being always present during training, became the natural "overflow drain".

So these first few tokens are called attention sinks — their KV carries no meaningful info, but must exist, or other tokens' attention probability distributions break, score ranges go haywire, generation quality collapses.

This finding produced three things:

• StreamingLLM: always keep first 4 + most recent N KV; toss the middle — enables infinite streaming.
• Explicit sink tokens at training time: Mistral et al. put a <sink> special token at training, providing a natural overflow drain at inference.
• KV cache compression (H2O, scissorhands): given only the first few + most recent matter, the middle KV can be compressed or dropped more aggressively.

This area was hot in 2023-2024 but not widely deployed in production — StreamingLLM is more used in non-chat scenarios (pure-stream log processing); chat is bounded enough that attention-sink optimization's marginal gain is small.

Besides quantization (C18 fp8 KV), another way to shrink KV cache is proactively drop parts of it — keep "important" tokens, drop "unimportant" ones. How to define "important" is the topic of a series of papers:

• H2O (Heavy-Hitter Oracle, 2023): maintain per-token cumulative attention received. At generation, keep attention sinks + recent tokens + history heavy hitters; drop tokens "never attended to". ~50% KV dropped with near-zero PPL change.
• Scissorhands (2023): similar to H2O but with finer "importance score" tracking, allowing "dynamic in/out" of the cache.
• SnapKV (2024): instead of dropping during generation, after prompt prefill completes, use "attention from the prompt's last token to each history token" to pick top-N key tokens; keep only their KV. Simple but effective — long-prompt KV cache cut by 80% with no accuracy loss.
• PyramidKV (2024): observation: different layers need different amounts of long-range context — early layers need more context, deep layers need only a few key tokens. Different cache sizes per layer, ~70% total savings.

These techniques are orthogonal to quantization — they stack. fp8 KV halves memory + H2O drops half the tokens = total KV down to 1/4. This is the 2024 long-context inference optimization combo.

llama.cpp main hasn't integrated these (still simple ring buffer); experimental branches have. vLLM added SnapKV as an option in 2024. This is a fast-evolving field; no standard answer yet.

Naive softmax requires seeing all elements: first compute max (for numerical stability), then exp-sum (the normalization denominator), then divide. This is why attention's intermediate S = QKᵀ must be fully materialized to HBM before softmax — classic O(N²) memory.

FlashAttention's core math is online softmax (Milakov & Gimelshein, 2018): update incrementally; never need to see everything first. Maintain two accumulators that update as new elements stream in:

Before reading x_i · already processed i-1 elements:
m_{i-1} = max(x_1, ..., x_{i-1}) // running max
d_{i-1} = Σ exp(x_j - m_{i-1}) // running denominator

Read x_i, update:
m_i = max(m_{i-1}, x_i)
d_i = d_{i-1} · exp(m_{i-1} - m_i) + exp(x_i - m_i)

The exp(m_{i-1} - m_i) factor is a correction coefficient — all the prior exps were computed against the old max; when the max grows, all old values get uniformly shrunk a bit. After the update, d_i is "the correct normalization denominator for just the elements seen so far".

FlashAttention couples this online softmax with block matrix multiply: Q and K/V are tiled; each iteration loads one tile pair into SRAM. In SRAM, compute the tile's local S = QK^T, online-softmax-accumulate it onto the output O, discard, load next pair — the full S matrix is never materialized.

Reading the pseudocode you can feel FlashAttention's elegance: every "need to see the whole row first" dependency is dissolved by those two correction terms, alpha and beta. The inner loop needs only the current tiles in SRAM and a few scalars (m, d). After the outer loop, O_i is the correct softmax(QKᵀ)V[i] — bit-equivalent to the naive algorithm (modulo fp accumulation noise).

Measured win: on A100, 8K seq len, d=128 attention, naive uses ~70 GB HBM traffic (materialize S + back-and-forth). FlashAttention uses ~7 GB — 10× bandwidth saved, translating to ~5-7× real speedup. Max workable seq length grows from ~4K to ~64K.

FlashAttention has shipped two major versions since 2022, each tightly coupled to a hardware generation:

• FlashAttention v1 (2022): the pseudocode above is v1. On A100 / V100, attention MFU went from 5-15% to ~50%. The first time the field collectively realized "attention is memory-bound, not compute-bound".
• FlashAttention v2 (2023): swapped loop order — v1 was "outer Q block, inner K block" with some redundant work; v2 is "outer K block, inner Q block", math-equivalent but better aligned with GPU memory hierarchy. Also splits work finer, lifting SM occupancy. ~2× over v1 on A100, ~70% of theoretical peak. The mainstream impl 2024-2025.
• FlashAttention v3 (2024): Hopper-specific (H100), leveraging (1) WGMMA (warp-group async matrix multiply-accumulate); (2) TMA (Tensor Memory Accelerator hardware prefetch); (3) fp8 compute paths. 1.5-2× over v2 on H100, ~75% of theoretical peak.

An interesting legacy issue: vLLM started on FA v2, then added PagedAttention with its own kernel, then FA v3 was rewritten to support paged KV. Kernel implementations are engineering-heavy — same math, different hardware, separate writes. This is why "new hardware → full inference stack support" usually takes 6-12 months — model porting isn't slow; kernel porting is.

All of FlashAttention's wins come from "S not materialized to HBM". But at decode, Q has only 1 row — S = QK^T is just [1, N_kv], a few tens of KB to materialize, negligible against model weights. So FA's win at decode is essentially zero; the win is all on prefill attention.

This is why vLLM / TensorRT-LLM ship decode-specific attention kernels — they don't use FA but a "pure streaming" algorithm: one query token at a time, but reading KV via PagedAttention's block indexing. This is exactly what the paged_attention_v2 CUDA kernel does.

FlashDecoding (Tri Dao's team, 2023): tackled "even decode under-saturates the GPU". The idea: split KV cache along the sequence axis into chunks; each SM independently computes "this chunk's contribution to the current Q's attention"; merge afterward. SM occupancy goes from ~30% to ~70%, ~3-5× speedup on long-context decode. Single biggest 2024 long-context decode optimization.

MLA looks like magic; the key is "absorb the decompression matrix W_uk into Q's projection W_q". Expanded, it's just a chain of perfectly legal linear algebra.

Naive attention (single head, omitting scale and softmax):

O = softmax((Q · Kᵀ) / √d) · V
Q ∈ ℝ^[n, d] · K ∈ ℝ^[m, d] · V ∈ ℝ^[m, d]

MLA factors K = c_kv · W_uk (c_kv ∈ ℝ^[m, d_c], W_uk ∈ ℝ^[d_c, d]). Substitute into Q·Kᵀ:

Q · Kᵀ = Q · (c_kv · W_uk)ᵀ
= Q · W_ukᵀ · c_kvᵀ
= (Q · W_ukᵀ) · c_kvᵀ

And Q = h · W_q (h is the input hidden state), so:

Q · W_ukᵀ = h · W_q · W_ukᵀ = h · W_q'
where W_q' := W_q · W_ukᵀ ∈ ℝ^[d_model, d_c]

That's absorption: W_q' is precomputed once at model load; at runtime Q' = h · W_q' is one normal matmul, same cost as vanilla attention. Then:

Q · Kᵀ = Q' · c_kvᵀ
attention scores directly from the latent c_kv · no "decompress K" needed

Same trick on V: V = c_kv · W_uv, absorb W_uv into the output projection W_o, get W_o' := W_uv · W_o.

Result: KV cache stores only d_c-dim c_kv. Two matmuls at runtime but no FLOPs increase — you've just regrouped "decompress K · attention · decompress V · project O" into "compute Q' · attention · project O'". That's all absorption is: pure associativity-driven re-grouping of linear algebra.

Even better: at large batch size, W_uk / W_uv are never computed — they live only on the weight side, invisible on the inference path. A rare case of model architecture getting multi-fold memory savings from a pure geometric trick.

The absorption proof above had a hidden assumption: Q and K are linear functions. RoPE is not linear — it's a rotation; Q and K are each multiplied by a position-dependent rotation matrix.

Concretely, RoPE makes K(m) = R(m) · K_orig, where R(m) is the rotation for position m. To absorb W_uk into W_q you'd need to absorb R(m) too — but R(m) is per-token, different each time; you can't bake it into precomputed weights.

DeepSeek's fix: split K in two:

• c_kv (non-RoPE, d_c = 512 dims): goes through the MLA latent, gets absorbed;
• k_pe (RoPE-part, d_rope = 64 dims): stored raw (post-RoPE), bypasses the latent.

At attention time the two parts are concatenated to form K_final. So MLA's KV actually stores a [d_c + d_rope] = 576-dim mixed vector — most dims enjoy absorption's compression, a few take the standard RoPE path. That's where "576 = d_c 512 + d_rope 64" earlier came from.

Same design on Q: q_nope (non-RoPE) + q_rope (RoPE). So DeepSeek-V2/V3 attention nodes have a few extra concat operations vs. Llama, but KV cache shrinks enough that bandwidth gains dwarf concat cost. Model architecture co-designed with the inference engine — reading the paper alone isn't enough; you need to read it against the ggml cgraph nodes to see the whole picture.

671B-param DeepSeek-V3 won't fit on a single GPU — 8×H100 = 640 GB needs quantization to fit. Tensor Parallelism (cut along d_model) is the dense-model default but doesn't suit MoE: every expert is an independent FFN, cutting it horizontally helps little. The natural axis for MoE is expert parallelism — spread the 256 experts across cards, 32 experts per card.

This immediately introduces a new problem: a token's top-8 experts almost certainly don't all live on one card. Every FFN layer must broadcast this token's hidden state to the cards holding its selected experts, compute, then gather back — that's all-to-all communication, the biggest engineering challenge in MoE training/inference.

The bottleneck is the two all-to-alls in steps 2 and 4. 8-card NVLink 4.0 is ~450 GB/s one-way; each token hidden state is 7168 × 2 bytes ≈ 14 KB. Two all-to-alls for a 1024-token batch = ~30 MB ≈ 67 μs. Sounds small, but 32 layers × 67 μs = 2 ms of pure communication. 20-30% of MoE decode latency can be communication — which is why DeepSeek wrote their own high-performance all-to-all library DeepEP, and why "training MoE is much more engineering than training a dense model of the same FLOPs".

The classic MoE load-balancing trick is an auxiliary loss (L_aux): penalize "some experts get picked too often". But aux loss has two side effects:

• It tugs against the main task loss — sacrificing some quality for balance.
• Its weight is hard to tune — too small means imbalance, too large means quality drop.

DeepSeek-V3's auxiliary-loss-free method adds a dynamic bias b_i per expert. At routing time:

top-k selection by (gate_logit_i + b_i)
but the final "routing weight" still uses raw gate_logit_i

Clever bit: b_i affects who gets picked, not the weight of the pick. During training they track per-expert token counts; overloaded experts get b_i decreased (picked less next time), underloaded experts get b_i increased. It's external feedback, not a gradient signal — the main loss stays clean.

llama.cpp's build_moe_ffn supports this via the expert_bias arg: when loading DeepSeek-V3, the bias is part of the saved weights, added directly to gate logits at runtime. Training-time dynamic adjustment becomes inference-time static bias — another "training pays, inference takes" idiom.

MoE isn't one fixed recipe — each lab picks differently on number of experts / top-k / shared experts / gating function. A side-by-side of current production models:

Notable evolution lines:

• n_experts: 8 → 256: early Mixtral 8 large experts; recent DeepSeek 256 small ones. The trend is "finer grain" — more but smaller experts give the router more combinatorial room, better capacity utilization.
• Return of the shared expert: early Mixtral had no shared expert, all FFN went routed. From DeepSeek-V2 onward the "shared expert" idea returned (also in Switch Transformer earlier) — some "generic" knowledge is needed by every token; better to always activate a shared FFN than have the router repeatedly select for it, freeing routed experts to specialize.
• softmax → sigmoid: DeepSeek-V3 and Llama-4 both switched from softmax to sigmoid routers. Softmax forces routing weights to sum to 1 — all experts "fight over one pie". Sigmoid gives each expert independent "should I fire" probabilities, making top-k more stable. New consensus in 2024 MoE papers.
• top-1 vs top-k > 1: Mixtral / DBRX use top-2 or top-4 — a token visits multiple experts. Llama-4 boldly uses top-1 — lower comm cost, friendlier to large-scale distributed deployment, at the cost of weaker model compositionality.

This table shows the field's methodological convergence: people explored different points but all moved toward "more small experts + shared expert + sigmoid router + aux-free balance". The "textbook MoE" of late 2024 looks very different from Switch Transformer's "textbook MoE" of 2023.

In MoE training, each expert receives a variable number of tokens per batch — GPU memory allocation is awkward; you don't know how much to reserve per expert. Early Switch Transformer introduced "capacity factor" — cap each expert at 1.5 × (n_tokens / n_experts); overflow tokens are dropped (skip FFN via residual).

Acceptable in training; unacceptable in inference — drop one token and generation breaks. So inference MoE implementations don't cap capacity: Mixtral / DeepSeek let each expert process all tokens routed to it. Cost: uneven per-expert token counts — some get 50 tokens, others 5. This makes grouped GEMM load-imbalanced; some SMs finish early and idle.

This is the root reason "MoE inference speedup falls short of the theoretical active-ratio". 671B/37B should be 18× compute saving; in practice ~10×. The 5-8× loss is in load imbalance + comm overhead. vLLM and SGLang invest heavily in expert grouping (placing hot experts on different cards) and dynamic scheduling, but it's an open problem.

Llama-3's 128k vocab + 1 GB LM head still fits one H100. But multilingual models (NLLB, XLM-R) push vocabs to 250k-1M, LM head balloons to 4-10 GB. Large model + large vocab → LM head becomes a tensor that doesn't fit on one card.

The fix: vocab-parallel sharding. Shard LM head along the vocab axis across cards; each card computes logits for one slice. Under TP=8, card 0 handles vocab[0:16000], card 1 handles [16000:32000], etc. Sampling then needs one cross-card aggregation (all-gather all logits to one card to sample, or local top-k per card then aggregate).

This is distinct from "shard along d_model" TP — TP cuts input/output feature dims in attention/FFN; vocab is a new parallel axis. So vocab-parallel is conceptually a fourth parallelism dimension (alongside TP/PP/EP). Rarely standalone in production, usually layered with TP. Megatron-LM's VocabParallelEmbedding class does this.

A related trick: logit soft-capping (Gemma). If LM head output is unbounded, some token logits can blow up to ±100, softmax degenerates to one-hot, sampling loses diversity. Wrap LM head with tanh(x / cap) × cap (cap ~30) to bring it back to [-cap, cap] — stable, leaves room for probability distribution. llama.cpp implements via f_logit_scale in build.

Ch.18's Q4_K_M = "mixed precision" leaves LM head at Q6_K instead of quantizing to Q4. Worth unpacking why.

Quantization errors in regular layers (attention / FFN) get smoothed over by subsequent layers: a 5% weight error, after 32 layers of transformer flattening, contributes only ~0.1% to the final hidden state. LM head is different — it's the last layer; no transformer behind to smooth. Its output is directly sampling's input.

Worse: LM head is [n_embd × n_vocab] with n_vocab = 128k independent rows. One weight error affects just one vocab id's probability — but which id is random. If the misquantized id happens to be a high-frequency token ("the", " ", "的"), generation quality drops off a cliff.

Measurements: Llama-3-8B with LM head quantized to Q4 alone → PPL up ~8% (worse than the whole model at Q4); keeping it at Q6_K → PPL up only ~0.5%. 1 GB more memory for 6% accuracy — clearly worth it.

General principle: layers near the model's IO boundary are more quant-sensitive. LM head and embedding are "boundary layers" — quantize carefully. Middle attention/FFN is much thicker, more tolerant. Understand this and you'll see why GGUF quantization is so fiddly (different bits per layer) — it's a per-layer empirical strategy crafted from extensive testing.

Top-p has a widely known but rarely articulated issue: its cutoff isn't adaptive. Top-p=0.9 keeps too many in high-certainty situations — if the next token is "." with p_max = 0.85, top-p=0.9 still pulls in one or two ~0.05 alternatives that may be semantically unrelated, pure temperature noise. Top-p=0.9 keeps too few in high-uncertainty situations — if the top 20 tokens all hover 0.04-0.06, top-p=0.9 cuts at ~18 and drops legitimate 19th and 20th candidates.

Min-p (2024 paper Min P Sampling: Balancing Creativity and Coherence at High Temperature) fixes this. Instead of cumulative probability, use "relative-to-max": keep all tokens with p ≥ min_p × p_max.

• High certainty (p_max = 0.85), min_p=0.1 → threshold = 0.085 — almost only the top token.
• High uncertainty (p_max = 0.06), min_p=0.1 → threshold = 0.006 — keep all ≥ 0.006 — maybe 30 tokens — giving sampling enough variety.

Result: better for high-temperature creative writing + avoids low-prob noise tokens. llama.cpp shipped support early; vLLM added it in 2024. "The 2024 default sampler is min_p, not top_p" is community consensus now.

"Repetition penalty" (in C14's main table) is the earliest anti-loop sampler, but too blunt — discounts any token that appeared, often killing necessary repeats (commas, spaces, "the"). The 2024 community (mostly KoboldAI / SillyTavern circles) came up with two finer-grained alternatives:

• DRY (Don't Repeat Yourself, 2024): penalize sequences, not single tokens. Algorithm: scan history; find tokens whose selection would form "an n-gram already seen"; apply exponentially-decaying penalty (longer repeats → heavier). The model avoids repeating whole phrases but keeps natural function-word repeats.
• XTC (Exclude Top Choices, 2024): an anti-sampler — actively chop the highest-probability tokens to force the model to pick "second-best-but-still-reasonable" alternatives. Use during creative writing to ensure the model "doesn't take the most obvious path". Combined with high temperature, dramatically boosts diversity in fiction.

Neither sampler is in mainstream academic papers — they're pure llama.cpp-community engineering folklore. They show that the sampling design space is far from exhausted. Research focuses on the model; sampling is the layer closest to user experience, and small tweaks can change a lot.

Mirostat is a strange-but-elegant sampler. Its goal isn't "cut low-prob tokens" but "keep generation surprise at a target setpoint" — specifically, average -log(p) per generated token at tau (user setpoint, e.g. 5.0).

It's literally a feedback controller:

• Set tau — target "perplexity" (technically log-perplexity equivalent).
• Maintain a dynamic mu — it's like an effective top-k cap.
• After each token, compare its actual surprise to tau: too high → shrink mu (tighter cap); too low → grow mu (more candidates).
• Equivalent to a PID controller regulating the model's "uncertainty output" toward tau.

User feel: tau=3 → cautious, conservative (every token "certain"); tau=8 → bold, creative (every token "slightly surprising"). More semantically controllable than temperature — temperature is input, perplexity is output; Mirostat drives the output directly.

llama.cpp ships mirostat v1 / v2, each a single sampler in the chain. Notably better than fixed temperature for "maintain consistent tone" scenarios (fiction, roleplay) — it actively corrects local perplexity drift.

Before 2018's BERT, all seq2seq translation / summarization used beam search: maintain k candidate sequences; at each step expand each by all possible next tokens; keep the top k by probability. No dice rolling; pure heuristic search for the "globally most likely sequence".

But LLM inference almost never uses beam search:

• Speed: beam_width=4 means 4× the KV cache; decode drops to 1/4 the speed.
• Weird outputs: searching for "global optimum" leads to excessive safety — LLM beam search routinely produces "I am an AI language model and I cannot" boilerplate because that has the highest cumulative probability.
• Diversity: sampling N times (temperature + top-p) + picking the best empirically beats beam search. That's why ChatGPT's API has n.

Beam survives in: translation (where "correct" is relatively objective); speculative decoding's draft model (top-k candidates); code completion (under syntactic constraints, beam is steadier). But chat / creative writing all uses sampling. This is the Transformer-era sampling philosophy: model is good enough that diversity beats forced optimality.

OpenAI's API accepts stop=["\n\n", "User:", "###"] — model output stops the moment it matches these strings. Looks simple; three traps:

1. Strings don't align to tokens: stop is strings, output is tokens. "User:" may span multiple tokens — "Use"+"r:". So the engine must after every token, re-concat the last N tokens' strings and check suffix-match.
2. What if it's already past?: say stop is "User:" and the model just emitted "User"; next token might be ":" → triggers; might be " feedback" → doesn't. But "User" already streamed to the client. If stop triggers, you either don't stream ("buffer until resolved", latency up) or truncate the streamed output (you sent "User" — can't unsend). OpenAI picked "no streaming until stop is resolved" — that's why you'll observe streaming API occasionally "hitching" before continuing: it's the engine resolving stop.
3. Performance: per-token suffix matching is O(stop_len × n_stops); 20 stops totaling 200 chars adds μs per token. In batched serving this is not negligible. vLLM uses an Aho-Corasick automaton to scan all stops in one O(1) amortized pass.

In llama.cpp this lives in common_sampler::stop_strings; matching happens in the main loop. Many wrapper libraries get this wrong — a common bug: stop triggers but already-streamed tokens aren't rolled back, and the client UI shows content that shouldn't have appeared.

Ch.23 covered "format conversation into the model's preferred string" — the forward direction. The reverse direction — parsing model output back into structured fields — matters equally. Function calling especially: the model might emit:

This output has four things to handle separately:

• The leading <|start_header_id|>...<|end_header_id|> is the role header — strip it.
• The body "I'll check the weather for you." is normal output — stream to client.
• After <|python_tag|> is a tool call — parse JSON, trigger external function.
• <|eom_id|> (end of message) instead of <|eot_id|> (end of turn) — meaning "I'm done with this segment, but more is coming" (waiting for tool result).

So chat completion APIs are internally state machines, switching states based on special tokens while streaming (normal stream / tool call parse / role switch). OpenAI's chat completion protocol wraps all this into "look-simple" JSON, but underneath is the full chat template + reverse-parsing + state-machine machinery.

llama.cpp handles this in tools/server/utils.hpp's oaicompat_chat_params_parse; vLLM has its own openai/serving_chat.py. This is the most business-logic-heavy part of the inference stack — buggier than the attention kernel, because it must handle a dozen different chat-template variations.

Another less-obvious reason vLLM beats llama.cpp on speed: CUDA Graph. The setup: one decode forward involves ~hundreds of CUDA kernel launches; each launch has ~3-5 μs fixed overhead on H100 — 100 kernels × 5 μs = ~500 μs of pure launch time. This time is unrelated to model compute — purely CPU → GPU instruction transfer.

CUDA Graph's idea: "record" the static sequence of those 100 kernels once into a graph object; subsequent forwards just replay the entire graph — one graph launch, all kernels execute in order, launch overhead amortized to 0. Gives 10-20% decode latency reduction.

But CUDA Graph has strict requirements: tensor shapes inside the graph must be fully fixed; kernel sequence must be fully determined. This conflicts with vLLM's dynamic batching — batch sizes change constantly. Fix: vLLM captures a graph for each common batch size (1, 2, 4, 8, 16, ..., 256); at runtime picks the nearest captured graph based on actual batch (with padding). Cost: long warmup (capture dozens of graphs at startup), 1-2 GB extra memory.

llama.cpp didn't use CUDA Graph for a long time — its graph builder (ggml) is dynamically constructed, incompatible with CUDA Graph's "static" assumption. Experimental support arrived in 2024 but is less mature than vLLM's. This is one source of "vLLM 100 tokens/s, llama.cpp 60 tokens/s on the same model and hardware" numbers.

vLLM's kernels aren't all hand-written CUDA. It's a three-layer stack:

• cuBLAS / cuDNN (NVIDIA closed-source): general GEMM / GEMV via cuBLAS, standard conv / RNN via cuDNN. NVIDIA engineers' most hardware-tuned code; the inference engine just calls them.
• Triton (OpenAI's open-source JIT compiler): vLLM writes attention / norm / quantize / rope and other "non-standard" kernels in Triton. Triton is a Python-like DSL; it auto-manages SRAM tiles, warp allocation, async copies — 5-10× faster to write than CUDA, achieving 80-95% of cuBLAS performance.
• Hand-written CUDA: the most critical kernels (FlashAttention family, PagedAttention) are hand-written CUDA / CUTLASS, to squeeze every hardware feature (WGMMA / TMA / async pipeline). Specialists like Tri Dao's team work here.

llama.cpp takes a completely different path: implements ggml itself, all kernels hand-written C++/CUDA, no cuBLAS dependency. This gives cross-platform capability (Metal/Vulkan/SYCL each implement the same op set in their own backend), at the cost of having to rewrite every new hardware feature. Triton's no help — Triton only runs NVIDIA GPUs.

This is the technical root of the "llama.cpp cross-platform vs vLLM CUDA-only" fundamental divide: Python engines can treat NVIDIA as first-class via Triton; C++ engines balancing multi-platform must write their own kernels. Each path makes sense but locks the trajectory.

fp8 has two mainstream formats; the bit allocation differs:

• e4m3: 1 sign + 4 exponent + 3 mantissa · range ±448 · more precision, narrower range · used for weights (typically ±10, precision matters)
• e5m2: 1 sign + 5 exponent + 2 mantissa · range ±57344 · wider range, less precision · used for gradients / activations / KV (large dynamic range, tolerates noise)

NVIDIA Hopper (H100) introduced native hardware fp8 support; fp8 GEMM throughput is 2× fp16. Double the TFLOPS translates directly to decode speed. So fp8 KV cache on H100 isn't just memory savings — it's time savings too.

llama.cpp's GGUF quantization is just one approach. In production you'll encounter at least five mainstream schemes, each diverging on the core question "how to pick scales":

Three key observations:

• "Calibration" is why GPTQ / AWQ beat GGUF: they forward a real text sample, record per-layer activation distributions, then reverse-compute "which weight quantization errors don't matter" (because they multiply near-zero activations); conversely give the important weights more bits. Cost: quant time — a 70B model takes 30 minutes under GPTQ, a few minutes under GGUF.
• SmoothQuant solves "activation outliers": LLM activations occasionally have extremely large "outlier" values — these can't be quantized (int8 truncation causes huge error). SmoothQuant's trick: shift the activation's scale onto the weight — divide activation by s, multiply weight by s, mathematically equivalent but activations become balanced and quantizable. Prerequisite for W8A8 (8-bit weight + 8-bit activation).
• HQQ uses Halton-sampled initialization + one gradient step to find optimal scales — no calibration data needed. Good for "we don't know what the user will run" scenarios (e.g. open-sourcing the quant tool for users to self-quantize).

Real choices: llama.cpp users use GGUF; vLLM / TGI deployments use AWQ; TensorRT-LLM uses SmoothQuant. The same Llama-3-70B on HuggingFace Hub may have 6-8 different quantized variants — pick by which engine you're using.

The 5 speculative variants in the main table deserve unpacking. Each solves "cheaply guess the next token" differently:

• Vanilla (Leviathan et al. 2022): use an independent small model (e.g. 1B) as draft for the big model (70B). Simple, but needs two trained models, and the small model's sampling distribution must match the big one closely — often hard; hit rate ~50%.
• Medusa (Cai et al. 2024): no independent draft model — add multiple parallel LM heads to the main model. Each medusa head doesn't predict "next token" but "next-next / next-next-next token". One forward emits k candidates — the big model is its own draft. Pro: no separate model, fast head training. Con: lower hit rate than EAGLE — medusa heads see only hidden state, not intermediate tokens.
• EAGLE (Li et al. 2024): builds on Medusa — draft heads see not just hidden state but a "lightweight transformer layer"'s intermediate. Lightweight layer is small (tens of M params), but seeing the main model's semantic representation lifts hit rate to 75-85%. Current SOTA.
• Lookahead (Fu et al. 2023): no model guesses — mine n-gram frequencies from current generation history; guess "given this context, what k tokens likely come next". Ultra-simple but only works for repetition-heavy text (code / structured output).
• Spec + prefix sharing (2024-2025): in one batch, multiple users share the draft model — run draft once per batch, all users use the same candidate tokens. At high concurrency this amortizes draft to near-zero cost.

vLLM 2024 has built-in EAGLE; SGLang followed. llama.cpp main only has vanilla speculative; EAGLE port is WIP but not merged. In reasoning model scenarios (C26), EAGLE accelerates dramatically — reasoning has high semantic continuity, draft hit rate reaches 85%+.

Ch.2 said prefill and decode are two different workloads. So how can they share one batch? This deserves an explanation, since it's not obvious.

Key: model forward's input abstraction is llama_ubatch{ token, pos, seq_id, ... } (Ch.5) — it doesn't distinguish "prefill vs decode token", only "which sequence does this token belong to, at what position". One batch:

• seq_0 tokens 0,1,2,3,4,5 (prefill 6 tokens, shared seq_id)
• seq_1 token 100 (decode, 1 token)
• seq_2 tokens 0,1,...,199 (prefill 200 tokens)
• seq_3 token 50 (decode)

Batch total: 6 + 1 + 200 + 1 = 208 tokens. One forward processes 208 tokens, Q matrix [208, d_head]. Attention's mask is computed from seq_id + pos — seq_0's token 5 sees only seq_0's tokens 0..5, not seq_2's — that's per-user isolation inside the batch.

This works great on H100 because there are 132 SMs; one forward easily absorbs the mixed compute/bandwidth load of prefill and decode. But the kernel must support this mixed batch — FlashAttention v2/v3 explicitly added this (the varlen API, accepting Q rows of different lengths).

One footgun: in a mixed batch, prefill has large N, decode N=1. If the batch's total token budget (max_num_batched_tokens) is capped, one long prefill crowds out many decodes — P99 latency spikes. vLLM's --max-num-seqs flag and chunked prefill (Ch.21) exist precisely to fight this unfairness.

Chunked prefill mixes prefill and decode logically. 2024 brought a more radical idea — split them physically across different GPUs:

• Sarathi-Serve (MSR 2023): showed prefill and decode have wildly different hardware needs; proposed chunked prefill into decode batch. Adopted by vLLM.
• DistServe (SOSP 2024, UC San Diego): more radical — dedicate one GPU pool to prefill, another to decode; bridge via KV cache transfer. A user's prompt prefills on the "prefill cluster"; the resulting KV moves wholesale to the "decode cluster" to start decoding. Wins on both latency and throughput, at the cost of network KV transfer overhead.
• Splitwise (MSR 2024): same idea, but on different GPU SKUs — prefill on compute-rich H100, decode on bandwidth-rich-but-compute-lean A100. Makes sense in cost-sensitive scenarios.

This line is still in the research → production transition. The biggest real deployments are at Anthropic-tier scale and a few hyperscalers — you can't see their code, but P99 latency improvements give it away. Next-gen inference engines will likely natively support this "disaggregated prefill/decode" pattern.

Llama-3-405B in fp8 = ~405 GB; GQA-8 at 128K context = ~10 GB KV cache per user. One 8×H100 node (640 GB):

• fp8 weights: 405 GB / 8 = ~51 GB/card
• CUDA context + activation buffers: ~10 GB/card
• Remaining for KV cache: 80 - 51 - 10 = ~19 GB/card
• Node-wide KV capacity: 19 × 8 = 152 GB
• Concurrent users (128K context, ~10 GB/user): ~15

This SKU is luxurious: one node at ~$32/h, 15 users = $2.13/user-hour. Hence why 405B's per-token pricing is 5× a 70B's — service density is 10× lower per GPU.

Next-gen directions:

• Llama-3.1-405B-Instruct-MoE (hypothetical) → MoE cuts active params, lifts decode speed, shrinks KV. Meta hasn't shipped this; DeepSeek-V3 / Llama-4 are on this path.
• Lower precision: fp4 (MXFP4) → model ~200 GB, fits in 5 cards → 30 users per node.
• Cross-node PP: 2 nodes / 16 cards, PP=2 × TP=8 → ~25 GB weights per card, KV capacity doubles. Cost: cross-node InfiniBand traffic.

Ch.3 covered tokenize. The reverse (detokenize) looks easy — look each id up in vocab and concat. But you can't just do that: Llama-3's vocab has thousands of tokens with leading spaces (" the", " hello"); naive concat gives "the helloworld" — spaces in the wrong places.

Correct: each token's string already carries its leading space (BPE training did this for you). So detokenize is just concat — don't add spaces. But special tokens like <|begin_of_text|> must be eaten at detokenize, not shown to the user. llama.cpp distinguishes via llama_vocab::detokenize(..., remove_special=true).

In streaming, detokenize runs incrementally per token, but you can't assume each token decodes to a clean utf-8 substring. The Chinese "你" (id 47045) detokenizes to 3 utf-8 bytes E4 BD A0 — clean. But "🤖" (U+1F916) may be split across 2 BPE tokens; each individually is invalid utf-8. Servers must maintain a "pending byte buffer", checking how much of it forms complete characters, emitting only that, leaving the partial bytes for the next concat.

Constrained decoding has a weakness: it forces the model into specific tokens, but the model may be bad at choosing those — forcing a JSON-naive model to emit JSON makes it pick keys somewhat randomly (its probability mass isn't concentrated on sensible keys).

OpenAI's / Anthropic's structured output SOTA is constraint + finetune in tandem: SFT (supervised fine-tuning) heavy on "schema → schema-valid JSON" pairs, so the model learns "see schema description, naturally emit valid JSON". Layer constrained decoding on top — double insurance and semantically natural.

This is why GPT-4o structured outputs are semantically a class above Llama-3 + GBNF JSON — the former is "model wants to write it this way, grammar backstop"; the latter is "model isn't great, grammar forces it". Same tool, different semantic base, very different quality.

The above — "prepend visual tokens to text tokens" — is early fusion (LLaVA / Llama-3.2-Vision). Two other approaches:

• Late fusion (Flamingo 2022): separate vision encoder; inject "visual info" only at specific LLM layer boundaries via extra cross-attention layers that let text tokens "look at" the image. Pros: visual tokens don't occupy LLM sequence length, saving KV cache. Cons: architectural complexity, deployment difficulty.
• Cross-attention (some Llama-3.2-Vision variants): similar to Flamingo, more tightly integrated. Meta's Llama-3.2-90B Vision uses this — inserting a cross-attention block every 4 layers in the main LLM path; vision features flow through these blocks without taking sequence space.
• Native multimodal (GPT-4o, Gemini): no separation between vision encoder and main LLM — same transformer trained on text and visual tokens together from the start. Architecturally identical to LLaVA-style but with totally different training objectives. Better quality but models are 2× heavier.

In real inference engines: llama.cpp mainly supports early fusion (LLaVA-compatible); Llama-3.2-Vision's cross-attention variant needs dedicated support (extra attention nodes in the graph, different KV cache layout). Supporting new multimodal architectures is what inference-engine teams pull overtime for — each new model means kernel and graph changes.

The same "encode anything as tokens" idea works for other modalities:

• Audio (Whisper / GPT-4o-realtime / Qwen2-Audio): 1s of 16kHz audio → mel-spectrogram → audio encoder → ~50 audio tokens. Voice is denser than images — 15s of speech = ~750 tokens. This is why "real-time voice AI" inference is so expensive — every second of input continuously eats 50 prefill tokens.
• Video (Gemini 1.5 / Qwen2-VL): 1s of video ≈ 1 frame × 256 visual tokens + 1s audio × 50 audio tokens = ~306 tokens/s. A 60-second video = ~18000 tokens — instantly torches an 8K window; you need Gemini-class 1M+ context to handle it.
• 3D / point clouds (Llama-Mesh, very early): encode 3D point clouds as tokens — papers work, quality improving.
• Embodied AI (Octo / RT-2): robot sensor readings + camera + joint states all encoded as tokens; the model outputs action tokens. Same "token-in, token-out" structure.

So LLM inference engines are essentially becoming "universal token processors" — what encoder feeds the front-end doesn't matter, as long as it produces tokens. Everything from the previous 24 chapters (KV cache / attention / sampling / batching / quantization) applies. That's the power of the LLM paradigm: unify all heterogeneous data into token sequences and have only one optimization path.

NVIDIA's B200 / GB200 NVL72 systems (2025) mark a new watershed for LLM inference. Two key shifts:

• fp4 hardware (MXFP4 / NVFP4): 4-mantissa-bit floats — higher precision than int4 (preserves dynamic range), GEMM throughput 2× fp8. On B200, a 405B model in fp4 fits ~200 GB; one GB200 node (1.8 TB/s 5th-gen NVLink) serves 70B-class throughput.
• GB200 NVL72: single rack with 72 B200 + 36 Grace CPUs, all NVLink-interconnected — what previously required cross-node InfiniBand for 405B / 671B / 1T+ models can now run TP=72 within a single rack. Cross-node latency bottlenecks vanish by ~70%.

Deeper still: Blackwell's "Transformer Engine 2.0" puts training-time fp8 auto-calibration and inference-time fp4 deployment in the same software stack — a model trained fp8 can deploy fp4 directly, with losses auto-measured and compensated. Training and inference toolchains begin to merge — a first in five years. The next step is probably 2026-2027's "train natively optimized for fp4 deployment" (quantization-aware training as default).

NVIDIA's 80% LLM-inference market share is challenged, but each path differs:

• Google TPU v5p / v6: internal-only, not for sale. Big advantage: 2D mesh interconnect — 256 TPUs directly linked, higher bandwidth than NVLink. Gemini trains and runs entirely on TPU. But the software stack is JAX/XLA, isolated from PyTorch's ecosystem.
• AMD MI300X: 192 GB HBM3 per card, fp16 compute close to H100, 30% cheaper. Weakness: software — ROCm trails CUDA by 5 years; kernel libraries (rocBLAS / rocFFT) lag CUDA's. vLLM added ROCm support in 2024 but production deployments are rare.
• AWS Trainium / Inferentia: AWS's in-house inference chips; Trainium 2 launched 2024. Good price-perf but only on AWS, ecosystem-locked.
• Cerebras / SambaNova / Groq: weird architectures. Groq replaces HBM with massive SRAM — entire model weights in SRAM. Llama-3-70B decode ~750 tokens/s (industry-leading), but can't run > 80B models and no batching (per-query only). Cerebras uses wafer-scale single-chip to fit a 100B model on one wafer. Niche "bet the architecture" plays.

Trend: NVIDIA's software ecosystem + compute lead persists for 2-3 more years; AMD MI series most likely to catch up next (software gradually closed by vLLM / SGLang and other open-source efforts). TPU and Trainium are "walled gardens" — technically competitive but ecosystem-locked.