Speculative decoding — how does it accelerate LLMs?

hard

Answer

$Small 'draft' model generates K tokens fast.$
$Large 'target' model verifies all K in single forward pass (parallel).$
$Accepted tokens (matching what large would have generated) are kept; on first mismatch, use large model's token + restart.$
$Amortizes large-model forward pass across accepted tokens. 2-3x speedup with correct output distribution (matches large model exactly).$
$Variants: Medusa (multi-head), Eagle, TriForce.$
$Modern standard for LLM serving speedup.$

Check yourself — multiple choice

Random
Small draft generates K tokens; large target verifies parallel; keep accepted, restart on mismatch; amortizes large forward across accepted → 2-3x speedup + exact output distribution; Medusa/Eagle/TriForce variants
Just guess
Not real

Speculative: draft + parallel verify; 2-3x speedup, exact output.

#serving#llmops

215 interview questions in this topic.