What is speculative decoding?

hard

Answer

$Use a small 'draft' model to propose K tokens ahead, then have the large 'target' model verify all K in a single parallel forward pass.$
$Whichever prefix the target agrees with is accepted; the first disagreement position is corrected and the rest re-drafted.$
$Output is identical to target-only decoding.$
$Speedup: 2-3x when draft agreement is high.$
$Foundation of vLLM's speculative decoding, Medusa, EAGLE.$
$Draft model choice: same-family small model (Llama-3-8B drafting for Llama-3-70B) or a trained n-gram head.$

Check yourself — multiple choice

Sacrifices output identity
Small draft proposes K tokens → target verifies all in one pass → identical output, 2-3× faster
Doubles latency
Only for training

Speculative decoding: draft model proposes → target verifies in parallel → 2-3× speedup, exact same output.

#inference#generation

214 interview questions in this topic.