EasyDeepLearn
LLMs & GenAI · section 8 of 18

Decoding & sampling

16 interview questions on decoding & sampling, each answered in full. Free to read, no account needed.

What do temperature and top-p do at generation time?

easy
  • Temperature scales logits before softmax: low T (e.g., 0-0.3) makes outputs sharp and deterministic — use for factual answers and code.
  • High T (0.7-1.2) makes outputs more diverse — use for creative writing.
  • Top-p (nucleus) restricts sampling to the smallest set of tokens whose cumulative probability exceeds p.
  • Combined: 'topp=0.9\mathrm{top}_{p} = 0.9, temperature=0.7' is a common balanced default.
#generation#decoding#samplingPermalink & quiz →

How does JSON mode actually work?

hard
  • The provider's inference engine tracks a JSON parser state alongside decoding.
  • At each step, it computes the set of allowed next tokens (given the parser state and, optionally, a JSON schema), and masks the logits to zero out disallowed tokens before sampling.
  • Output is guaranteed to be syntactically valid JSON (and, with schema, matches the schema).
  • Costs: ~5-15% throughput vs unconstrained decoding — often worth it for reliability.
  • Libraries: Outlines, XGrammar, GBNF (llama.cpp).
#generation#productionPermalink & quiz →

Greedy decoding vs sampling — when do you pick each?

easy
  • Greedy (T=0): argmax at each step.
  • Deterministic, reproducible, often suboptimal on open-ended tasks because it commits to the highest-probability token even when it leads to bad continuations.
  • Sampling (T>0 + top-p/top-k): stochastic, produces more varied and often more natural outputs.
  • Best practice: greedy or very low T for factual / code / structured; T=0.7-1.0 with topp=0.9\mathrm{top}_{p} = 0.9 for creative / conversational; T=0.2-0.5 with topp=0.95\mathrm{top}_{p} = 0.95 for balanced.
#generation#decoding#samplingPermalink & quiz →

How does top-k differ from top-p sampling?

easy
  • Top-k: sample from the k highest-probability tokens each step, ignoring the rest.
  • Fixed cardinality regardless of the distribution's shape.
  • Top-p (nucleus): sample from the smallest set of tokens whose cumulative probability  ≥  p\mathrm{probability}\; \ge \;p.
  • Adapts to the distribution — takes fewer tokens when the model is confident, more when it's uncertain.
  • Top-p is usually preferred as it maintains diversity where useful and precision where confident.
  • Combining top-k=50 + top-p=0.95 is a common safety net.
#generation#decoding#samplingPermalink & quiz →

What is min-p sampling?

hard
  • min-p (Nguyen et al., 2024): filter tokens with probability  <  min⁡p  ⋅  P(toptoken)\mathrm{probability}\; < \;\operatorname{min}_{p}\; \cdot \;P(\mathrm{top}_{\mathrm{token}}).
  • E.g., min⁡p=0.05\operatorname{min}_{p} = 0.05 keeps tokens with at least 5% of the top token's probability.
  • Adaptive to confidence: when the model is very confident, only the top few tokens survive; when it's uncertain, many pass through.
  • Empirically produces higher-quality creative writing than top-p at similar diversity.
  • Adopted in llama.cpp, vLLM, HuggingFace.
#generation#decoding#samplingPermalink & quiz →

What is repetition penalty and its trade-off?

medium
  • Penalize logits for tokens that have already appeared in the context: logit(t) /= penalty (or -= alpha) if t ∈ recent history.
  • Reduces the model's tendency to loop ('the the the').
  • Typical values: 1.1-1.3.
  • Trade-off: aggressive penalties hurt natural repetition (poetry, code with repeated function calls, technical terms).
  • Alternatives: presencepenalty\mathrm{presence}_{\mathrm{penalty}} (per-appearance), frequencypenalty\mathrm{frequency}_{\mathrm{penalty}} (proportional to count) — used in OpenAI API.
  • Modern LLMs need less penalty because RLHF trains against loops.
#generation#decodingPermalink & quiz →

What are stop sequences and why do they matter in production?

easy
  • Strings that, when generated, terminate the response early.
  • Uses: (1) chat markers like '<|user|>' or '\n\n' to prevent the model from continuing beyond its turn; (2) end-of-JSON marker for structured output; (3) tool-call markers.
  • Without stops, models often hallucinate a new user turn or keep going past the answer.
  • Providers set defaults (assistant turn markers) but exposing them in the API is essential for custom formats.
#generation#productionPermalink & quiz →

What is logit bias and when do you use it?

medium
  • Additive bias added to specific token logits before sampling: logits[tokenid]\mathrm{logits}[\mathrm{token}_{\mathrm{id}}] += bias.
  • Uses: (1) ban tokens (bias=-100 → probability ~0) — e.g., forbid a competitor's name; (2) encourage a token slightly (bias=+2); (3) force a token (very high positive bias).
  • Available in OpenAI API.
  • Not a substitute for prompt engineering or fine-tuning, but useful for last-mile constraints: banning specific tokens, forcing JSON to start with '{', etc.
#generation#productionPermalink & quiz →

How should max⁡tokens\operatorname{max}_{\mathrm{tokens}} be set in production?

easy
  • Set it to the maximum reasonable output length + a small buffer.
  • Too low: outputs get truncated mid-sentence.
  • Too high: encourages the model to ramble (RLHF sometimes over-generates), and increases latency + cost.
  • Also matters for cost estimation and SLA — a max⁡tokens\operatorname{max}_{\mathrm{tokens}} cap prevents pathological runaway generations.
  • In streaming APIs, max⁡tokens\operatorname{max}_{\mathrm{tokens}} caps the total tokens streamed, so users see truncation.
  • Best practice: measure typical output lengths from your prompts and set max⁡tokens  =  P95  ⋅  1.5\operatorname{max}_{\mathrm{tokens}}\; = \;\mathrm{P95}\; \cdot \;1.5.
#generation#production#costPermalink & quiz →

Why is streaming output important in production LLM apps?

easy
  • LLM generation is autoregressive at ~30-200 tokens/sec — even a 500-token response takes several seconds.
  • Streaming yields tokens as they're generated, so users start reading immediately (TTFT = 'time to first token').
  • Perceived latency drops 5-10x.
  • Also enables early cancellation (stop mid-generation if the user closes the tab), progress indicators, and interactive UIs.
  • Implementation: SSE (Server-Sent Events) or WebSocket; all major LLM APIs support streaming.
#production#generation#latencyPermalink & quiz →

What is speculative decoding?

hard
  • Use a small 'draft' model to propose K tokens ahead, then have the large 'target' model verify all K in a single parallel forward pass.
  • Whichever prefix the target agrees with is accepted; the first disagreement position is corrected and the rest re-drafted.
  • Output is identical to target-only decoding.
  • Speedup: 2-3x when draft agreement is high.
  • Foundation of vLLM's speculative decoding, Medusa, EAGLE.
  • Draft model choice: same-family small model (Llama-3-8B drafting for Llama-3-70B) or a trained n-gram head.
#inference#generationPermalink & quiz →

Why don't chat LLMs use beam search?

medium
  • Beam search maximizes joint probability but produces 'safe, boring' outputs — highest-probability sequences are generic and repetitive.
  • Human raters strongly prefer sampled outputs (with T + top-p) in open-ended generation.
  • Beam search still shines in constrained-output tasks (structured NER, MT) where the objective is exact-match precision.
  • RLHF and DPO further shift the optimal decoding distribution away from mode-seeking beam search.
#generation#decodingPermalink & quiz →

Structured Outputs (OpenAI) / Constrained decoding — how do they differ from JSON mode?

hard
  • JSON mode: guarantees valid JSON syntax.
  • Structured Outputs / Constrained decoding: guarantees the JSON matches a specific schema (types, required fields, enums, nested structures).
  • Implementation: build a grammar / automaton from the JSON schema and mask logits to only allow tokens that keep the output on a valid grammar path.
  • OpenAI's Structured Outputs, XGrammar, Outlines.
  • Standard for tool calling and complex extraction where schema violations would break downstream code.
#generation#productionPermalink & quiz →

How do Medusa and EAGLE differ from vanilla speculative decoding?

hard
  • Vanilla speculative decoding uses a separate small draft model — must train / maintain two models, and inference bounces between them.
  • Medusa (Cai 2024) trains multiple parallel 'draft heads' attached to the target model that predict the next K tokens directly — one model, one forward pass drafts K candidates.
  • EAGLE (Li 2024) trains a small LSTM-style head to predict future embeddings from current hidden states → higher acceptance rate than Medusa.
  • Both give 2-3× decode speedup without a separate draft model.
#inference#generationPermalink & quiz →

How do you make an LLM reliably return valid JSON?

medium
  • Constrain the decoder rather than asking politely.
  • Grammar-constrained or schema-constrained decoding masks tokens that cannot continue a valid document, so malformed output becomes impossible rather than unlikely.
  • Most providers expose this as a structured-output or JSON-schema mode.
  • Supplement it with a strict parse and a bounded retry that feeds the validation error back.
  • Keep schemas shallow, since deeply nested unions raise the failure rate and confuse the model.
  • Avoid asking for JSON and prose in the same response, and remember that constrained decoding guarantees the shape, not the correctness of the values.
#generation#productionPermalink & quiz →

Does temperature 0 make an LLM deterministic?

hard
  • Not in practice.
  • Temperature 0 makes the sampling step greedy, but it does not remove every source of variation.
  • Floating-point non-determinism in batched GPU kernels means the logits themselves can differ slightly between runs, and ties can flip.
  • Requests are batched together differently depending on concurrent traffic, which changes reduction order.
  • Mixture-of-experts routing can vary with batch composition.
  • Providers also update model versions behind a stable alias.
  • Treat low temperature as reduced variance, not a guarantee, and if you need reproducibility, pin the model version and store the outputs you depend on.
#decoding#sampling#productionPermalink & quiz →

Practise LLMs & GenAI

All 18 sectionsFlashcardsQuiz