How does JSON mode actually work?
hardAnswer
- The provider's inference engine tracks a JSON parser state alongside decoding.
- At each step, it computes the set of allowed next tokens (given the parser state and, optionally, a JSON schema), and masks the logits to zero out disallowed tokens before sampling.
- Output is guaranteed to be syntactically valid JSON (and, with schema, matches the schema).
- Costs: ~5-15% throughput vs unconstrained decoding — often worth it for reliability.
- Libraries: Outlines, XGrammar, GBNF (llama.cpp).
Check yourself — multiple choice
- Prompt engineering only
- Parser-state-aware logit masking at each step → guaranteed valid JSON (with schema if provided)
- Post-processing regex
- Not available in modern APIs
JSON mode: parser-guided logit masking → schema-valid output guaranteed.
#generation#production
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- How do you force structured output (JSON) from an LLM?
- What are stop sequences and why do they matter in production?
- What is logit bias and when do you use it?
- How should max_tokens be set in production?
- Why is streaming output important in production LLM apps?
- Structured Outputs (OpenAI) / Constrained decoding — how do they differ from JSON mode?