How do you make an LLM reliably return valid JSON?
mediumAnswer
- Constrain the decoder rather than asking politely.
- Grammar-constrained or schema-constrained decoding masks tokens that cannot continue a valid document, so malformed output becomes impossible rather than unlikely.
- Most providers expose this as a structured-output or JSON-schema mode.
- Supplement it with a strict parse and a bounded retry that feeds the validation error back.
- Keep schemas shallow, since deeply nested unions raise the failure rate and confuse the model.
- Avoid asking for JSON and prose in the same response, and remember that constrained decoding guarantees the shape, not the correctness of the values.
Check yourself — multiple choice
- Ask nicely in the prompt
- Use schema-constrained decoding so invalid tokens are masked, plus strict parsing and bounded retries — shape is guaranteed, values are not
- Post-process with regex only
- Raise the temperature
Constrained decoding enforces the grammar; validation still has to check the values.
#generation#production
Practise LLMs & GenAI
214 interview questions in this topic.