How should $\operatorname{max}_{\mathrm{tokens}}$ be set in production?

easy

Answer

$Set it to the maximum reasonable output length + a small buffer.$
$Too low: outputs get truncated mid-sentence.$
$Too high: encourages the model to ramble (RLHF sometimes over-generates), and increases latency + cost.$
$\operatorname{max}_{\mathrm{tokens}}$
$\operatorname{max}_{\mathrm{tokens}}$
$\operatorname{max}_{\mathrm{tokens}}\; = \;\mathrm{P95}\; \cdot \;1.5$

Check yourself — multiple choice

Set to 4096 always
Measure P95 output length × 1.5 → too low truncates, too high rambles + adds cost/latency
No max needed
Same as temperature

$\operatorname{max}_{\mathrm{tokens}}$ : sized by observed P95 output + buffer; controls cost / latency / truncation.

#generation#production#cost

214 interview questions in this topic.