What is repetition penalty and its trade-off?

medium

Answer

$Penalize logits for tokens that have already appeared in the context: logit(t) /= penalty (or -= alpha) if t \in recent history.$
$Reduces the model's tendency to loop ('the the the').$
$Typical values: 1.1-1.3.$
$Trade-off: aggressive penalties hurt natural repetition (poetry, code with repeated function calls, technical terms).$
$\mathrm{presence}_{\mathrm{penalty}}$
$Modern LLMs need less penalty because RLHF trains against loops.$

Check yourself — multiple choice

No trade-off
Penalize recently-seen tokens' logits to reduce loops; aggressive penalties hurt natural repetition (code, poetry, terms)
Only for RNNs
Removes all repetition

Repetition penalty: reduce loops at the cost of natural repetition; tune carefully.

#generation#decoding

Practise LLMs & GenAI

214 interview questions in this topic.

Read the lesson Flashcards Take the quiz

Related questions