How do you choose the discount factor, and what goes wrong at the extremes?

medium

Answer

$It sets the effective horizon, roughly one over one minus gamma steps, so pick it from how far ahead consequences actually matter in your problem rather than by convention.$
$Too low and the agent is myopic: it ignores delayed reward entirely, which in a game looks like refusing to invest a move for a later gain.$
$Too high and two things degrade, because the variance of returns grows with horizon length and the value function has to represent a much larger range, so bootstrapped estimates become unstable and slow to converge.$
$In episodic tasks with a natural end, a value near one is defensible; in continuing tasks it is a variance knob you tune.$
$A common practical trick is to train with a lower value and anneal it upward as the value function stabilizes.$

Check yourself — multiple choice

Always use 0.99
It sets the effective horizon: too low is myopic, too high inflates return variance and value range, so match it to how far consequences matter and consider annealing it upward
It only affects computation time
It must equal 1

The discount trades myopia against return variance, and it should follow the problem's real horizon.

#theory#fundamentals

214 interview questions in this topic.