How do you explore safely when the environment is real paying users?
hardAnswer
- Bound the damage rather than trusting the algorithm.
- Restrict exploration to a small traffic slice, so a bad action affects a known fraction of users, and exclude segments where the cost of a mistake is high.
- Constrain the action set to options you have vetted, which turns unbounded exploration into a choice among acceptable alternatives.
- Use off-policy evaluation on logged data to estimate a candidate policy's value before it ever serves traffic, remembering that this requires logged action probabilities, so instrument them from day one.
- Monitor guardrail metrics with automatic rollback, and prefer optimistic or posterior-sampling exploration over epsilon-greedy, since random actions spend your budget on options already known to be bad.
Check yourself — multiple choice
- Use epsilon-greedy on all traffic
- Limit exploration to a small vetted traffic slice with a restricted action set, log propensities for off-policy evaluation, add guardrails with rollback, and prefer posterior sampling to random actions
- Never explore in production
- Exploration is only a simulation concern
Safe exploration comes from bounded traffic, vetted actions, logged propensities and guardrails.
#exploration#safety#applications
Practise Reinforcement Learning
214 interview questions in this topic.