Your policy is perfect in simulation and useless on the real robot. How do you close the gap?
hardAnswer
- Assume the policy has learned to exploit simulator inaccuracies, because that is the usual cause rather than a lack of capacity.
- Domain randomization is the main tool: vary masses, friction, latencies, sensor noise and lighting during training so the policy must be robust across a distribution of dynamics rather than optimal for one wrong set.
- Add realistic actuation delay and observation noise, which simulators tend to omit and which change control problems qualitatively.
- Identify parameters from real measurements where you can, narrowing the randomization around reality.
- Then fine-tune on a small amount of real data, and design the observation space to use signals whose simulated version you trust, since a policy that depends on an unrealistic sensor cannot transfer.
Check yourself — multiple choice
- Train longer in simulation
- Randomize dynamics, latency and sensor noise so the policy must be robust, calibrate the randomization to measurements, fine-tune on real data, and avoid observations the simulator gets wrong
- Use a larger network
- Sim-to-real transfer is impossible
Robustness across a distribution of dynamics, plus calibration and light real-world fine-tuning, is the standard recipe.
#applications#engineering
Practise Reinforcement Learning
214 interview questions in this topic.