Alert button

Stabilizing Off-Policy Deep Reinforcement Learning from Pixels

Jul 03, 2022
Edoardo Cetin, Philip J. Ball, Steve Roberts, Oya Celiktutan

Figure 1 for Stabilizing Off-Policy Deep Reinforcement Learning from Pixels
Figure 2 for Stabilizing Off-Policy Deep Reinforcement Learning from Pixels
Figure 3 for Stabilizing Off-Policy Deep Reinforcement Learning from Pixels
Figure 4 for Stabilizing Off-Policy Deep Reinforcement Learning from Pixels

Share this with someone who'll enjoy it:

Off-policy reinforcement learning (RL) from pixel observations is notoriously unstable. As a result, many successful algorithms must combine different domain-specific practices and auxiliary losses to learn meaningful behaviors in complex environments. In this work, we provide novel analysis demonstrating that these instabilities arise from performing temporal-difference learning with a convolutional encoder and low-magnitude rewards. We show that this new visual deadly triad causes unstable training and premature convergence to degenerate solutions, a phenomenon we name catastrophic self-overfitting. Based on our analysis, we propose A-LIX, a method providing adaptive regularization to the encoder's gradients that explicitly prevents the occurrence of catastrophic self-overfitting using a dual objective. By applying A-LIX, we significantly outperform the prior state-of-the-art on the DeepMind Control and Atari 100k benchmarks without any data augmentation or auxiliary losses.

* Short presentation at ICML 2022  
View paper onarxiv icon

Share this with someone who'll enjoy it: