Alert button

Your Policy Regularizer is Secretly an Adversary

Apr 01, 2022
Rob Brekelmans, Tim Genewein, Jordi Grau-Moya, Grégoire Delétang, Markus Kunesch, Shane Legg, Pedro Ortega

Figure 1 for Your Policy Regularizer is Secretly an Adversary
Figure 2 for Your Policy Regularizer is Secretly an Adversary
Figure 3 for Your Policy Regularizer is Secretly an Adversary
Figure 4 for Your Policy Regularizer is Secretly an Adversary

Share this with someone who'll enjoy it:

Policy regularization methods such as maximum entropy regularization are widely used in reinforcement learning to improve the robustness of a learned policy. In this paper, we show how this robustness arises from hedging against worst-case perturbations of the reward function, which are chosen from a limited set by an imagined adversary. Using convex duality, we characterize this robust set of adversarial reward perturbations under KL and alpha-divergence regularization, which includes Shannon and Tsallis entropy regularization as special cases. Importantly, generalization guarantees can be given within this robust set. We provide detailed discussion of the worst-case reward perturbations, and present intuitive empirical examples to illustrate this robustness and its relationship with generalization. Finally, we discuss how our analysis complements and extends previous results on adversarial reward robustness and path consistency optimality conditions.

* 10 pages main text; added worked example  
View paper onarxiv icon

Share this with someone who'll enjoy it: