Get our free extension to see links to code for papers anywhere online!

Chrome logo Add to Chrome

Firefox logo Add to Firefox

Plannable Approximations to MDP Homomorphisms: Equivariance under Actions

Feb 27, 2020
Elise van der Pol, Thomas Kipf, Frans A. Oliehoek, Max Welling

This work exploits action equivariance for representation learning in reinforcement learning. Equivariance under actions states that transitions in the input space are mirrored by equivalent transitions in latent space, while the map and transition functions should also commute. We introduce a contrastive loss function that enforces action equivariance on the learned representations. We prove that when our loss is zero, we have a homomorphism of a deterministic Markov Decision Process (MDP). Learning equivariant maps leads to structured latent spaces, allowing us to build a model on which we plan through value iteration. We show experimentally that for deterministic MDPs, the optimal policy in the abstract MDP can be successfully lifted to the original MDP. Moreover, the approach easily adapts to changes in the goal states. Empirically, we show that in such MDPs, we obtain better representations in fewer epochs compared to representation learning approaches using reconstructions, while generalizing better to new goals than model-free approaches.

* To appear in Proceedings of the International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS 2020) 

Share this with someone who'll enjoy it:

   Access Paper Source

Share this with someone who'll enjoy it: