Alert button

Dynamic Horizon Value Estimation for Model-based Reinforcement Learning

Sep 21, 2020
Junjie Wang, Qichao Zhang, Dongbin Zhao, Mengchen Zhao, Jianye Hao

Figure 1 for Dynamic Horizon Value Estimation for Model-based Reinforcement Learning
Figure 2 for Dynamic Horizon Value Estimation for Model-based Reinforcement Learning
Figure 3 for Dynamic Horizon Value Estimation for Model-based Reinforcement Learning
Figure 4 for Dynamic Horizon Value Estimation for Model-based Reinforcement Learning

Share this with someone who'll enjoy it:

Existing model-based value expansion methods typically leverage a world model for value estimation with a fixed rollout horizon to assist policy learning. However, the fixed rollout with an inaccurate model has a potential to harm the learning process. In this paper, we investigate the idea of using the model knowledge for value expansion adaptively. We propose a novel method called Dynamic-horizon Model-based Value Expansion (DMVE) to adjust the world model usage with different rollout horizons. Inspired by reconstruction-based techniques that can be applied for visual data novelty detection, we utilize a world model with a reconstruction module for image feature extraction, in order to acquire more precise value estimation. The raw and the reconstructed images are both used to determine the appropriate horizon for adaptive value expansion. On several benchmark visual control tasks, experimental results show that DMVE outperforms all baselines in sample efficiency and final performance, indicating that DMVE can achieve more effective and accurate value estimation than state-of-the-art model-based methods.

* 9 pages, 6 figures  
View paper onarxiv icon

Share this with someone who'll enjoy it: