Many actor-critic deep reinforcement learning (DRL) algorithms have achieved cutting-edge performance in tackling various challenging reinforcement learning (RL) problems, including complex control tasks with high-dimensional continuous state and action spaces. Despite of widely reported success, existing DRL algorithms often suffer from the ineffective exploration issue, resulting in limited learning stability and performance. To address this limitation, several ensemble DRL algorithms have been proposed recently to boost exploration and stabilize the learning process. However, many existing ensemble algorithms are designed to train each base learner individually without controlling explicitly the collaboration among the trained base learners. In this paper, we propose a new technique to train an ensemble of base learners based on the multi-step integration methods. The new multi-step training technique enables us to develop a new hierarchical training algorithm for ensemble DRL that promotes inter-learner collaboration through explicit inter-learner parameter sharing. The design of our new algorithm is verified theoretically. The algorithm is also shown empirically to outperform several cutting-edge DRL algorithms on multiple benchmark RL problems.