Human motion prediction plays a vital role in human-robot interaction with various applications such as family service robot. Most of the existing works did not explicitly model velocities of skeletal motion that carried rich motion dynamics, which is critical to predict future poses. In this paper, we propose a novel feedforward network, AGVNet (Attention Guided Velocity Learning Network), to predict future poses, which explicitly models the velocities at both Encoder and Decoder. Specifically, a novel two-stream Encoder is proposed to encode the skeletal motion in both velocity space and position space. Then, a new feedforward Decoder is presented to predict future velocities instead of position poses, which enables the network to predict multiple future velocities recursively like RNN based Decoder. Finally, a novel loss, ATPL (Attention Temporal Prediction Loss), is designed to pay more attention to the early predictions, which can efficiently guide the recursive model to achieve more accurate predictions. Extensive experiments show that our method achieves state-of-the-art performance on two benchmark datasets (i.e. Human3.6M and 3DPW) for human motion prediction, which demonstrates the effectiveness of our proposed method. The code will be available if the paper is accepted.