Pedestrian trajectory prediction in dynamic scenes remains a challenging and critical problem in numerous applications, such as self-driving cars and socially aware robots. Challenges concentrate on capturing pedestrians' social interactions and handling their future uncertainties. Pedestrians' head orientations can be used as an oracle that indicates relevant pedestrians[1], thus is beneficial to model social interactions. Moreover, latent variable distributions of pedestrians'future trajectories can be termed as another oracle. However, few works fully utilize these oracle information for an improved prediction performance. In this work, we propose GTPPO (Graph-based Trajectory Predictor with Pseudo Oracle), which is a generative model-based trajectory predictor. Pedestrians'social interactions are captured by the proposed GA2T (Graph Attention social Attention neTwork) module. Social attention is calculated on the basis of pedestrians' moving directions, which are termed as a pseudo oracle. Moreover, we propose a latent variable predictor to learn the latent variable distribution from observed trajectories. Such latent variable distribution reflects pedestrians'future trajectories, and therefore can be taken as another pseudo oracle. We compare the performance of GTPPO with several recently proposed methods on benchmarking datasets. Quantitative evaluations demonstrate that GTPPO outperforms state-of-the-art methods with lower average and final displacement errors. Qualitative evaluations show that GTPPO successfully recognizes the sudden motion changes since the estimated latent variable reflects the future trajectories.