The task of learning to navigate in the complex environment is often tackled in the deep reinforcement learning framework utilizing a reactive policy or general-purposed recurrent policy. Unfortunately, these two kinds of policy are insufficient to deal with long-term memory issue in visual navigation and causing a long learning period. To address this issue, this paper proposes a graph attention memory (GAM) based navigation system which includes three modules: a memory construction module, a graph attention module, and a control module. The memory construction module builds the topological graph based on supervised learning by taking the exploration prior. Then guided attention features are extracted from the graph attention module. Finally, the deep reinforcement learning based control module makes the decision by taking the visual observation and guided attention features. The proposed method is validated in a complex 3D environment. The results show that the GAM-based navigation system outperforms all baselines both in the learning speed and the success rate. We also provide a detailed analysis of the topological occupancy of the graph based on the manual and random exploration strategies.