Animals are able to discover the topological map (graph) of surrounding environment, which will be used for navigation. Inspired by this biological phenomenon, researchers have recently proposed to generate topological map (graph) representation for Markov decision process (MDP) and use such graphs for planning in reinforcement learning (RL). However, existing graph generation methods suffer from many drawbacks. One drawback is that existing methods do not learn an abstraction for graphs, which results in high memory cost. Another drawback is that these existing methods can only work in some specific settings, which limits their application. In this paper, we propose a new method, called TOpological Map Abstraction (TOMA), for graph generation. TOMA can generate an abstract graph representation for MDP, which costs much less memory than existing methods. Furthermore, the generated graphs of TOMA can be used as a basic multi-purpose tool for different RL applications. As an application example, we propose planning to explore, in which TOMA is used to accelerate exploration by guiding the agent towards unexplored states. A novel experience replay module called vertex memory is also proposed to improve exploration performance. Experimental results show that TOMA can robustly generate abstract graph representation on several 2D world environments with different types of observation. Under the guidance of such graph representation, the agent can escape local minima during exploration.