Self-supervised goal proposal and reaching is a key component of efficient policy learning algorithms. Such a self-supervised approach without access to any oracle goal sampling distribution requires deep exploration and commitment so that long horizon plans can be efficiently discovered. In this paper, we propose an exploration framework, which learns a dynamics-aware manifold of reachable states. Given a new goal, our proposed method visits a state at the current frontier of reachable states (commitment/reaching) and then explores to reach the goal (exploration). This allocates exploration budget near the frontier of the reachable region instead of its interior. We target the challenging problem of policy learning from initial and goal states specified as images, and do not assume any access to the underlying ground-truth states of the robot and the environment. To keep track of reachable latent states, we propose a distance conditioned reachability network that is trained to infer whether one state is reachable from another within the specified latent space distance. So, given an initial state, we obtain a frontier of reachable states from that state. By incorporating a curriculum for sampling easier goals (closer to the start state) before more difficult goals, we demonstrate that the proposed self-supervised exploration algorithm, can achieve 20% superior performance on average compared to existing baselines on a set of challenging robotic environments, including on a real robot manipulation task.