Visual localization techniques often comprise a hierarchical localization pipeline, with a visual place recognition module used as a coarse localizer to initialize a pose refinement stage. While improving the pose refinement step has been the focus of much recent research, most work on the coarse localization stage has focused on improvements like increased invariance to appearance change, without improving what can be loose error tolerances. In this letter, we propose two methods which adapt image retrieval techniques used for visual place recognition to the Bayesian state estimation formulation for localization. We demonstrate significant improvements to the localization accuracy of the coarse localization stage using our methods, whilst retaining state-of-the-art performance under severe appearance change. Using extensive experimentation on the Oxford RobotCar dataset, results show that our approach outperforms comparable state-of-the-art methods in terms of precision-recall performance for localizing image sequences. In addition, our proposed methods provides the flexibility to contextually scale localization latency in order to achieve these improvements. The improved initial localization estimate opens up the possibility of both improved overall localization performance and modified pose refinement techniques that leverage this improved spatial prior.