Generalizable person re-identification has recently got increasing attention due to its research values as well as practical values. However, the efficiency of learning from large-scale data has not yet been much studied. In this paper, we argue that the most popular random sampling method, the well-known PK sampler, is not informative and efficient for deep metric learning. Though online hard example mining improves the learning efficiency to some extent, the mining in mini batches after random sampling is still limited. Therefore, this inspires us that the hard example mining should be shifted backward to the data sampling stage. To address this, in this paper, we propose an efficient mini batch sampling method called Graph Sampling (GS) for large-scale metric learning. The basic idea is to build a nearest neighbor relationship graph for all classes at the beginning of each epoch. Then, each mini batch is composed of a randomly selected class and its nearest neighboring classes so as to provide informative and challenging examples for learning. Together with an adapted competitive baseline, we improve the previous state of the arts in generalizable person re-identification significantly, by up to 22.3% in Rank-1 and 15% in mAP. Besides, the proposed method also outperforms the competitive baseline by up to 4%, with the training time significantly reduced by up to x6.6, from 12.2 hours to 1.8 hours in training a large-scale dataset RandPerson with 8,000 IDs. Code is available at \url{https://github.com/ShengcaiLiao/QAConv}.