Recommending points of interest is a difficult problem that requires precise location information to be extracted from a location-based social media platform. Another challenging and critical problem for such a location-aware recommendation system is modelling users' preferences based on their historical behaviors. We propose a location-aware recommender system based on Bidirectional Encoder Representations from Transformers for the purpose of providing users with location-based recommendations. The proposed model incorporates location data and user preferences. When compared to predicting the next item of interest (location) at each position in a sequence, our model can provide the user with more relevant results. Extensive experiments on a benchmark dataset demonstrate that our model consistently outperforms a variety of state-of-the-art sequential models.