Coronavirus disease 2019 has affected the world seriously, because people cannot work as usual in case of infection. One of the effective protection methods for human beings is to wear masks in public areas. Furthermore, many public service providers require customers to use the service only if they wear masks correctly. However, there are only a few research studies about face mask detection. To contribute to public healthcare for human beings, we propose RetinaMask, which is a high-accuracy and efficient face mask detector. The proposed RetinaMask is a one-stage detector, which consists of a feature pyramid network to fuse high-level semantic information with multiple feature maps, and a novel context attention module to focus on detecting face masks. In addition, we also propose a novel cross-class object removal algorithm to reject predictions with low confidences and the high intersection of union. Experimental results show that RetinaMask achieves state-of-the-art results on a public face mask dataset with 2.3% and 1.5% higher than the baseline result in the face and mask detection precision, respectively, and $11.0\%$ and $5.9\%$ higher than baseline for recall. Besides, we also explore the possibility of implementing RetinaMask with a light-weighted neural network MobileNet for embedded or mobile devices.