With the increasing demand for millions of COVID-19 screenings, Computed Tomography (CT) based test has emerged as a promising alternative to the gold standard RT-PCR test. However, it is primarily provided in hospital setting due to the need for expensive equipment and experienced radiologists. An accurate, rapid yet inexpensive test that is suitable for COVID-19 population screenings at mobile, urgent and primary care clinics is urgently needed. We present COVID-MobileXpert: a lightweight deep neural network (DNN) based mobile app that can use noisy snapshots of chest X-ray (CXR) for point-of-care COVID-19 screening. We design and implement a novel three-player knowledge transfer and distillation (KTD) framework including a pre-trained attending physician (AP) network that extracts CXR imaging features from large scale of lung disease CXR images, a fine-tuned resident fellow (RF) network that learns the essential CXR imaging features to discriminate COVID-19 from pneumonia and/or normal cases using a small amount of COVID-19 cases, and a trained lightweight medical student (MS) network that performs on-device COVID-19 screening. To accommodate the need for screening using noisy snapshots of CXR images, we employ novel loss functions and training schemes for the MS network to learn the robust imaging features for accurate on-device COVID-19 screening. We demonstrate the strong potential of COVID-MobileXpert for rapid deployment via extensive experiments with diverse MS network architecture, CXR imaging quality, and tuning parameter settings. The source code of cloud and mobile based models are available from https://github.com/xinli0928/COVID-Xray.