Medical-VLBERT: Medical Visual Language BERT for COVID-19 CT Report Generation With Alternate Learning

Medical imaging technologies, including computed tomography (CT) or chest X-Ray (CXR), are largely employed to facilitate the diagnosis of the COVID-19. Since manual report writing is usually too time-consuming, a more intelligent auxiliary medical system that could generate medical reports automatically and immediately is urgently needed. In this article, we propose to use the medical visual language BERT (Medical-VLBERT) model to identify the abnormality on the COVID-19 scans and generate the medical report automatically based on the detected lesion regions. To produce more accurate medical reports and minimize the visual-and-linguistic differences, this model adopts an alternate learning strategy with two procedures that are knowledge pretraining and transferring. To be more precise, the knowledge pretraining procedure is to memorize the knowledge from medical texts, while the transferring procedure is to utilize the acquired knowledge for professional medical sentences generations through observations of medical images. In practice, for automatic medical report generation on the COVID-19 cases, we constructed a dataset of 368 medical findings in Chinese and 1104 chest CT scans from The First Affiliated Hospital of Jinan University, Guangzhou, China, and The Fifth Affiliated Hospital of Sun Yat-sen University, Zhuhai, China. Besides, to alleviate the insufficiency of the COVID-19 training samples, our model was first trained on the large-scale Chinese CX-CHR dataset and then transferred to the COVID-19 CT dataset for further fine-tuning. The experimental results showed that Medical-VLBERT achieved state-of-the-art performances on terminology prediction and report generation with the Chinese COVID-19 CT dataset and the CX-CHR dataset. The Chinese COVID-19 CT dataset is available at https://covid19ct.github.io/.

Interpretable Machine Learning for COVID-19: An Empirical Study on Severity Prediction Task

Black-box nature hinders the deployment of many high-accuracy models in medical diagnosis. It is risky to put one's life in the hands of models that medical researchers do not trust. However, to understand the mechanism of a new virus, such as COVID-19, machine learning models may catch important symptoms that medical practitioners do not notice due to the surge of infected patients during a pandemic. In this work, the interpretation of machine learning models reveals that a high C-reactive protein (CRP) corresponds to severe infection, and severe patients usually go through a cardiac injury, which is consistent with well-established medical knowledge. Additionally, through the interpretation of machine learning models, we find phlegm and diarrhea are two important symptoms, without which indicate a high risk of turning severe. These two symptoms are not recognized at the early stage of the outbreak, whereas our findings are corroborated by later autopsies of COVID-19 patients. We find patients with a high N-terminal pro B-type natriuretic peptide (NTproBNP) have a significantly increased risk of death which does not receive much attention initially but proves true by the following-up study. Thus, we suggest interpreting machine learning models can offer help to diagnosis at the early stage of an outbreak.

Is AI Model Interpretable to Combat with COVID? An Empirical Study on Severity Prediction Task

Black-box nature hinders the deployment of many high-accuracy models in medical diagnosis. Putting one's life in the hands of models that medical researchers do not trust it's irresponsible. However, to understand the mechanism of a new virus, such as COVID-19, machine learning models may catch important symptoms that medical practitioners do not notice due to the surge of infected patients caused by a pandemic. In this work, the interpretation of machine learning models reveals a high CRP corresponds to severe infection, and severe patients usually go through a cardiac injury, which is consistent with medical knowledge. Additionally, through the interpretation of machine learning models, we find phlegm and diarrhea are two important symptoms, without which indicate a high risk of turning severe. These two symptoms are not recognized at the early stage of the outbreak, but later our findings are corroborated by autopsies of COVID-19 patients. And we find patients with a high NTproBNP have a significantly increased risk of death which does not receive much attention initially but proves true by the following-up study. Thus, we suggest interpreting machine learning models can offer help to understanding a new virus at the early stage of an outbreak.