Evaluation of General Large Language Models in Contextually Assessing Semantic Concepts Extracted from Adult Critical Care Electronic Health Record Notes

The field of healthcare has increasingly turned its focus towards Large Language Models (LLMs) due to their remarkable performance. However, their performance in actual clinical applications has been underexplored. Traditional evaluations based on question-answering tasks don't fully capture the nuanced contexts. This gap highlights the need for more in-depth and practical assessments of LLMs in real-world healthcare settings. Objective: We sought to evaluate the performance of LLMs in the complex clinical context of adult critical care medicine using systematic and comprehensible analytic methods, including clinician annotation and adjudication. Methods: We investigated the performance of three general LLMs in understanding and processing real-world clinical notes. Concepts from 150 clinical notes were identified by MetaMap and then labeled by 9 clinicians. Each LLM's proficiency was evaluated by identifying the temporality and negation of these concepts using different prompts for an in-depth analysis. Results: GPT-4 showed overall superior performance compared to other LLMs. In contrast, both GPT-3.5 and text-davinci-003 exhibit enhanced performance when the appropriate prompting strategies are employed. The GPT family models have demonstrated considerable efficiency, evidenced by their cost-effectiveness and time-saving capabilities. Conclusion: A comprehensive qualitative performance evaluation framework for LLMs is developed and operationalized. This framework goes beyond singular performance aspects. With expert annotations, this methodology not only validates LLMs' capabilities in processing complex medical data but also establishes a benchmark for future LLM evaluations across specialized domains.

Reconsideration on evaluation of machine learning models in continuous monitoring using wearables

This paper explores the challenges in evaluating machine learning (ML) models for continuous health monitoring using wearable devices beyond conventional metrics. We state the complexities posed by real-world variability, disease dynamics, user-specific characteristics, and the prevalence of false notifications, necessitating novel evaluation strategies. Drawing insights from large-scale heart studies, the paper offers a comprehensive guideline for robust ML model evaluation on continuous health monitoring.

Photoplethysmography based atrial fibrillation detection: an updated review from July 2019

Atrial fibrillation (AF) is a prevalent cardiac arrhythmia associated with significant health ramifications, including an elevated susceptibility to ischemic stroke, heart disease, and heightened mortality. Photoplethysmography (PPG) has emerged as a promising technology for continuous AF monitoring for its cost-effectiveness and widespread integration into wearable devices. Our team previously conducted an exhaustive review on PPG-based AF detection before June 2019. However, since then, more advanced technologies have emerged in this field. This paper offers a comprehensive review of the latest advancements in PPG-based AF detection, utilizing digital health and artificial intelligence (AI) solutions, within the timeframe spanning from July 2019 to December 2022. Through extensive exploration of scientific databases, we have identified 59 pertinent studies. Our comprehensive review encompasses an in-depth assessment of the statistical methodologies, traditional machine learning techniques, and deep learning approaches employed in these studies. In addition, we address the challenges encountered in the domain of PPG-based AF detection. Furthermore, we maintain a dedicated website to curate the latest research in this area, with regular updates on a regular basis.

Welch-GAN: Generating realistic photoplethysmography signal from frequency-domain for atrial fibrillation detection

Training machine learning algorithms from a small and imbalanced dataset is often a daunting challenge in medical research. However, it has been shown that the synthetic data generated by data augmentation techniques can enlarge the dataset and contribute to alleviating the imbalance situation. In this study, we propose a novel generative adversarial network (GAN) architecture-Welch-GAN and focused on examining how its influence on classifier performance is related to signal quality and class imbalance within the context of photoplethysmography (PPG)-based atrial fibrillation (AF) detection. Pulse oximetry data were collected from 126 adult patients and augmented using the permutation technique to build a large training set for training an AF detection model based on a one-dimensional residual neural network. To test the model, PPG data were collected from 13 stroke patients and utilized. Four data augmentation methods, including both traditional and GANs, are leveraged as baseline in this study. Three different experiments are designed to investigate each data augmentation methods from the aspect of performance gain, robustness to motion artifact and training sample size, respectively. Compared to the un-augmented data, by training the same AF classification algorithm using augmented data, the AF detection accuracy was significantly improved from 80.36% to over 90% with no compromise on sensitivity nor on negative predicted value. Within each data augmentation techniques, Welch-GAN has shown around 3% superiority in terms of AF detection accuracy compared to the baseline methods, which suggests the state-of-the-art of our proposed Welch-GAN.