A Multi-agent Large Language Model Framework to Automatically Assess Performance of a Clinical AI Triage Tool

Purpose: The purpose of this study was to determine if an ensemble of multiple LLM agents could be used collectively to provide a more reliable assessment of a pixel-based AI triage tool than a single LLM. Methods: 29,766 non-contrast CT head exams from fourteen hospitals were processed by a commercial intracranial hemorrhage (ICH) AI detection tool. Radiology reports were analyzed by an ensemble of eight open-source LLM models and a HIPAA compliant internal version of GPT-4o using a single multi-shot prompt that assessed for presence of ICH. 1,726 examples were manually reviewed. Performance characteristics of the eight open-source models and consensus were compared to GPT-4o. Three ideal consensus LLM ensembles were tested for rating the performance of the triage tool. Results: The cohort consisted of 29,766 head CTs exam-report pairs. The highest AUC performance was achieved with llama3.3:70b and GPT-4o (AUC= 0.78). The average precision was highest for Llama3.3:70b and GPT-4o (AP=0.75 & 0.76). Llama3.3:70b had the highest F1 score (0.81) and recall (0.85), greater precision (0.78), specificity (0.72), and MCC (0.57). Using MCC (95% CI) the ideal combination of LLMs were: Full-9 Ensemble 0.571 (0.552-0.591), Top-3 Ensemble 0.558 (0.537-0.579), Consensus 0.556 (0.539-0.574), and GPT4o 0.522 (0.500-0.543). No statistically significant differences were observed between Top-3, Full-9, and Consensus (p > 0.05). Conclusion: An ensemble of medium to large sized open-source LLMs provides a more consistent and reliable method to derive a ground truth retrospective evaluation of a clinical AI triage tool over a single LLM alone.

Coronary Artery Classification and Weakly Supervised Abnormality Localization on Coronary CT Angiography with 3-Dimensional Convolutional Neural Networks

We propose a fully automated algorithm based on a deep-learning framework enabling screening of a Coronary Computed Tomography Angiography (CCTA) examination for confident detection of the presence or complete absence of atherosclerotic plaque of the coronary arteries. The system starts with extracting the coronary arteries and their branches from CCTA datasets and representing them with multi-planar reformatted volumes; pre-processing and augmentation techniques are then applied to increase the robustness and generalization ability of the system. A 3-Dimensional Convolutional Neural Network (3D-CNN) is utilized to model pathological changes (e.g., calcification) in coronary arteries/branches. The system then learns the discriminatory features between vessels with and without atherosclerosis. The discriminative features at the final convolutional layer are visualized with a saliency map approach to localize the visual clues related to atherosclerosis. We have evaluated the system on a reference dataset representing 247 patients with atherosclerosis and 246 patients free of atherosclerosis. With 5-fold cross-validation, an accuracy = 90.9%, with Positive Predictive Value = 58.8%, Sensitivity = 68.9%, Specificity of 93.6%, and Negative Predictive Value = 96.1% are achieved at the artery/branch level with a threshold of 0.5. The average area under the curve = 0.91. The system indicates a high negative predictive value, which may be potentially useful for assisting physicians in identifying patients with no coronary atherosclerosis that need no further diagnostic evaluation.