Mahmoudi, Elham

Abstract Background Automated data extraction from echocardiography reports could facilitate large-scale registry creation and clinical surveillance of valvular heart diseases (VHD). We evaluated the performance of open-source large language models (LLMs) guided by prompt instructions and chain of thought (CoT) for this task. Methods From consecutive transthoracic echocardiographies performed in our center, we utilized 200 random reports from 2019 for prompt optimization and 1000 from 2023 for evaluation. Five instruction-tuned LLMs (Qwen2.0-72B, Llama3.0-70B, Mixtral8-46.7B, Llama3.0-8B, and Phi3.0-3.8B) were guided by prompt instructions with and without CoT to classify prosthetic valve presence and VHD severity. Performance was evaluated using classification metrics against expert-labeled ground truth. Mean squared error (MSE) was also calculated for predicted severity’s deviation from actual severity. Results With CoT prompting, Llama3.0-70B and Qwen2.0 achieved the highest performance (accuracy: 99.1% and 98.9% for VHD severity; 100% and 99.9% for prosthetic valve; MSE: 0.02 and 0.05, respectively). Smaller models showed lower accuracy for VHD severity (54.1%-85.9%) but maintained high accuracy for prosthetic valve detection (>96%). Chain of thought reasoning yielded higher accuracy for larger models while increasing processing time from 2-25 to 67-154 seconds per report. Based on CoT reasonings, the wrong predictions were mainly due to model outputs being influenced by irrelevant information in the text or failure to follow the prompt instructions. Conclusions Our study demonstrates the near-perfect performance of open-source LLMs for automated echocardiography report interpretation with the purpose of registry formation and disease surveillance. While larger models achieved exceptional accuracy through prompt optimization, practical implementation requires balancing performance with computational efficiency.

Introduction: Calcium scoring computed tomography (CAC) is a widely available and inexpensive modality of screening for atherosclerosis. This study aims to improve the predictive value of CAC by incorporating pericardial fat characteristics in a CatBoost model. Methods: A sample of 7,000 non-contrast clinically indicated CAC scans were randomly selected from a cohort of consecutive Olmsted County MN residents. Known history of atherosclerotic cardiovascular disease or impaired images were excluded. A pericardial fat segmentation model, trained on 191 images of the same dataset with a dice similarity score of 94%, was applied to 6,194 CT scans; Volume and mean Hounsfield unit (mHU) of the output masks were added to age at imaging, radiologist-measured calcium score, male gender, diabetes, and smoking (Figure 1). A 70,20 and 10% split was used to produce the training, validation and test set, respectively. Features were inputted to CatBoost classifier, as the best model tested, to predict four outcomes: acute coronary syndrome (ACS), heart failure (HF), ischemic stroke, and all-cause mortality. Model was evaluated by area under the ROC curve (AUC) on the unseen test set. Results: Mean age was 55.2 (8.8), 55.2 (8.8) and 54.8 (8.5) years in the training, validation and test sets, respectively, and 65%, 67% and 66% of the populations were male. The mean AUC, obtained through bootstrapping with 100 iterations, was 0.81 [0.77-0.85], 0.86 [0.60-0.92], 0.80 [0.73-0.87] and 0.87 [0.81-0.91] for HF, ACS, stroke and death, respectively; The same analysis excluding pericardial fat measurements showed a 3% reduction in the AUC for all of the investigated outcomes (Figure 2). Conclusion: Our study presents a robust risk prediction model for ACS, HF, stroke, and mortality based on CAC scans. The improved performance of the model by incorporating pericardial fat characteristics, highlights the potentials of deep learning in timely inexpensive risk prediction in large populations.