Banerjee, Imon

Abstract Aims The increasing need for diagnostic echocardiography tests presents challenges in preserving the quality and promptness of reports. While Large Language Models (LLMs) have proven effective in summarizing clinical texts, their application in echo remains underexplored. Methods and results Adult echocardiography studies, conducted at the Mayo Clinic from 1 January 2017 to 31 December 2017, were categorized into two groups: development (all Mayo locations except Arizona) and Arizona validation sets. We adapted open-source LLMs (Llama-2, MedAlpaca, Zephyr, and Flan-T5) using In-Context Learning and Quantized Low-Rank Adaptation fine-tuning (FT) for echo report summarization from ‘Findings’ to ‘Impressions.’ Against cardiologist-generated Impressions, the models’ performance was assessed both quantitatively with automatic metrics and qualitatively by cardiologists. The development dataset included 97 506 reports from 71 717 unique patients, predominantly male (55.4%), with an average age of 64.3 ± 15.8 years. EchoGPT, a fine-tuned Llama-2 model, outperformed other models with win rates ranging from 87% to 99% in various automatic metrics, and produced reports comparable to cardiologists in qualitative review (significantly preferred in conciseness (P < 0.001), with no significant preference in completeness, correctness, and clinical utility). Correlations between automatic and human metrics were fair to modest, with the best being RadGraph F1 scores vs. clinical utility (r = 0.42) and automatic metrics showed insensitivity (0–5% drop) to changes in measurement numbers. Conclusion EchoGPT can generate draft reports for human review and approval, helping to streamline the workflow. However, scalable evaluation approaches dedicated to echo reports remains necessary.

Abstract Critical findings in radiology reports are life threatening conditions that need to be communicated promptly to physicians (“critical findings”) for timely man-agement of patients. Flagging radiology reports of such incidents could facilitate opportune communication of critical findings. With advancements in natural language processing (NLP), large language models (LLMs) can be trained with task-specific instructions and examples to mine information from narrative texts. We believe that similar methods can be applied to radiology reports to identify and extract critical findings from these reports. However, due to the rarity of such critical events, there is a dearth of manually labeled datasets of critical findings in radiology reports. To overcome this limitation, we train instruction-tuned MISTRAL-based language models in a two-phase weakly supervised fine-tuning setup on unlabeled radiology reports from Mayo Clinic (n=15000). The weakly fine-tuned model is then used to automatically extract critical terms from both internal and external test datasets - Mayo Clinic (n=80) and MIMIC-III (n=123) 1 respectively against the expert-annotation. We also evaluated performance of the models on a large-scale MIMIC-IV reports (n=5000) using automated LLM-aided evaluation metrics - G-eval and Prometheus. We observe that, for both manual and LLM-based evaluations, weakly supervised fine-tuning improves model performance, showing successful task-specific alignment. For community use, we released the trained model with open-source academic license∗

Abstract Objective To develop machine learning models using patient and migraine features that can predict treatment responses to commonly used migraine preventive medications. Background Currently, there is no accurate way to predict response to migraine preventive medications, and the standard trial‐and‐error approach is inefficient. Methods In this cohort study, we analyzed data from the Mayo Clinic Headache database prospectively collected from 2001 to December 2023. Adult patients with migraine completed questionnaires during their initial headache consultation to record detailed clinical features and then at each follow‐up to track preventive medication changes and monthly headache days. We included patients treated with at least one of the following migraine preventive medications: topiramate, beta‐blockers (propranolol, metoprolol, atenolol, nadolol, timolol), tricyclic antidepressants (amitriptyline, nortriptyline), verapamil, gabapentin, onabotulinumtoxinA, and calcitonin gene‐related peptide (CGRP) monoclonal antibodies (mAbs) (erenumab, fremanezumab, galcanezumab, eptinezumab). We pre‐trained a deep neural network, “TabNet,” using 145 variables, then employed TabNet‐embedded data to construct prediction models for each medication to predict binary outcomes (responder vs. non‐responder). A treatment responder was defined as having at least a 30% reduction in monthly headache days from baseline. All model performances were evaluated, and metrics were reported in the held‐out test set (train 85%, test 15%). SHapley Additive exPlanations (SHAP) were conducted to determine variable importance. Results Our final analysis included 4260 patients. The responder rate for each medication ranged from 28.7% to 34.9%, and the mean time to treatment outcome for each medication ranged from 151.3 to 209.5 days. The CGRP mAb prediction model achieved a high area under the receiver operating characteristics curve (AUC) of 0.825 (95% confidence interval [CI] 0.726, 0.920) and an accuracy of 0.80 (95% CI 0.70, 0.88). The AUCs of prediction models for beta‐blockers, tricyclic antidepressants, topiramate, verapamil, gabapentin, and onabotulinumtoxinA were: 0.664 (95% CI 0.579, 0.745), 0.611 (95% CI 0.562, 0.682), 0.605 (95% CI 0.520, 0.688), 0.673 (95% CI 0.569, 0.724), 0.628 (0.533, 0.661), and 0.581 (95% CI 0.550, 0.632), respectively. Baseline monthly headache days, age, body mass index (BMI), duration of migraine attacks, responses to previous medication trials, cranial autonomic symptoms, family history of headache, and migraine attack triggers were among the most important variables across all models. A variable could have different contributions; for example, lower BMI predicts responsiveness to CGRP mAbs and beta‐blockers, while higher BMI predicts responsiveness to onabotulinumtoxinA, topiramate, and gabapentin. Conclusion We developed an accurate prediction model for CGRP mAbs treatment response, leveraging detailed migraine features gathered from a headache questionnaire before starting treatment. Employing the same methods, the model performances for other medications were less impressive, though similar to the machine learning models reported in the literature for other diseases. This may be due to CGRP mAbs being migraine‐specific. Incorporating medical comorbidities, genomic, and imaging factors might enhance the model performance. We demonstrated that migraine characteristics are important in predicting treatment responses and identified the most crucial predictors for each of the seven types of preventive medications. Our results suggest that precision migraine treatment is feasible.

PURPOSE Patient-centered outcomes (PCOs) are pivotal in cancer treatment, as they directly reflect patients' quality of life. Although multiple studies suggest that factors affecting breast cancer–related morbidity and survival are influenced by treatment side effects and adherence to long-term treatment, such data are generally only available on a smaller scale or from a single center. The primary challenge with collecting these data is that the outcomes are captured as free text in clinical narratives written by clinicians. MATERIALS AND METHODS Given the complexity of PCO documentation in these narratives, computerized methods are necessary to unlock the wealth of information buried in unstructured text notes that often document PCOs. Inspired by the success of large language models (LLMs), we examined the adaptability of three LLMs, GPT-2, BioGPT, and PMC-LLaMA, on PCO tasks across three institutions, Mayo Clinic, Emory University Hospital, and Stanford University. We developed an open-source framework for fine-tuning LLM that can directly extract the five different categories of PCO from the clinic notes. RESULTS We found that these LLMs without fine-tuning (zero-shot) struggle with challenging PCO extraction tasks, displaying almost random performance, even with some task-specific examples (few-shot learning). The performance of our fine-tuned, task-specific models is notably superior compared with their non–fine-tuned LLM models. Moreover, the fine-tuned GPT-2 model has demonstrated a significantly better performance than the other two larger LLMs. CONCLUSION Our discovery indicates that although LLMs serve as effective general-purpose models for tasks across various domains, they require fine-tuning when applied to the clinician domain. Our proposed approach has the potential to lead more efficient, adaptable models for PCO information extraction, reducing reliance on extensive computational resources while still delivering superior performance for specific tasks.

Acute kidney injury (AKI) is a commonly encountered medical problem that is associated with poor health outcomes in AKI survivors, including increased mortality and re-admission to the hospital. Despite their high-risk status, only a small fraction (< 10%) of patients receive specialized nephrologist follow-up after AKI event. To address the gap in care for AKI patients, this work proposes an artificial intelligence (AI) based fusion technique that combines patient's single-lead electrocardiograph (ECG) and demographics to predict AKI recurrence 3-7 days before onset. The ECG data is analyzed with an on-chip reservoir-computer (RC) prototyped in 28nm CMOS process to create a compressed representation for predicting AKI onset from ECG. After fusion with demographics, the proposed technique is able to predict AKI recurrence 3-7 days before onset with 75.8% accuracy when evaluated on a retrospective patient dataset collected from Mayo Clinic Enterprise.

Abstract Deep learning may detect biologically important signals embedded in tumor morphologic features that confer distinct prognoses. Tumor morphologic features were quantified to enhance patient risk stratification within DNA mismatch repair (MMR) groups using deep learning. Using a quantitative segmentation algorithm (QuantCRC) that identifies 15 distinct morphologic features, we analyzed 402 resected stage III colon carcinomas [191 deficient (d)-MMR; 189 proficient (p)-MMR] from participants in a phase III trial of FOLFOX-based adjuvant chemotherapy. Results were validated in an independent cohort (176 d-MMR; 1,094 p-MMR). Association of morphologic features with clinicopathologic variables, MMR, KRAS, BRAFV600E, and time-to-recurrence (TTR) was determined. Multivariable Cox proportional hazards models were developed to predict TTR. Tumor morphologic features differed significantly by MMR status. Cancers with p-MMR had more immature desmoplastic stroma. Tumors with d-MMR had increased inflammatory stroma, epithelial tumor-infiltrating lymphocytes (TIL), high-grade histology, mucin, and signet ring cells. Stromal subtype did not differ by BRAFV600E or KRAS status. In p-MMR tumors, multivariable analysis identified tumor-stroma ratio (TSR) as the strongest feature associated with TTR [HRadj 2.02; 95% confidence interval (CI), 1.14–3.57; P = 0.018; 3-year recurrence: 40.2% vs. 20.4%; Q1 vs. Q2–4]. Among d-MMR tumors, extent of inflammatory stroma (continuous HRadj 0.98; 95% CI, 0.96–0.99; P = 0.028; 3-year recurrence: 13.3% vs. 33.4%, Q4 vs. Q1) and N stage were the most robust prognostically. Association of TSR with TTR was independently validated. In conclusion, QuantCRC can quantify morphologic differences within MMR groups in routine tumor sections to determine their relative contributions to patient prognosis, and may elucidate relevant pathophysiologic mechanisms driving prognosis. Significance: A deep learning algorithm can quantify tumor morphologic features that may reflect underlying mechanisms driving prognosis within MMR groups. TSR was the most robust morphologic feature associated with TTR in p-MMR colon cancers. Extent of inflammatory stroma and N stage were the strongest prognostic features in d-MMR tumors. TIL density was not independently prognostic in either MMR group.

Abstract Artificial intelligence (AI) models based on deep learning of mammogram images have been developed for lesion detection and diagnosis as well as breast cancer (BC) risk. We and others have shown that these AI models, coupled with breast density, predict long-term breast cancer risk. Common germline genetic variation, in particular a polygenic risk score (PRS) that captures the cumulative impact of hundreds of common risk variants, has been shown associated with 3-fold increased BC risk. The BC-PRS may complement imaging risk factors to improve identification of women at high BC risk and better target prevention strategies. We evaluated the contribution of a BC-PRS to two imaging-based AI models (Transpara AI cancer detection system and Mirai 5-year risk) and breast density measures to long-term BC risk. We hypothesized that the BC-PRS will improve long-term risk prediction above AI and density measures. We examined this hypothesis within a cohort of 10,271 women enrolled to the Mayo Clinic Biobank between 2009 and 2015. Eligible women had no prior cancer, were ages 35 to 85, had a full-field digital screening mammogram within two years of enrollment and genotyping by Regeneron. Incident invasive and DCIS BC was identified through the Mayo tumor registry and follow-up questionnaires, and person years were calculated from date of mammogram through date of BC, last mammogram at Mayo or last questionnaire. We estimated Transpara AI malignancy score (1-10), 5-year Mirai risk and Volpara volumetric density measures (volumetric percent density, dense volume) on screening mammograms at enrollment and obtained clinical BI-RADS density. We calculated the BC-PRS using 268 SNPs (of the 313-SNP score) and the weighted sum of the SNPs using odds ratios from the largest BC GWAS of European ancestry to date. We performed cox proportional hazards regression adjusting for age and BMI to estimate hazard ratios (HRs), 95% confidence intervals (CI), and C-statistics (AUC) to describe the contribution of BC-PRS (per standard deviation (SD)) to the association of AI scores and breast density with BC risk. Likelihood ratio tests (LRT) and bootstrapping were used to compare model performance with vs. without inclusion of BC-PRS, at 10-years follow-up. Comparisons to the Tyrer-Cuzick model are ongoing. Over a median of 9.1 years (Interquartile range, 6.8 to 10.6), 446 incident BC were identified of which 285 invasive and 103 DCIS were confirmed by pathology report (60 were self-reported); 399 (256 invasive) occurred within 10 years and were used in primary analyses. Transpara score [HR per 1 unit=1.18 (1.14, 1.23), AUC=0.655 (0.628, 0.682)] and Mirai risk [HR per SD log risk=1.54 (1.43, 1.65), AUC=0.681 (0.655, 0.706)] were both associated with BC. All three breast density measures were also significantly associated with BC, including in models with AI scores (Table). Addition of the BC-PRS to models with AI scores and breast density improved both model fit (All PLRT &lt; 0.001; Table) and discrimination, with increases in AUC ranging from 0.033 to 0.034 for models with Transpara and density and 0.025 to 0.026 for models with Mirai and density (all P-values &lt; 0.001; Table). Analyses restricted to invasive BC showed identical results. But, the BC-PRS contribution at 5-year follow-up was attenuated. In this cohort study, we found BC-PRS contributed to long-term risk prediction of BC above imaging AI models and breast density. Imaging and genetic risk factors are complementary and may outperform existing risk models and provide more accurate risk assessment to aid in precision prevention strategies. Citation Format: Celine Vachon, Christopher Scott, Imon Banerjee, Ramon Correa Moderno, Dan Hursh, Matthew Jensen, Yvonne Wang, Kathleen Brandt, Karla Kerlikowske, Sandhya Pruthi, Fergus Couch, Stacey Winham. Contribution of a breast cancer polygenic risk score to mammography artificial intelligence models and breast density for long term breast cancer risk prediction [abstract]. In: Proceedings of the 2023 San Antonio Breast Cancer Symposium; 2023 Dec 5-9; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2024;84(9 Suppl):Abstract nr PS10-06.

Background: Differential kidney function assessment is an important part of preoperative evaluation of various urological interventions. It is obtained through dedicated nuclear medical imaging and is not yet implemented through conventional Imaging. Objective: We assess if differential kidney function can be obtained through evaluation of contrast-enhanced computed tomography(CT) using a combination of deep learning and (2D and 3D) radiomic features. Methods: All patients who underwent kidney nuclear scanning at Mayo Clinic sites between 2018–2022 were collected. CT scans of the kidneys were obtained within a 3-month interval before or after the nuclear scans were extracted. Patients who underwent a urological or radiological intervention within this time frame were excluded. A segmentation model was used to segment both kidneys. 2D and 3D radiomics features were extracted and compared between the two kidneys to compute delta radiomics and assess its ability to predict differential kidney function. Performance was reported using receiver operating characteristics, sensitivity, and specificity. Results: Studies from Arizona & Rochester formed our internal dataset ( n = 1,159). Studies from Florida were separately processed as an external test set to validate generalizability. We obtained 323 studies from our internal sites and 39 studies from external sites. The best results were obtained by a random forest model trained on 3D delta radiomics features. This model achieved an area under curve (AUC) of 0.85 and 0.81 on internal and external test sets, while specificity and sensitivity were 0.84,0.68 on the internal set, 0.70, and 0.65 on the external set. Conclusion: This proposed automated pipeline can derive important differential kidney function information from contrast-enhanced CT and reduce the need for dedicated nuclear scans for early-stage differential kidney functional assessment. Clinical Impact: We establish a machine learning methodology for assessing differential kidney function from routine CT without the need for expensive and radioactive nuclear medicine scans.