Mugu, Vamshi

Abstract Objectives Large language models (LLMs) offer potential in natural language processing tasks in healthcare. Due to the need for high accuracy, understanding their limitations is essential. The purpose of this study was to evaluate the performance of LLMs in classifying radiology reports for the presence of pulmonary embolism (PE) under various conditions, including different prompt designs and data perturbations. Materials and Methods In this retrospective, institutional review board approved study, we evaluated 3 Google’s LLMs including Gemini-1.5-Pro, Gemini-1.5-Flash-001, and Gemini-1.5-Flash-002, in classifying 11 999 pulmonary CT angiography radiology reports for PE. Ground truth labels were determined by concordance between a computer vision-based PE detection (CVPED) algorithm and multiple LLM runs under various configurations. Discrepancies between algorithms’ classifications were aggregated and manually reviewed. We evaluated the effects of prompt design, data perturbations, and repeated analyses across geographic cloud regions. Performance metrics were calculated. Results Of 11 999 reports, 1296 (10.8%) were PE-positive. Accuracy across LLMs ranged between 0.953 and 0.996. The highest recall rate for a prompt modified after a review of the misclassified cases (up to 0.997). Few-shot prompting improved recall (up to 0.99), while chain-of-thought generally degraded performance. Gemini-1.5-Flash-002 demonstrated the highest robustness against data perturbations. Geographic cloud region variability was minimal for Gemini-1.5+-Pro, while the Flash models showed stable performance. Discussion and Conclusion LLMs demonstrated high performance in classifying radiology reports, though results varied with prompt design and data quality. These findings underscore the need for systematic evaluation and validation of LLMs for clinical applications, particularly in high-stakes scenarios.

Abstract Incidental findings in diagnostic imaging are common, but follow-up recommendations often lack consistency. The Society of Radiologists in Ultrasound (SRU) issued guidelines in 2021 for managing incidentally detected gallbladder polyps, aiming to balance follow-up with avoiding overtreatment. There is variable adherence to these guidelines in radiology reports, however, which makes it difficult for the clinician to pursue appropriate follow-up for the patient. The purpose of this project is to test the feasibility of a Large Language Model (LLM)–based tool to incorporate SRU guidelines into radiology reports. Additionally, we propose a framework for closely integrating societal follow-up recommendations into radiology reports, using this tool as an example. Following institutional review board approval, we retrospectively reviewed gallbladder ultrasound examinations performed on adult ED patients in 2022. Data on patient demographics and radiology report content were collected. Using the 2021 SRU guidelines, we developed an interactive tool employing a retriever-augmented generator (RAG) and prompt engineering. A board-certified radiologist tested the accuracy, whereas a board-certified emergency medicine physician assessed the clarity and consistency of the recommendations. The interactive tool, GB-PRL, outperformed leading closed-source and open-source LLMs, achieving 100% accuracy in risk categorization and follow-up recommendations on hypothetical user queries (P < 0.001). The tool also showed superior accuracy compared to radiology reports on retrospective data (P = 0.04). Although GB-PRL demonstrated greater clarity and consistency, the improvement from the radiology reports was not statistically significant (P = 0.22). Further work is needed for prospective testing of GB-PRL before integrating it into clinical practice.

To retrospectively evaluate the trends in utilization and results of computed tomography pulmonary angiography (CTPA study) for detection of acute pulmonary embolism (PE) in the hospital inpatients during different phases of COVID-19 public health emergency. We conducted an Institutional Review Board (IRB)-approved retrospective review of CTPA studies for our hospital inpatients in the years from 2019 to 2023, ranging from the prepandemic year (2019) to the year coinciding with the end of public health declarations (2023). Collected characteristics included patient age, patient sex, and result of the study. The utilization of CTPA studies for inpatients was higher in 2020 to 2023, compared to 2019 (3.8% in 2023 compared to 2.4% in 2019, P < .001). The increase in utilization was also statistically significant in each group, when stratified by age and sex (OR = 1.46 for female, OR = 1.71 for male, OR = 1.93 for elderly, and OR = 1.29 for non-elderly inpatients in 2023, compared to 2019). The positivity rate of acute PE in the inpatients was overall higher in 2021, 2022, and 2023, compared to 2019 (for example, 18.5% in 2023 compared to 14.3% in 2019, P = .01). When stratified by age and sex, only the non-elderly patients continued to have a significantly higher rate of acute PE in 2023, compared to 2019 (OR = 1.39). While studies spanning longer time frames and involving multiple institutions are needed to understand and generalize this conclusion, we conclude that the utilization and positivity rates of CTPA studies in inpatients remains high at the end of the COVID-19 pandemic public health emergency.

Evaluate the quantitative, subjective (Deauville score [DS]) and reader agreement differences between standard ordered subset expectation maximization (OSEM) and Bayesian penalized likelihood (BPL) positron emission tomography (PET) reconstruction methods. A retrospective review of 104 F-18 fluorodeoxyglucose PET/computed tomography (CT) exams among 52 patients with diffuse large B-cell lymphoma. An unblinded radiologist moderator reviewed both BPL and OSEM PET/CT exams. Four blinded radiologists then reviewed the annotated cases to provide a visual DS for each annotated lesion. Significant (P < .001) differences in BPL and OSEM PET methods were identified with greater standard uptake value (SUV) maximum and SUV mean for BPL. The DS was altered in 25% of cases when BPL and OSEM were reviewed by the same radiologist. Interobserver DS agreement was higher for OSEM (>1 cm lesion = 0.89 and ≤1 cm lesion = 0.84) compared to BPL (>1 cm lesion = 0.85 and ≤1 cm lesion = 0.81). Among the 4 readers, average intraobserver visual DS agreement between OSEM and BPL was 0.67 for lesions >1cm and 0.4 for lesions ≤1 cm. F-18 Fluorodeoxyglucose PET/CT of diffuse large B-cell lymphoma reconstructed with BPL has higher SUV values, altered DSs and reader agreement when compared to OSEM. This report finds volumetric PET measurements such as metabolic tumor volume to be similar between BPL and OSEM PET reconstructions. Efforts such as adoption of European Association Research Ltd accreditation should be made to harmonize PET data with an aim at balancing the need for harmonization and sensitivity for lesion detection.