Cook, Cole J.

Abstract Radiological protocol selection is a critical but time-consuming step in clinical workflow, requiring radiologists to match patient indications with the appropriate MRI or CT protocol. Manual selection can be prone to delays or potential errors, and automated approaches must contend with substantial data imbalance, site-specific variation, and evolving nomenclature. We investigated whether a large language model (LLM) can support reliable protocol selection at scale and whether retrieval-augmented generation (RAG) offers operational advantages over direct fine-tuning. Using 498,228 patient reports collected across three Mayo Clinic sites (Arizona, Florida, and Rochester) spanning six radiological divisions, we trained site-specific Llama 3.2 3B models for use with and without retrieval augmentation. Division-scoped Facebook AI Similarity Search (FAISS) indexes constructed from procedure and diagnosis text were used to supply contextual evidence in the RAG framework. Both fine-tuned and RAG-augmented models achieved strong performance across sites, with F1 scores of 0.88–0.90. RAG matched or modestly trailed direct fine-tuning overall but delivered consistent gains in specific divisions (e.g., musculoskeletal imaging). Importantly, the RAG model introduced abstention behavior (Not Enough Information), which concentrated in linguistically diverse divisions and provided an interpretable signal of uncertainty. These findings suggest that RAG-based models are viable for division-scoped protocol selection and offer practical advantages. Retrieval indexes can be refreshed far more easily and with fewer resources than retraining LLMs, enabling continual adaptation to evolving clinical workflows. Future prospective deployment will evaluate real-time accuracy, agreement with practitioners, and the role of abstention as a safety mechanism in clinical decision support.

Abstract Objectives Large language models (LLMs) offer potential in natural language processing tasks in healthcare. Due to the need for high accuracy, understanding their limitations is essential. The purpose of this study was to evaluate the performance of LLMs in classifying radiology reports for the presence of pulmonary embolism (PE) under various conditions, including different prompt designs and data perturbations. Materials and Methods In this retrospective, institutional review board approved study, we evaluated 3 Google’s LLMs including Gemini-1.5-Pro, Gemini-1.5-Flash-001, and Gemini-1.5-Flash-002, in classifying 11 999 pulmonary CT angiography radiology reports for PE. Ground truth labels were determined by concordance between a computer vision-based PE detection (CVPED) algorithm and multiple LLM runs under various configurations. Discrepancies between algorithms’ classifications were aggregated and manually reviewed. We evaluated the effects of prompt design, data perturbations, and repeated analyses across geographic cloud regions. Performance metrics were calculated. Results Of 11 999 reports, 1296 (10.8%) were PE-positive. Accuracy across LLMs ranged between 0.953 and 0.996. The highest recall rate for a prompt modified after a review of the misclassified cases (up to 0.997). Few-shot prompting improved recall (up to 0.99), while chain-of-thought generally degraded performance. Gemini-1.5-Flash-002 demonstrated the highest robustness against data perturbations. Geographic cloud region variability was minimal for Gemini-1.5+-Pro, while the Flash models showed stable performance. Discussion and Conclusion LLMs demonstrated high performance in classifying radiology reports, though results varied with prompt design and data quality. These findings underscore the need for systematic evaluation and validation of LLMs for clinical applications, particularly in high-stakes scenarios.