Testagrose, Conrad

Abstract Radiological protocol selection is a critical but time-consuming step in clinical workflow, requiring radiologists to match patient indications with the appropriate MRI or CT protocol. Manual selection can be prone to delays or potential errors, and automated approaches must contend with substantial data imbalance, site-specific variation, and evolving nomenclature. We investigated whether a large language model (LLM) can support reliable protocol selection at scale and whether retrieval-augmented generation (RAG) offers operational advantages over direct fine-tuning. Using 498,228 patient reports collected across three Mayo Clinic sites (Arizona, Florida, and Rochester) spanning six radiological divisions, we trained site-specific Llama 3.2 3B models for use with and without retrieval augmentation. Division-scoped Facebook AI Similarity Search (FAISS) indexes constructed from procedure and diagnosis text were used to supply contextual evidence in the RAG framework. Both fine-tuned and RAG-augmented models achieved strong performance across sites, with F1 scores of 0.88–0.90. RAG matched or modestly trailed direct fine-tuning overall but delivered consistent gains in specific divisions (e.g., musculoskeletal imaging). Importantly, the RAG model introduced abstention behavior (Not Enough Information), which concentrated in linguistically diverse divisions and provided an interpretable signal of uncertainty. These findings suggest that RAG-based models are viable for division-scoped protocol selection and offer practical advantages. Retrieval indexes can be refreshed far more easily and with fewer resources than retraining LLMs, enabling continual adaptation to evolving clinical workflows. Future prospective deployment will evaluate real-time accuracy, agreement with practitioners, and the role of abstention as a safety mechanism in clinical decision support.