Loaiza‐Bonilla, Arturo

1617 Background: Large language models (LLMs) are increasingly evaluated for oncology decision support, yet guideline concordance and hallucinations remain variable across cancer types and tasks. Retrieval-augmented generation (RAG) may improve safety by constraining the evidence base used during generation. Prior oncology LLM studies report discordance with guideline-based care, but few describe a reproducible health-system “vetting mechanism” that couples guideline-only retrieval with clinician-scored safety rubrics across multiple oncology domains. Methods: We assembled 316 de-identified oncology cases in a tumor-board template: medical oncology (breast, GI, GU, leukemia; n=216), gynecologic oncology (n=50), and neuro-oncology/CNS metastatic cancer (n=50). Each case was run on three systems: baseline GPT-4 (O1), NCCN-guideline anchored RAG (O2; guideline-constrained retrieval), and a literature-anchored system (OpenEvidence; O3). Oncologists independently scored outputs using a modified Generative Performance Score (mGPS; −1 to +1) combining guideline concordance with hallucination penalties; readability/rationality was scored on a 1–5 Likert scale. Inter-rater reliability was assessed (κ =.63). This workflow was designed as a pre-deployment model credentialing step for health-system governance (source policy → case bank → clinician-scored acceptance criteria). Results: Across all 316 cases (case-averaged), O2 achieved the highest mean total mGPS (0.75) compared with O1 (0.43) and O3 (0.59). In the medical oncology cohort (n=216), O2 outperformed O1 and O3 (mean mGPS 0.72 vs 0.35 and 0.54), with higher guideline concordance (0.85 vs 0.68 and 0.75) and lower hallucination penalties (−0.09 vs −0.25 and −0.19); readability was also highest for O2 (4.17 vs 3.22 and 3.86). In gynecologic oncology (n=50), O2 again performed best (mGPS 0.827) relative to O3 (0.697) and O1 (0.647). In the CNS cohort (n=50), mean composite mGPS favored O2 (0.82) over O3 (0.68) and O1 (0.55), while readability remained stable across models (3.55–3.73). Severe hallucination/unsafe recommendation rate: O1 [26%], O2 [5%], O3 [17%] (pre-specified definition; e.g., major fabricated citation or guideline-discordant harmful recommendation). Conclusions: In a 316-case, multi-domain clinician-scored evaluation, limiting retrieval to oncology guideline sources improved safety-aligned performance relative to unconstrained generation or literature-only anchoring. A guideline-constrained RAG configuration paired with clinician-scored rubrics and tail-risk reporting can serve as a practical health-system vetting mechanism for governance, re-validation, and change control of LLM decision support, including in resource-limited settings where reliable guideline-consistent recommendations are essential.

Retrieval-augmented generation (RAG) is rapidly emerging as a transformative paradigm for large language models (LLMs), especially in high-stakes domains like oncology that demand precision, factual grounding, and up-to-date knowledge. By pairing LLMs with external knowledge repositories, RAG systems explicitly ground model outputs in relevant retrieved documents, helping to reduce hallucinations and ensure responses reflect current evidence. In oncology, where clinical knowledge evolves continually with new research and drug approvals, RAG offers a way to integrate the latest data (e.g., trial results, guidelines, genomic databases) into decision-making. This review synthesizes the technical foundations of RAG, including its architecture and key components, and examines current applications in oncology such as clinical decision support, patient education, radiology reporting, pathology analysis, and genomics-driven precision medicine. We highlight recent studies that demonstrate RAG’s potential—for instance, improving treatment recommendations by incorporating genetic profiles and literature, and enhancing diagnostic accuracy by integrating guidelines. We also discuss emerging developments like multimodal RAG (combining text with imaging or other data), ensemble model approaches, and new explainability tools that trace model outputs to sources. Finally, we critically analyze the limitations and challenges of deploying RAG in healthcare, including computational costs, retrieval errors, noise or conflicts in retrieved information, and ethical and regulatory considerations. While RAG-based systems show promise in augmenting oncologists’ expertise with timely knowledge, careful implementation, high-quality curation of knowledge bases, and human oversight will be crucial for safe and effective adoption in clinical practice.

Abstract While the use of NCCN guidelines is essential in lymphoma care, adherence among oncologists remains highly variable. Real-world concordance ranges from 65–80% overall, dropping to as low as 11–46% for certain lymphoma subtypes. Although 88% of clinicians report using NCCN guidelines for Hodgkin lymphoma, adherence to PET-adapted approaches is inconsistent. This gap highlights a critical need for tools that support guideline-concordant care.Large language models (LLMs) such as GPT–4 promise decision support, yet hallucinations and divergence from evidence-based care remain obstacles. We assessed whether retrieval-augmented generation (RAG) with National Comprehensive Cancer Network (NCCN) guidelines improves GPT–4 performance in lymphoma treatment recommendation tasks. To address this, we developed a RAG-enhanced GPT-4 agent (RAG-GPT) by indexing the 2025 NCCN Lymphoma Guidelines into a vector database. Fifty clinical vignettes spanning diverse lymphoma subtypes were compiled from published case reports; this sample provides 80 % power (α = 0.05) to detect a mean score difference of ≥ 0.18 on the modified Generative AI Performance Score (mG-PS). Guideline concordance was assessed based on NCCN guidelines and clinical judgment reflecting real-world oncological decision-making. Scores were normalized between -1.0 and 1.0 for consistent evaluation. Readability and rationale clarity were rated on a 5-point Likert scale. Baseline GPT-4 and RAG-GPT generated treatment plans via standardized prompts. An oncologist scored all outputs with mG-PS, which rates NCCN concordance (0.0–1.0) and penalizes hallucinations (-0.25 to –1.0). RAG-GPT achieved a higher mean mG-PS (0.32) than baseline GPT–4 (0.10) (t(98)=-2.11, p=0.038; mean score difference ΔmG-PS=0.22). Importantly, RAG-GPT eliminated severe hallucinations, reinforcing confidence in the safety of its treatment reccomendations. Fully concordant “gold-standard” plans were produced in 46 % of vignettes with RAG-GPT versus 22 % with baseline. Readability scores favored RAG-GPT modestly without statistical significance. Lymphoma's bimodal age distribution, heterogeneity, and lack of standardized treatment algorithms, coupled with regimen toxicity and the need to account for patient frailty and comorbidities, create complex clinical scenarios that constrain LLM applicability for nuanced decision-making. Non-Hodgkin lymphoma, particularly relapsed/refractory, transformed disease, or rare subtypes, lacks a defined standard of care, leading reviewers to rely on clinical judgment or published case series beyond NCCN guidelines. This underscores a key limitation of LLM-supported tools: their performance is highly dependent on the presence of clear algorithmic consensus within a given disease. Linking GPT-4 to NCCN guidelines via RAG significantly improved treatment accuracy and reduced hallucination severity in lymphoma, though performance remained lower than prior RAG studies in cancers like breast and CNS suggesting a greater impact on hallucination reduction than on retrieval of a consistent gold standard regimen. These findings support continued development and prospective validation of guideline-anchored LLMs for clinical use.

e18635 Background: With almost 85% of patients treated outside NCI-designated cancer centers, providing comprehensive, state-of-the-art cancer care to millions of patients in the U.S. remains a significant challenge. We aimed to combine cutting edge AI based technology of DLVTB with high value-added services to provide evidence-based care management recommendations and implementation support to the patients and physicians in the community. To further demonstrate the feasibility, reproducibility, scalability, and benefits of DLVTB, we evaluated key outcomes from incorporating DLVTB in a cohort of 35 patients with advanced colorectal adenocarcinoma (CRC). Methods: Our core-enabling technology is a deep learning based Natural Language Understanding Engine that employs (1) natural language processing for medical text digestion and structuring, (2) decision trees and multilayer perception models to produce evidence-based treatment protocols, and (3) natural language based report generation. The technology platform is combined with human support from oncology sub-specialists to deliver a comprehensive Interpretive Report with a prioritized list of recommendations for each patient. These recommendations are operationalized by a case management team to ensure care implementation and monitoring of outcomes. Results: Thirty-five patients with CRC were referred for incorporation of DLVTB into clinical practice. Median age of patients was 57 years with 68.6% males. About 88.6% of the patients were Stage IV and 82.9% were treated by community practice oncologists. Median time since diagnosis was 17 months (1-73 months). Overall, DLVTB-cohort demonstrated an increase in median Overall Survival of 12 months per patient in comparison with historical cohorts. More specifically, DLVTB recommended initial and/or additional biomarker testing for 71% of the patients with further precision oncology-guided treatment recommendation (e.g. an EGFR inhibitor) for 80% of the patients. 63% of the patients were eligible for at least one clinical trial. 58% of the trials identified were within proximity (≤50 miles) of the patient’s primary residence. Thus, DLVTB was able to identify a trial that did not require extensive travel, provided eligibility and actionability closer to the point-of-care. 14% of the DLVTB evaluated patients subsequently enrolled in the recommended clinical trial, far surpassing the national average (3%). Moreover, DLVTB recommended treatments achieved on average savings of $39,194 per patient which is 35% of the average drug cost per patient. Conclusions: Our results demonstrate the feasibility and benefits of incorporating DLVTB into clinical practice. The utilization of DLVTB as a clinical trial enrollment tool and an engine for development of pathways resulting in improved clinical outcomes, in a cost-effective, and innovative care delivery model.