Yost, Connor

Artificial intelligence (AI) is now applied across oncology, including imaging, pathology, therapy selection, toxicity assessment, and survival modeling, and large language models (LLMs) have been adopted particularly quickly. Reported physician use of AI roughly doubled between 2023 and 2026. The frameworks used to evaluate these tools have not advanced at the same rate. Many are available, but each was designed for a single stage of the AI lifecycle, and none directly answers the question most relevant to a treating clinician: whether a given tool is appropriate for the individual patient under care. We reviewed the frameworks in current use, including TRIPOD+AI, PROBAST+AI, CLAIM, SPIRIT-AI, CONSORT-AI, DECIDE-AI, MINIMAR, and CREMLS, together with the oncology-specific ESMO EBAI and the newer LLM-specific guidance (TRIPOD-LLM, MI-CLEAR-LLM, CHART, and ESMO ELCAP). Each addresses part of the evaluation problem, but none is sufficient on its own for oncology, where the standard of care changes rapidly, assays drift, biomarker-defined subgroups are small, and a model validated in one period may perform poorly in the next. LLMs introduce additional challenges, including sensitivity to prompting, undisclosed vendor updates, behavioral drift, and hallucination rates approaching 50% on clinical quality-assurance tasks. The predictive-model frameworks were not designed to capture these failure modes. The central argument of this review is that the frameworks the field requires already exist; what is missing is their mandatory adoption. We propose that journals and regulators move from recommending these frameworks to requiring their use, and we outline a cancer-aware evaluation pathway together with the specific responsibilities of authors, reviewers, journals, and regulators in implementing it.

1617 Background: Large language models (LLMs) are increasingly evaluated for oncology decision support, yet guideline concordance and hallucinations remain variable across cancer types and tasks. Retrieval-augmented generation (RAG) may improve safety by constraining the evidence base used during generation. Prior oncology LLM studies report discordance with guideline-based care, but few describe a reproducible health-system “vetting mechanism” that couples guideline-only retrieval with clinician-scored safety rubrics across multiple oncology domains. Methods: We assembled 316 de-identified oncology cases in a tumor-board template: medical oncology (breast, GI, GU, leukemia; n=216), gynecologic oncology (n=50), and neuro-oncology/CNS metastatic cancer (n=50). Each case was run on three systems: baseline GPT-4 (O1), NCCN-guideline anchored RAG (O2; guideline-constrained retrieval), and a literature-anchored system (OpenEvidence; O3). Oncologists independently scored outputs using a modified Generative Performance Score (mGPS; −1 to +1) combining guideline concordance with hallucination penalties; readability/rationality was scored on a 1–5 Likert scale. Inter-rater reliability was assessed (κ =.63). This workflow was designed as a pre-deployment model credentialing step for health-system governance (source policy → case bank → clinician-scored acceptance criteria). Results: Across all 316 cases (case-averaged), O2 achieved the highest mean total mGPS (0.75) compared with O1 (0.43) and O3 (0.59). In the medical oncology cohort (n=216), O2 outperformed O1 and O3 (mean mGPS 0.72 vs 0.35 and 0.54), with higher guideline concordance (0.85 vs 0.68 and 0.75) and lower hallucination penalties (−0.09 vs −0.25 and −0.19); readability was also highest for O2 (4.17 vs 3.22 and 3.86). In gynecologic oncology (n=50), O2 again performed best (mGPS 0.827) relative to O3 (0.697) and O1 (0.647). In the CNS cohort (n=50), mean composite mGPS favored O2 (0.82) over O3 (0.68) and O1 (0.55), while readability remained stable across models (3.55–3.73). Severe hallucination/unsafe recommendation rate: O1 [26%], O2 [5%], O3 [17%] (pre-specified definition; e.g., major fabricated citation or guideline-discordant harmful recommendation). Conclusions: In a 316-case, multi-domain clinician-scored evaluation, limiting retrieval to oncology guideline sources improved safety-aligned performance relative to unconstrained generation or literature-only anchoring. A guideline-constrained RAG configuration paired with clinician-scored rubrics and tail-risk reporting can serve as a practical health-system vetting mechanism for governance, re-validation, and change control of LLM decision support, including in resource-limited settings where reliable guideline-consistent recommendations are essential.

Abstract Guideline-concordant breast cancer treatment selection is increasingly complex due to rapid evolution in biomarker-driven therapies. Large language models (LLMs), such as GPT-4, offer a potential tool to support clinical decision-making. However, hallucinations and deviation from evidence-based guidelines remain significant concerns. This study evaluated whether retrieval-augmented generation (RAG) using National Comprehensive Cancer Network (NCCN) guidelines improves GPT-4 performance in treatment recommendation tasks. We developed a retrieval-augmented GPT-4 agent (RAG-GPT) by indexing 2025 NCCN Breast Cancer Guidelines into a vector database using sentence embeddings. Forty clinical vignettes were created from published case reports to reflect diverse scenarios, including HER2-low, triple-negative, and BRCA-mutated disease across all stages. Two models—a baseline GPT-4 (without augmentation) and the RAG-GPT—were prompted with structured templates requesting treatment plans, rationale, clinical trial options, and citations.Outputs were independently scored by two blinded oncology-trained reviewers using the modified Generative AI Performance Score (mG-PS). The mG-PS scored outputs for 2 categories, guideline concordance (Gold standard = 1.0, Acceptable = 0.5, Non-concordant = 0.0) and hallucination errors (Severe = -1.0, Moderate = -0.5, Mild = -0.25, None = 0.0). Guideline Concordance was deemed based on NCCN guidelines and what an oncologist would due based on using the guidelines for each case. Hallucinations are a term used for falsehoods in Artificial Intelligence, like made-up facts or studies, incorrect information, improper doses, or inappropriate or wrong treatments. To ensure consistency and minimize inter-rater variability, reviewers were provided with comprehensive scoring guidelines and detailed instructions for each evaluation metric. Scores were normalized between -1.0 and 1.0 for improved evaluation of tools. Readability and rationale clarity were rated on a 5-point Likert scale. Inter-rater reliability was assessed using Cohen’s kappa. The RAG-GPT demonstrated a significantly higher mean mG-PS score (0.599 ± 0.102, 95% CI: 0.498-0.702) compared to the baseline GPT-4, which not only scored lower but had a negative mean (-0.21 ± 0.143, 95% CI: -0.353 to -0.067). For the mG-PS Cohen’s Kappa test resulted in a score of .69 with substantial aggreament. An independent sample t-test showed a difference (t(78) = 29.1, p < 0.0001). Severe hallucinations were also absent in RAG-GPT outputs (0/40) but occurred in 8/40 baseline GPT-4 outputs. The evaluation of readability and rationality also showed notable differences, with the augmented LLM scoring significantly higher (M = 4.35, SD = 0.66) compared to the naïve LLM (M = 3.05, SD = 1.06). An independent samples t-test confirmed this difference t(65.3) = 6.58, p < 0.0001, further supporting the benefit of augmented Large Language models. Cohen’s Kappa test resulted in a score of .76 for a substantial agreement for readability and rational clarity scale. Limitations include small sample size; further fine-tuning and larger real-world validation studies remain necessary. Linking GPT-4 to NCCN guidelines via retrieval-augmented generation significantly improves treatment accuracy and reduces hallucination severity in breast cancer care scenarios. While further refinement is needed, the reduced incidence of severe hallucinations, increase in guideline-concordant recommendations, and enhanced readability findings support further development and prospective validation of guideline-anchored LLMs for clinical use. Citation Format: C. Yost, A. Aseem, B. Callas, S. Monick, L. Bonilla, R. Nguyen, R. Galamaga, S. Osborn, Y. Kumar, N. Ertz-Archambault. Retrieval-augmented GPT-4 improves NCCN-concordant breast cancer treatment recommendations [abstract]. In: Proceedings of the San Antonio Breast Cancer Symposium 2025; 2025 Dec 9-12; San Antonio, TX. Philadelphia (PA): AACR; Clin Cancer Res 2026;32(4 Suppl):Abstract nr PS3-06-29.

Abstract While the use of NCCN guidelines is essential in lymphoma care, adherence among oncologists remains highly variable. Real-world concordance ranges from 65–80% overall, dropping to as low as 11–46% for certain lymphoma subtypes. Although 88% of clinicians report using NCCN guidelines for Hodgkin lymphoma, adherence to PET-adapted approaches is inconsistent. This gap highlights a critical need for tools that support guideline-concordant care.Large language models (LLMs) such as GPT–4 promise decision support, yet hallucinations and divergence from evidence-based care remain obstacles. We assessed whether retrieval-augmented generation (RAG) with National Comprehensive Cancer Network (NCCN) guidelines improves GPT–4 performance in lymphoma treatment recommendation tasks. To address this, we developed a RAG-enhanced GPT-4 agent (RAG-GPT) by indexing the 2025 NCCN Lymphoma Guidelines into a vector database. Fifty clinical vignettes spanning diverse lymphoma subtypes were compiled from published case reports; this sample provides 80 % power (α = 0.05) to detect a mean score difference of ≥ 0.18 on the modified Generative AI Performance Score (mG-PS). Guideline concordance was assessed based on NCCN guidelines and clinical judgment reflecting real-world oncological decision-making. Scores were normalized between -1.0 and 1.0 for consistent evaluation. Readability and rationale clarity were rated on a 5-point Likert scale. Baseline GPT-4 and RAG-GPT generated treatment plans via standardized prompts. An oncologist scored all outputs with mG-PS, which rates NCCN concordance (0.0–1.0) and penalizes hallucinations (-0.25 to –1.0). RAG-GPT achieved a higher mean mG-PS (0.32) than baseline GPT–4 (0.10) (t(98)=-2.11, p=0.038; mean score difference ΔmG-PS=0.22). Importantly, RAG-GPT eliminated severe hallucinations, reinforcing confidence in the safety of its treatment reccomendations. Fully concordant “gold-standard” plans were produced in 46 % of vignettes with RAG-GPT versus 22 % with baseline. Readability scores favored RAG-GPT modestly without statistical significance. Lymphoma's bimodal age distribution, heterogeneity, and lack of standardized treatment algorithms, coupled with regimen toxicity and the need to account for patient frailty and comorbidities, create complex clinical scenarios that constrain LLM applicability for nuanced decision-making. Non-Hodgkin lymphoma, particularly relapsed/refractory, transformed disease, or rare subtypes, lacks a defined standard of care, leading reviewers to rely on clinical judgment or published case series beyond NCCN guidelines. This underscores a key limitation of LLM-supported tools: their performance is highly dependent on the presence of clear algorithmic consensus within a given disease. Linking GPT-4 to NCCN guidelines via RAG significantly improved treatment accuracy and reduced hallucination severity in lymphoma, though performance remained lower than prior RAG studies in cancers like breast and CNS suggesting a greater impact on hallucination reduction than on retrieval of a consistent gold standard regimen. These findings support continued development and prospective validation of guideline-anchored LLMs for clinical use.

e21573 Background: In melanoma, biomarkers such as lactate dehydrogenase (LDH), circulating tumor DNA (ctDNA), and the neutrophil-lymphocyte ratio (NLR) have been linked to disease progression. This study aims to compare survival outcomes in melanoma patients based on high versus low levels of these biomarkers, assessing their impact on survival within their respective groups. By evaluating these biomarkers, we hope to identify potential prognostic indicators that could guide treatment decisions and improve personalized care in melanoma management. Methods: We conducted a retrospective survival analysis of 43 melanoma patients, excluding 16 due to incomplete data at Dignity Health Cancer Institute in Phoenix, Arizona. Kaplan-Meier survival analysis was used to compare survival between groups with high vs low levels of biomarkers. Patients were stratified by median values for LDH, NLR, and the presence of detectable ctDNA. Patients included in the analysis were treated systemically with pembrolizumab, encorafenib/binimetinib, nivolumab, or ipilimumab/nivolumab. Survival curves were generated, and statistical significance was assessed using log-rank tests. Data was sourced from our electronic medical record and Signatera™ assay to obtain the primary endpoints. Specimens were standardized in accordance with Natera’s™ specifications. All analyses were executed in GraphPad Prism software. Results: When stratifying by ctDNA, 18 patients had undetectable ctDNA, while 9 had detectable ctDNA with an average level of 1330 (SD = 2067). The median survival for the high ctDNA group was 220 days, whereas the low ctDNA group had significantly longer survival, with only two deaths observed (p = 0.006) within the time period. For LDH, using a cutoff of 204 (IQR = 90.5), patients with high LDH had a median survival of 220 days, while those with low LDH had a significantly longer survival (p = 0.004). When comparing NLR with a cutoff at 2.94 (IQR = 3.7), no significant difference in survival was found between high and low NLR groups (p = 0.72). Conclusions: Our findings show that patients with higher levels of ctDNA and LDH have significantly shorter survival compared to those with low levels of these biomarkers. While NLR did not show a significant survival difference, the study suggests that higher ctDNA and LDH could be important markers for identifying patients at greater risk of poor outcomes. These results underscore the potential value of monitoring these biomarkers to assess disease progression and survival in melanoma patients. Further studies with larger cohorts are needed to validate these findings and refine melanoma prognostic models.