Abstract Background: Accurate staging is essential for treatment selection, prognosis assessment, and trial eligibility. While prior studies attempted automating pathological (p) staging, few address clinical (c) staging and no studies have compared LLM-derived staging with clinician staging during clinical visit and retrospective cancer registry staging. Methods: We developed a multi-agent framework using Gemini-2.0-Flash-001 and GPT-5 to (1) identify reports (2) extract key data, and (3) apply AJCC 8th edition staging criteria. LLM staging was compared with clinician documentation and registry staging in breast cancer patients. Clinician-registry agreement served as the reference standard. Two independent breast oncologists reviewed discordant cases. Results: We analyzed 122 randomly selected breast cancer patients across all three Mayo Clinic sites from 2018 - 2023. LLM performance matched or exceeded human inter-rater agreement for pathological staging (95-99.2% vs. 95-98.4%). Clinical staging was more challenging, with LLM concordance of 73-77.9% for cT and 87-89.3% for cN versus 87.8% and 91.0% clinician-registry agreement. Among 89 patients with clinician-registry concordance, LLM concordance was 98.9% for pT/pN, 79.8% for cT, and 91.0% for cN. Experts favored LLM staging in 27.8% (5/18) of cT and 25.0% (2/8) of cN discordances. Error analysis revealed LLM challenges in handling non-mass enhancements (6 cases), selecting radiologic measurements (2 cases), identifying discrete masses (2 cases), and miscellaneous (3 cases). Conclusion: LLMs achieved human-level performance for pathological staging (98%), supporting human-in-the-loop deployment. For clinical staging (79.8% cT, 91.0% cN), future work must enhance multimodal reasoning and integrate physical examination data. Prospective validation is needed to assess real-world impact. With targeted refinements and oversight, automated staging can transform registry workflows while augmenting clinical decision-making. Citation Format: Arshad Mohammed, Umair Ayub, Pooja Advani, Shakeela W. Bahadur, Amye J. Tevaarwerk, Tufia C. Haddad, Elisabeth I. Heath, Brenda J. Ernst, Ben Zhou, Cui Tao, Sara J. Holton, Karthik V. Giridhar, Irbaz B. Riaz. Automating clinical and pathological staging for breast cancer patients [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2026; Part 1 (Regular Abstracts); 2026 Apr 17-22; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2026;86(7 Suppl):Abstract nr 2744.
Leveraging large language models (LLM) to guide adjuvant chemotherapy (AC) recommendations in stage III colorectal cancer (CRC).
103 Background: Clinicians spend significant time extracting and synthesizing information to risk-stratify and make adjuvant chemotherapy (AC) recommendations for CRC. By automating parts of this process, LLMs could help reduce documentation burden and variability in decision-making. Methods: We selected 144 stage III, pMMR CRC patients treated at Mayo Clinic (2017–2021). HIPAA-compliant Gemini-2.0-Flash-001 and GPT-4o extracted clinical variables (age, neuropathy grade, pathological stage) from EHRs and generated AC recommendations. Two approaches were compared: (1) a rule-based algorithm with nested if/else statements translating variables into NCCN-based recommendations, and (2) a dynamic LLM agent generating recommendations using a structured prompt informed by NCCN guidelines. Concordance with treating physicians was assessed, with disagreements reviewed by a blinded GI oncologist. Results: The dataset included 144 patients with median age 67 (IQR 66–68), 48.6% female. The LLM pipeline extracted neuropathy grade with 100% accuracy and pathological T/N stages with 99.3% accuracy. Concordance with physician recommendations was 93.1% for the rule-based algorithm and 62.5% for the LLM agent. In discordant cases, blinded adjudication by a GI oncologist favored the algorithm and LLM agent over physicians in most cases (Table 1). Conclusions: This exploratory work shows LLM agents can reliably extract variables and generate evidence-based treatment recommendations for stage III CRC. The high rate of expert preference for LLM recommendations in discordant cases is intriguing but limited by small sample size and possible adjudicator bias. Discordant physician recommendations reflect the complexity of decision-making, including patient preferences, performance status, comorbidities, and subtle findings not fully captured in EHRs. Rather than replacing judgment, we envision LLMs as decision support tools—analogous to an advanced trainee reviewing charts and presenting evidence-based recommendations to the attending physician with ultimate authority. Larger multi-institutional studies with multiple adjudicators are needed to validate these findings and clarify AI’s role in oncology. Patient characteristics, LLM extraction accuracy, and concordance with physician recommendations. Metric Result Total patients 144 Median age (IQR) 67 (66-68) Female (%) 48.6% LLM extraction accuracy – Neuropathy grade 100% LLM extraction accuracy – Pathological T and N stages 99.3% Primary Concordance – Rule-based algorithm vs. physician 93.1% Adjudicator preference of rule-based algorithm over treating physician for discordant cases 100% Concordance – LLM agent vs. physician 62.5% Adjudicator preference of LLM agent over treating physician for discordant cases 82.0%
Reimagining Cancer Care With Generative Artificial Intelligence: The Promise of Large Language Models
The emergence of state-of-the-art large language models (LLMs), which hold the ability to generalize to diverse natural language processing tasks, has led to new opportunities in health care. Oncology is especially well-suited to leverage these resources as the journeys of patients with cancer inherently yield extensive, longitudinal data sets comprising clinical narratives, pathology and radiology reports, and genomic sequencing reports. This review begins with an overview of the fundamental concepts behind LLMs, including the definitions, architecture, training paradigm, and performance optimization through prompt engineering and retrieval-augmented generation. We also take a moment to explore the newly emerging paradigm of LLMs in a multiagentic framework. We then synthesize current research on how LLMs may benefit stakeholders within the practice of oncology, including patients, oncologists, researchers, and learners. Finally, we address the limitations and risks of LLMs, including hallucinations, inherent biases, patient privacy, and clinician deskilling. While research thus far shows significant potential for LLMs to transform cancer care, necessary future directions include studies emphasizing patient stakeholder perspectives on LLM incorporation in clinical workflows, the development of relevant clinical benchmarks for LLM evaluation, a greater focus on real-world prospective testing, and deeper exploration of LLM reasoning capabilities.
Inferring ECOG performance status (PS) using large language models in patients with advanced prostate cancer.
136 Background: The ECOG performance status (PS) is a critical measure for evaluating functional capacity and determining clinical trial eligibility in patients with advanced prostate cancer. We aim to assess the capability of large language models (LLMs) to accurately infer ECOG PS from unstructured oncology notes. Methods: This retrospective study included patients with metastatic castration-resistant prostate cancer (mCRPC) receiving Lutetium-177–PSMA-617 ( 177 Lu) at the Mayo Clinic (2022-2023). Baseline ECOG PS scores were manually curated by a trained clinician from clinical notes recorded within four weeks prior to 177 Lu initiation. Structured zero-shot prompts were then used to prompt GPT-4 to infer ECOG PS scores from unstructured oncology notes. Total dataset was randomly sampled to prompt development (20%) and held-out test set (80%). Three experimental setups were employed to assess model performance: (i) utilizing unmasked clinical notes, (ii) clinical notes with only numerical ECOG PS scores masked while retaining narrative descriptive/assessment by oncologist of ECOG score, and (iii) clinical notes with both numerical scores and narrative ECOG descriptors masked. Final performance was assessed using evaluation metrics (precision, recall, F1 score) on the held-out test set. Results: A total of 240 patients (n: 50 prompt development; n: 190 test set) were included in the analysis. The median age was 70 (IQR: 66-76); a majority of patients were White (n: 229; 95%) and non-Hispanic (n: 233; 97%). The most prevalent ECOG PS score was 0 (n: 140; 58%), followed by 1 (n: 88; 37%), and 2 (n: 12; 5%). The performance was numerically higher when using unmasked notes (weighted average precision: 0.89; recall: 0.89; F1: 0.89), compared to using notes with masked scores only (weighted average precision: 0.76; recall: 0.79; F1: 0.75) and notes with both scores and narrative masked (weighted average precision: 0.57; recall: 0.53; F1: 0.48). Metrics by each ECOG PS score is outlined (Table). Conclusions: Large language models demonstrate significant potential in precisely extracting ECOG performance status from unstructured oncology notes, offering valuable support for treatment decision-making and clinical trial enrollment. However, model performance declines as reliance on implicit reporting within clinical notes increases. Future efforts focusing on prediction of ECOG using a series of notes over time will be the next step. ECOG Unmasked oncology notes ECOG Score masked only ECOG Score + ECOG narrative marked Precision Recall F1 Score Precision Recall F1 Score Precision Recall F1 Score 0 0.89 0.92 0.91 0.75 0.93 0.83 0.59 0.74 0.66 1 0.88 0.86 0.87 0.76 0.74 0.75 0.19 0.47 0.27 2 1.00 0.82 0.90 0.78 0.33 0.47 0.78 0.16 0.26
Guidelines for the use and interpretation of assays for monitoring autophagy (4th edition)
1