Callas, Bradley

Artificial intelligence (AI) is now applied across oncology, including imaging, pathology, therapy selection, toxicity assessment, and survival modeling, and large language models (LLMs) have been adopted particularly quickly. Reported physician use of AI roughly doubled between 2023 and 2026. The frameworks used to evaluate these tools have not advanced at the same rate. Many are available, but each was designed for a single stage of the AI lifecycle, and none directly answers the question most relevant to a treating clinician: whether a given tool is appropriate for the individual patient under care. We reviewed the frameworks in current use, including TRIPOD+AI, PROBAST+AI, CLAIM, SPIRIT-AI, CONSORT-AI, DECIDE-AI, MINIMAR, and CREMLS, together with the oncology-specific ESMO EBAI and the newer LLM-specific guidance (TRIPOD-LLM, MI-CLEAR-LLM, CHART, and ESMO ELCAP). Each addresses part of the evaluation problem, but none is sufficient on its own for oncology, where the standard of care changes rapidly, assays drift, biomarker-defined subgroups are small, and a model validated in one period may perform poorly in the next. LLMs introduce additional challenges, including sensitivity to prompting, undisclosed vendor updates, behavioral drift, and hallucination rates approaching 50% on clinical quality-assurance tasks. The predictive-model frameworks were not designed to capture these failure modes. The central argument of this review is that the frameworks the field requires already exist; what is missing is their mandatory adoption. We propose that journals and regulators move from recommending these frameworks to requiring their use, and we outline a cancer-aware evaluation pathway together with the specific responsibilities of authors, reviewers, journals, and regulators in implementing it.

1617 Background: Large language models (LLMs) are increasingly evaluated for oncology decision support, yet guideline concordance and hallucinations remain variable across cancer types and tasks. Retrieval-augmented generation (RAG) may improve safety by constraining the evidence base used during generation. Prior oncology LLM studies report discordance with guideline-based care, but few describe a reproducible health-system “vetting mechanism” that couples guideline-only retrieval with clinician-scored safety rubrics across multiple oncology domains. Methods: We assembled 316 de-identified oncology cases in a tumor-board template: medical oncology (breast, GI, GU, leukemia; n=216), gynecologic oncology (n=50), and neuro-oncology/CNS metastatic cancer (n=50). Each case was run on three systems: baseline GPT-4 (O1), NCCN-guideline anchored RAG (O2; guideline-constrained retrieval), and a literature-anchored system (OpenEvidence; O3). Oncologists independently scored outputs using a modified Generative Performance Score (mGPS; −1 to +1) combining guideline concordance with hallucination penalties; readability/rationality was scored on a 1–5 Likert scale. Inter-rater reliability was assessed (κ =.63). This workflow was designed as a pre-deployment model credentialing step for health-system governance (source policy → case bank → clinician-scored acceptance criteria). Results: Across all 316 cases (case-averaged), O2 achieved the highest mean total mGPS (0.75) compared with O1 (0.43) and O3 (0.59). In the medical oncology cohort (n=216), O2 outperformed O1 and O3 (mean mGPS 0.72 vs 0.35 and 0.54), with higher guideline concordance (0.85 vs 0.68 and 0.75) and lower hallucination penalties (−0.09 vs −0.25 and −0.19); readability was also highest for O2 (4.17 vs 3.22 and 3.86). In gynecologic oncology (n=50), O2 again performed best (mGPS 0.827) relative to O3 (0.697) and O1 (0.647). In the CNS cohort (n=50), mean composite mGPS favored O2 (0.82) over O3 (0.68) and O1 (0.55), while readability remained stable across models (3.55–3.73). Severe hallucination/unsafe recommendation rate: O1 [26%], O2 [5%], O3 [17%] (pre-specified definition; e.g., major fabricated citation or guideline-discordant harmful recommendation). Conclusions: In a 316-case, multi-domain clinician-scored evaluation, limiting retrieval to oncology guideline sources improved safety-aligned performance relative to unconstrained generation or literature-only anchoring. A guideline-constrained RAG configuration paired with clinician-scored rubrics and tail-risk reporting can serve as a practical health-system vetting mechanism for governance, re-validation, and change control of LLM decision support, including in resource-limited settings where reliable guideline-consistent recommendations are essential.

Background/Objectives: Primary spinal sarcomas, encompassing both bone and soft tissue histotypes, demand individualized reconstruction due to heterogeneous tumor biology, anatomic complexity, and host environments compromised by radiation, systemic therapy, or prior surgery. This narrative review reframes post-resection spinal reconstruction through a precision-medicine lens. Methods: A structured literature review was performed using PubMed and Scopus, targeting articles published between 2000 and 2026. Searches encompassed spinal sarcoma reconstruction, radiation and fusion, biologic reconstruction, and emerging technologies. Results: Tumor grade, radiation exposure, and systemic therapy timing emerge as multiplicative determinants of reconstructive environment quality, with drug-class-specific perioperative effects warranting stratified management. Vascularized bone grafts achieve reliable fusion in compromised hosts where avascular constructs fail. A precision-oriented reconstructive ladder is proposed as a conceptual, hypothesis-generating framework to guide strategy selection. Hybrid PSI-VBG constructs may further expand reconstructive possibilities. The evidence base remains largely composed of small, retrospective series. Conclusions: Individualized strategies anchored in tumor biology and host environment are the cornerstone of durable spinal sarcoma reconstructions. The proposed framework requires prospective, multi-institutional validation. Standardized outcome definitions, prospective registries, and histotype-stratified analyses are needed to advance the field.

Abstract Guideline-concordant breast cancer treatment selection is increasingly complex due to rapid evolution in biomarker-driven therapies. Large language models (LLMs), such as GPT-4, offer a potential tool to support clinical decision-making. However, hallucinations and deviation from evidence-based guidelines remain significant concerns. This study evaluated whether retrieval-augmented generation (RAG) using National Comprehensive Cancer Network (NCCN) guidelines improves GPT-4 performance in treatment recommendation tasks. We developed a retrieval-augmented GPT-4 agent (RAG-GPT) by indexing 2025 NCCN Breast Cancer Guidelines into a vector database using sentence embeddings. Forty clinical vignettes were created from published case reports to reflect diverse scenarios, including HER2-low, triple-negative, and BRCA-mutated disease across all stages. Two models—a baseline GPT-4 (without augmentation) and the RAG-GPT—were prompted with structured templates requesting treatment plans, rationale, clinical trial options, and citations.Outputs were independently scored by two blinded oncology-trained reviewers using the modified Generative AI Performance Score (mG-PS). The mG-PS scored outputs for 2 categories, guideline concordance (Gold standard = 1.0, Acceptable = 0.5, Non-concordant = 0.0) and hallucination errors (Severe = -1.0, Moderate = -0.5, Mild = -0.25, None = 0.0). Guideline Concordance was deemed based on NCCN guidelines and what an oncologist would due based on using the guidelines for each case. Hallucinations are a term used for falsehoods in Artificial Intelligence, like made-up facts or studies, incorrect information, improper doses, or inappropriate or wrong treatments. To ensure consistency and minimize inter-rater variability, reviewers were provided with comprehensive scoring guidelines and detailed instructions for each evaluation metric. Scores were normalized between -1.0 and 1.0 for improved evaluation of tools. Readability and rationale clarity were rated on a 5-point Likert scale. Inter-rater reliability was assessed using Cohen’s kappa. The RAG-GPT demonstrated a significantly higher mean mG-PS score (0.599 ± 0.102, 95% CI: 0.498-0.702) compared to the baseline GPT-4, which not only scored lower but had a negative mean (-0.21 ± 0.143, 95% CI: -0.353 to -0.067). For the mG-PS Cohen’s Kappa test resulted in a score of .69 with substantial aggreament. An independent sample t-test showed a difference (t(78) = 29.1, p < 0.0001). Severe hallucinations were also absent in RAG-GPT outputs (0/40) but occurred in 8/40 baseline GPT-4 outputs. The evaluation of readability and rationality also showed notable differences, with the augmented LLM scoring significantly higher (M = 4.35, SD = 0.66) compared to the naïve LLM (M = 3.05, SD = 1.06). An independent samples t-test confirmed this difference t(65.3) = 6.58, p < 0.0001, further supporting the benefit of augmented Large Language models. Cohen’s Kappa test resulted in a score of .76 for a substantial agreement for readability and rational clarity scale. Limitations include small sample size; further fine-tuning and larger real-world validation studies remain necessary. Linking GPT-4 to NCCN guidelines via retrieval-augmented generation significantly improves treatment accuracy and reduces hallucination severity in breast cancer care scenarios. While further refinement is needed, the reduced incidence of severe hallucinations, increase in guideline-concordant recommendations, and enhanced readability findings support further development and prospective validation of guideline-anchored LLMs for clinical use. Citation Format: C. Yost, A. Aseem, B. Callas, S. Monick, L. Bonilla, R. Nguyen, R. Galamaga, S. Osborn, Y. Kumar, N. Ertz-Archambault. Retrieval-augmented GPT-4 improves NCCN-concordant breast cancer treatment recommendations [abstract]. In: Proceedings of the San Antonio Breast Cancer Symposium 2025; 2025 Dec 9-12; San Antonio, TX. Philadelphia (PA): AACR; Clin Cancer Res 2026;32(4 Suppl):Abstract nr PS3-06-29.