Nguyen, Ryan H.

1617 Background: Large language models (LLMs) are increasingly evaluated for oncology decision support, yet guideline concordance and hallucinations remain variable across cancer types and tasks. Retrieval-augmented generation (RAG) may improve safety by constraining the evidence base used during generation. Prior oncology LLM studies report discordance with guideline-based care, but few describe a reproducible health-system “vetting mechanism” that couples guideline-only retrieval with clinician-scored safety rubrics across multiple oncology domains. Methods: We assembled 316 de-identified oncology cases in a tumor-board template: medical oncology (breast, GI, GU, leukemia; n=216), gynecologic oncology (n=50), and neuro-oncology/CNS metastatic cancer (n=50). Each case was run on three systems: baseline GPT-4 (O1), NCCN-guideline anchored RAG (O2; guideline-constrained retrieval), and a literature-anchored system (OpenEvidence; O3). Oncologists independently scored outputs using a modified Generative Performance Score (mGPS; −1 to +1) combining guideline concordance with hallucination penalties; readability/rationality was scored on a 1–5 Likert scale. Inter-rater reliability was assessed (κ =.63). This workflow was designed as a pre-deployment model credentialing step for health-system governance (source policy → case bank → clinician-scored acceptance criteria). Results: Across all 316 cases (case-averaged), O2 achieved the highest mean total mGPS (0.75) compared with O1 (0.43) and O3 (0.59). In the medical oncology cohort (n=216), O2 outperformed O1 and O3 (mean mGPS 0.72 vs 0.35 and 0.54), with higher guideline concordance (0.85 vs 0.68 and 0.75) and lower hallucination penalties (−0.09 vs −0.25 and −0.19); readability was also highest for O2 (4.17 vs 3.22 and 3.86). In gynecologic oncology (n=50), O2 again performed best (mGPS 0.827) relative to O3 (0.697) and O1 (0.647). In the CNS cohort (n=50), mean composite mGPS favored O2 (0.82) over O3 (0.68) and O1 (0.55), while readability remained stable across models (3.55–3.73). Severe hallucination/unsafe recommendation rate: O1 [26%], O2 [5%], O3 [17%] (pre-specified definition; e.g., major fabricated citation or guideline-discordant harmful recommendation). Conclusions: In a 316-case, multi-domain clinician-scored evaluation, limiting retrieval to oncology guideline sources improved safety-aligned performance relative to unconstrained generation or literature-only anchoring. A guideline-constrained RAG configuration paired with clinician-scored rubrics and tail-risk reporting can serve as a practical health-system vetting mechanism for governance, re-validation, and change control of LLM decision support, including in resource-limited settings where reliable guideline-consistent recommendations are essential.

Abstract Guideline-concordant breast cancer treatment selection is increasingly complex due to rapid evolution in biomarker-driven therapies. Large language models (LLMs), such as GPT-4, offer a potential tool to support clinical decision-making. However, hallucinations and deviation from evidence-based guidelines remain significant concerns. This study evaluated whether retrieval-augmented generation (RAG) using National Comprehensive Cancer Network (NCCN) guidelines improves GPT-4 performance in treatment recommendation tasks. We developed a retrieval-augmented GPT-4 agent (RAG-GPT) by indexing 2025 NCCN Breast Cancer Guidelines into a vector database using sentence embeddings. Forty clinical vignettes were created from published case reports to reflect diverse scenarios, including HER2-low, triple-negative, and BRCA-mutated disease across all stages. Two models—a baseline GPT-4 (without augmentation) and the RAG-GPT—were prompted with structured templates requesting treatment plans, rationale, clinical trial options, and citations.Outputs were independently scored by two blinded oncology-trained reviewers using the modified Generative AI Performance Score (mG-PS). The mG-PS scored outputs for 2 categories, guideline concordance (Gold standard = 1.0, Acceptable = 0.5, Non-concordant = 0.0) and hallucination errors (Severe = -1.0, Moderate = -0.5, Mild = -0.25, None = 0.0). Guideline Concordance was deemed based on NCCN guidelines and what an oncologist would due based on using the guidelines for each case. Hallucinations are a term used for falsehoods in Artificial Intelligence, like made-up facts or studies, incorrect information, improper doses, or inappropriate or wrong treatments. To ensure consistency and minimize inter-rater variability, reviewers were provided with comprehensive scoring guidelines and detailed instructions for each evaluation metric. Scores were normalized between -1.0 and 1.0 for improved evaluation of tools. Readability and rationale clarity were rated on a 5-point Likert scale. Inter-rater reliability was assessed using Cohen’s kappa. The RAG-GPT demonstrated a significantly higher mean mG-PS score (0.599 ± 0.102, 95% CI: 0.498-0.702) compared to the baseline GPT-4, which not only scored lower but had a negative mean (-0.21 ± 0.143, 95% CI: -0.353 to -0.067). For the mG-PS Cohen’s Kappa test resulted in a score of .69 with substantial aggreament. An independent sample t-test showed a difference (t(78) = 29.1, p < 0.0001). Severe hallucinations were also absent in RAG-GPT outputs (0/40) but occurred in 8/40 baseline GPT-4 outputs. The evaluation of readability and rationality also showed notable differences, with the augmented LLM scoring significantly higher (M = 4.35, SD = 0.66) compared to the naïve LLM (M = 3.05, SD = 1.06). An independent samples t-test confirmed this difference t(65.3) = 6.58, p < 0.0001, further supporting the benefit of augmented Large Language models. Cohen’s Kappa test resulted in a score of .76 for a substantial agreement for readability and rational clarity scale. Limitations include small sample size; further fine-tuning and larger real-world validation studies remain necessary. Linking GPT-4 to NCCN guidelines via retrieval-augmented generation significantly improves treatment accuracy and reduces hallucination severity in breast cancer care scenarios. While further refinement is needed, the reduced incidence of severe hallucinations, increase in guideline-concordant recommendations, and enhanced readability findings support further development and prospective validation of guideline-anchored LLMs for clinical use. Citation Format: C. Yost, A. Aseem, B. Callas, S. Monick, L. Bonilla, R. Nguyen, R. Galamaga, S. Osborn, Y. Kumar, N. Ertz-Archambault. Retrieval-augmented GPT-4 improves NCCN-concordant breast cancer treatment recommendations [abstract]. In: Proceedings of the San Antonio Breast Cancer Symposium 2025; 2025 Dec 9-12; San Antonio, TX. Philadelphia (PA): AACR; Clin Cancer Res 2026;32(4 Suppl):Abstract nr PS3-06-29.

OBJECTIVE Chordomas are rare tumors that originate from undifferentiated remnants of the notochord. Currently, there are no established guidelines regarding the choice of adjuvant radiation modality for patients surgically treated for chordomas. Using a nationwide, multicenter database, the authors aimed to compare long-term survival outcomes associated with the use of proton or photon adjuvant therapy for the management of chordomas of skull base and spine. METHODS The National Cancer Database (NCDB) was queried for chordoma cases from 2004 to 2017. Patient, tumor, and treatment characteristics were extracted from the database. The primary outcome was overall survival (OS). Kaplan-Meier survival analyses were conducted to investigate differences in outcome on propensity score–matched cohorts of patients treated with proton or photon adjuvant radiotherapy. RESULTS Of the 3490 patients available, 424 met the inclusion criteria for this study. In the prematching analysis, patients receiving adjuvant photon therapy were significantly older (median age 57.0 vs 45.0 years, p < 0.001) and were more commonly male (61% vs 43%, p < 0.001) compared with those receiving proton therapy. Races were equally distributed among radiotherapy modalities (p = 0.64). Patients with chordomas of the mobile spine or sacrum were less likely to receive proton compared with photon therapy (37% vs 58%). Patients receiving proton therapy were more often represented among private insurance holders (69% vs 52%, p < 0.001) as well as in the highest income quartile (52% vs 40%, p = 0.008). Patients traveled farther to receive proton, as opposed to photon, therapy (median 59.0 vs 34.9 miles, p < 0.001). On postmatching Kaplan-Meier analysis encompassing all chordoma cases, no difference in OS between photon and proton therapy was revealed (HR 0.75, 95% CI 0.39–1.44; p = 0.39). A Kaplan-Meier analysis only including patients with skull base chordomas reached similar results (HR 0.83, 95% CI 0.31–2.22; p = 0.71). In patients with spine chordomas, however, a significant difference was found, as proton therapy exhibited a superior OS over photon therapy (HR 0.28, 95% CI 0.09–0.81; p = 0.012). CONCLUSIONS Based on this nationwide analysis, patients with private insurance and higher income were more likely to receive proton adjuvant radiotherapy, while those with spinal or sacral chordomas were less likely to receive this modality. Despite this disparity, an OS benefit was observed in patients with chordomas of the spine and sacrum who received adjuvant proton therapy, in comparison with a matched cohort of patients treated with photon therapy. Conversely, this advantageous outcome was not evident in cases of chordomas located at the skull base.

11079 Background: Phase 3 randomized clinical trials (RCTs) inform the standard of care in cancer treatment yet have a mixed track record of success. In this study, we sought to 1) determine the overall success rate of phase 3 RCTs; 2) assess whether authorship patterns differ between successful (positive) and unsuccessful (negative) trials; and 3) explore if individual authors have distinct track records of success. Methods: The HemOnc knowledgebase (KB) was queried for initial publications of phase 3 RCTs of systemic anticancer therapy; those without primary outcome described were excluded. RCTs were labeled positive if the primary endpoint was met per HemOnc standard criteria (P value ≤ 0.05 or hazard ratio upper bound ≤ 1 for any experimental arm[s]) and negative if they did not meet these criteria or if the experimental arm was statistically inferior. Trial metadata and author information were extracted from the HemOnc KB. Author positive publication rate (PPR) was defined as % of publications reporting a positive result divided by total number of eligible publications. Prolific authors were defined as those with 6+ eligible publications. Author count for manuscripts reporting positive vs negative results was evaluated with the Mann-Whitney U test; other statistics were descriptive. Results: 3328 studies met criteria as of February 5, 2024; these were associated with 3305 initial publications between 1960-2023, involving 25,412 unique authors. 1474 (44.3%) of the studies were positive. Publications reporting positive trial results had, on average, 3 more authors, median (interquartile range) of 18 (12-23) vs 15 (11-20), p<0.0001. The mean (±SD) PPR for n=1967 prolific authors was 47.7% (±22.3%). The PPR was not related to individual author output; mean PPR for non-prolific authors was 48.0% (SD not reported). Differences in PPR were apparent by the disease under study and the years of publication (Table). Among the 46 prolific authors who had a PPR 2+ SD above the mean, 26 (55%) were from China followed by Italy with 3 (6%). Conclusions: Despite known positive publication bias, the success rate of published phase 3 RCTs remains <50%. Publications reporting positive results had significantly more authors; this finding requires further study including adjustment for trial sample size. Significant variation in PPR amongst fields could be due to watershed therapeutic advances, differential selection of primary endpoints, and/or selective reporting of results. Finally, a signal for geographic association of outlier PPRs suggests the need for further investigation. [Table: see text]

11023 Background: Randomized clinical trials (RCTs) in cancer typically compare experimental to control arm regimens. The choice of an appropriate control (ctrl) arm can have a major impact on the expected success of an RCT. The frequency with which cancer RCTs use control arms based on established dosing and scheduling protocols has not been described. Methods: The HemOnc knowledgebase (KB) was used to identify systemic anti-cancer therapy (SACT) regimen variants, defined as regimens with identical components that differ in dosing, scheduling, and/or route. Non-cancer, nonrandomized, and non-SACT studies were excluded. Study publication year was used to define standard variants as those evaluated in the experimental arm of a positive phase 3 RCT >1 year prior to publication as a control arm, regardless of cancer type or context of treatment. Study-variant dyads were evaluated to determine whether the variant was standard. Success rates of RCTs with standard vs non-standard control arms were evaluated with Fisher’s exact test. Results: 5221 studies were associated with ≥1 named variant in the HemOnc KB, as of 2024-02-06. After exclusions, there were 3511 study-variant dyads (2386 studies; 1714 variants). The 9 most common regimens are shown in the Table. The median (IQR) number of variants per regimen was 2 (1-3); carboplatin & paclitaxel (CP) had the most variants (n=33). Across all control arm study-variant dyads, 2228/3492 (64%) utilized non-standard variants. For example, 60/97 studies (62%) of docetaxel as control used the 75 mg/m2 q3wk variant after it was established in 2000, whereas n=27 others used 19 non-standard variants. Trials that used a standard control arm had a numerically higher success rate, 45% vs 42% (OR 1.10, 95% CI 0.92-1.32). Conclusions: Non-standard regimen variants are frequently used in cancer RCTs. Reasons for this could include toxicity, patient convenience, or emerging data from smaller studies that establish comparable efficacy or improved toxicity to standard variants. However, the a priori efficacy of non-standard control arms is less rigorously established, and trials with standard control arms might be more successful. Future directions include quantifying the granular differences between standard and non-standard variants and investigating whether certain types of non-standard variation (e.g. fewer cycles, lower doses) associate with trial success rates. [Table: see text]

e18790 Background: Patients with SARS-CoV-2 with a diagnosis of cancer have increased risk of severe COVID-19 outcomes compared to patients without cancer. However, little is known regarding outcomes of patients with COVID-19 and cancer in the setting of human immunodeficiency virus (HIV). Given the unique risks of this population, we sought to understand COVID-19 outcomes using registry data. Methods: This is a descriptive research study utilizing the CCC19 registry, an international multi-institutional registry with healthcare provider-reported cases of patients with cancer and COVID-19. Between March 2020-December 2021, 116 persons with HIV (PWH) and 10,642 persons without HIV (PWOH) with laboratory-confirmed SARS-CoV-2 infection were identified as eligible for the analysis. Results: Median follow-up time for both groups was 90 days, with interquartile range (IQR) 30-180 days. Most PWH were actively receiving antiretroviral therapy (ART) at the time of COVID-19 diagnosis, with 71% (n = 82) having named drug information available; bictegravir/emtricitabine/tenofovir was the most common ART (n = 25). PWH were of younger age (median 57.5 yrs [IQR 46.5-63.25] vs 65 yrs [IQR 55-74]), male (81% vs 47%), and either non-Hispanic Black or Hispanic (71% vs 34%) compared to PWOH. 12% of PWH (n = 14) were current smokers compared to 6% of PWOH (n = 638), and more than half in each group were never smokers (51% of PWH and 53% of PWOH). The following comorbidities were identified in PWH vs PWOH: cardiovascular (16% vs 20%), pulmonary (16% vs 20%), renal (15% vs 14%), and diabetes mellitus (18% vs 27%). A higher proportion of PWH had hematologic malignancy compared to PWOH (33% vs 19%). More PWH had active cancer which was progressing at the time of SARS-CoV-2 infection compared to PWOH (24% vs 14%). 44% of PWH (n = 51) had received active systemic anticancer therapy within the 3 months preceding SARS-CoV-2 infection (including cytotoxic, targeted, endocrine therapies, and immunotherapy) compared to 51% of PWOH (n = 5,420). PWH had an increased rate of hospitalization (58% vs 55%) compared to PWOH. Although a lower proportion of PWH required supplemental oxygen during hospitalization compared to PWOH (34% vs 38%) and ICU admission rates were identical between the two groups (16% vs 16%), PWH had an increased rate of mechanical ventilation (14% vs 10%) and death (24% vs 18%) compared to PWOH. Conclusions: This is the first known study describing outcomes of patients with cancer and COVID-19 in the PWH population from a large multinational dataset. PWH have characteristics associated with adverse outcomes in prior analyses (male sex, non-Hispanic Black or Hispanic, hematologic malignancy, progressing cancer) but are notably younger and have fewer comorbidities. HIV infection may portend increased risk of severe COVID-19 and death; however, additional analyses, including multivariable regression, are warranted.

Abstract Background : Patients (pts) with COVID-19 are reported to have increased risk of venous thromboembolism yet bleeding has been an under recognized complication. Rates of bleeding remain unexamined in all patients especially in pts with cancer and COVID-19. Aim: To estimate the incidence of bleeding complication in patients with cancer and COVID 19 Methods: The CCC19 international registry (NCT04354701) aims to investigate complications of COVID-19 in pts with cancer. Our aim was to investigate the frequency of bleeding in hospitalized adult pts with cancer andCOVID-19, enrolled between March 16, 2020 and Feb 8, 2021. The incidence of bleeding complications was captured as defined by CCC19 and included both major and non major bleeding . Associated baseline clinic-pathologic prognostic factors and outcomes such as need for mechanical ventilation, intensive care unit (ICU) admission and mortality rates were assessed Results :3849 pts met analysis inclusion criteria. Bleeding was reported in 276 (7%) pts with median age of 70years; incidence was 6.6 % in females and 7.6 % in males, 6.5% in non-Hispanic white pts, 8.2 % in non-Hispanic Black pts, and 7.8 % in Hispanic pts. 74% had solid cancer and 29% had hematologic malignancies, 33% had received anti-cancer therapy in preceding 30 days, and 8% had surgery within 4weeks. In pts taking antiplatelet or anticoagulant medications at baseline, 7.2% developed bleeding. Need for mechanical ventilation, ICU admission, 30-day mortality, and total mortality were significantly higher in those with bleeding complications compared to those without, p&lt;0.05 Conclusion : We describe the incidence of bleeding in a large cohort of pts with cancer and COVID-19. Bleeding events were observed in those with adverse outcomes including mechanical ventilation, ICU admission, and high mortality; the overall mortality of 43% in patients with bleeding complications is especially notable. This important complication may reflect underlying COVID-19 pathophysiology as well as iatrogenic causes. Figure 1 Figure 1. Disclosures Kumar: Diagnostica Stago: Honoraria. Zon: AMAGMA AND RLZ: Consultancy, Current holder of individual stocks in a privately-held company. Byeff: Pfizer, BMS, Takeda,Teva, Merck, United health: Consultancy, Current equity holder in publicly-traded company, Current holder of stock options in a privately-held company. Nagaraj: Novartis: Research Funding. Hwang: astrazaneca,Merck,bayer, Genentech: Consultancy, Research Funding. McKay: Myovant: Consultancy; Bayer: Membership on an entity's Board of Directors or advisory committees; AstraZeneca: Consultancy, Membership on an entity's Board of Directors or advisory committees; Exelixis: Consultancy, Membership on an entity's Board of Directors or advisory committees; Calithera: Membership on an entity's Board of Directors or advisory committees; Tempus: Research Funding; Merck: Consultancy, Membership on an entity's Board of Directors or advisory committees; Tempus: Membership on an entity's Board of Directors or advisory committees; Pfizer: Membership on an entity's Board of Directors or advisory committees, Research Funding; Janssen: Membership on an entity's Board of Directors or advisory committees; Bristol Myers Squibb: Consultancy, Membership on an entity's Board of Directors or advisory committees; Sanofi: Membership on an entity's Board of Directors or advisory committees; Novartis: Membership on an entity's Board of Directors or advisory committees; Dendreon: Consultancy; Caris: Other: Serves as a molecular tumor board ; Vividion: Consultancy; Sorrento Therapeutics: Consultancy; Bayer: Research Funding. Warner: Westat, Hemonc.org: Consultancy, Current holder of stock options in a privately-held company. Connors: Pfizer: Honoraria; CSL Behring: Research Funding; Alnylam: Consultancy; Bristol-Myers Squibb: Honoraria; takeda: Honoraria; Abbott: Consultancy. Rosovsky: Janssen: Consultancy, Research Funding; BMS: Consultancy, Research Funding; Inari: Consultancy, Membership on an entity's Board of Directors or advisory committees; Dova: Consultancy, Membership on an entity's Board of Directors or advisory committees.