Yum, Ji-Eun Irene

e17005 Background: We previously developed a hybrid framework combining LLMs and rule-based algorithms (RBA) for automated risk stratification in localized prostate cancer (locPCa), achieving 89% accuracy. However, clinical T-stage (cTS) inference from narrative MRI reports remained the primary source of errors. Herein, we evaluated whether ensemble approaches using multiple state-of-the-art LLMs could improve cTS inference and overall NCCN risk stratification accuracy. Methods: This study included patients with locPCa presenting with at least one positive prostate biopsy and MRI report available. Building on our previously validated hybrid framework that extracts structured phenotypic variables (PSA, prostate volume, cTS, Gleason patterns, grade group, positive and examined cores) from unstructured reports using LLMs and applies NCCN-concordant RBA for risk stratification, we focused on improving cTS inference. Four HIPAA-compliant LLMs (GPT-4.1, GPT-5.1, Gemini-2.5-Pro, Gemini-2.5-Flash) were evaluated using structured zero-shot prompts iteratively refined through oncologist feedback. Majority voting was employed to generate consensus cTS predictions. Performance was evaluated using weighted accuracy. Results: A total of 358 patients were included. The median age at diagnosis was 65 years (IQR: 60-69); the majority were White (92%) and non-Hispanic (94%). cTS distribution was T2a (51%), T2c (20%), T3a (16%), T3b (7%), T2b (4%), and T4 (2%). The most prevalent risk category was unfavorable intermediate (38%) followed by favorable intermediate (20%), high risk (19%), very high risk (16%), low (5%), and very low (1%). Individual LLM accuracy for cTS inference ranged from 85-90%, with Gemini-2.5-Pro achieving highest accuracy (90%), followed by GPT-4.1 (88%), GPT-5.1 (87%), and Gemini-2.5-Flash (85%). The four-model ensemble reached consensus in 92% of cases, achieving 93% cTS inference accuracy. When integrated into NCCN risk stratification schema, single-model approaches achieved 89-92% accuracy, while ensemble consensus cases achieved 95% accuracy. Conclusions: The ensemble framework using multiple LLMs significantly improves cTS inference and achieves 95% NCCN risk stratification accuracy in consensus cases (92% of patients), representing a 6-percentage-point improvement over our previous single-model framework (89%). This strategy demonstrates that ensemble methods can overcome individual model limitations for complex clinical tasks and holds promise for reliable AI-assisted risk stratification.

5108 Background: We previously proposed a hybrid framework combining LLMs and rule based algorithm (RBA) for automating risk stratification in locPCa. Herein, we aim to validate the risk stratification agent (RSA) in a prospective cohort, develop & evaluate the treatment recommendation agent (TRA) based on NCCN guidelines, and develop an interactive interface to facilitate clinicians for accurate risk stratification and treatment recommendations at the point of care. Methods: This study included pts with locPCa (2004-2024) presenting at Mayo Clinic with at least 1 positive prostate biopsy and MRI report available. For RSA prospective validation, GPT4 extracted key phenotypic variables (PSA, T stage, prostate volume, number of cores, Gleason patterns, grade group) from unstructured MRI and biopsy reports using a zeroshot prompt. An RBA then classified pts into NCCN risk groups. The agent performance was compared with the treating clinician's documentation and evaluated against gold-standard labels manually annotated by two independent clinicians. For development of TRA, two experiments were performed using GPT4 with and without retrieval-augmented generation (RAG) to generate treatment plans. Generated treatment plans were evaluated using NCCN guideline-informed treatment decision tree based algorithm (DTA). Evaluation metrics, weighted – accuracy (acc) and F1, were computed. A clinician facing interface (lisr.org/risk) was developed to provide accurate risk stratification and treatment plans. Results: A total of 858 pts were included (500 for prospective validation, 358 for treatment recommendations). Prospective validation for RSA demonstrated a higher F1 score of 0.89 compared to the treating clinician (F1: 0.58). Treatment recommendation experiments showed that GPT4 with RAG achieved higher acc (64% full – all correct treatment options and 36% partial – at least 1 correct treatment option) compared to GPT4 alone (35% and 65%, respectively). Sensitivity analysis using DTA-informed GPT4 note generation achieved 94% full acc and 6% partial acc. GPT4 alone generated hallucinated treatment options in 71% of cases, while GPT4 with RAG reduced this to 32%. Conclusions: The multi-agent framework based on LLM and RBA achieves high accuracy in risk stratification and treatment recommendation for locPCa. A multi-agent framework with an interactive interface holds high promise to enable efficient, accurate decision-making, and improve locPCa management at the point of care Performance evaluation. RSA(acc; F1) % TRA(full acc; partial acc) % Clinician RSA* GPT4 RAG+GPT4 DTA+GPT4 Overall 60; 58 89; 89 29; 71 64; 36 94; 6 V. High 94; 33 96; 96 55; 45 69; 31 93; 7 High 46; 52 92; 89 - - - Int. Unf 80; 61 91; 90 5; 95 38; 62 93; 7 Int. Fav 57; 64 84; 87 0; 100 98; 2 100; 0 Low 57; 65 87; 80 100; 0 95; 5 90; 10 V. Low 8; 13 50; 67^ - - - *Statistical significance (p<0.01) except for ^V. Low.

745 Background: The combination of EV-P therapy has recently been approved as first-line (1L) therapy for patients with LA/mUC based on the results of the EV-302 trial. Herein, we report the updated results from a cohort of patients with LA/mUC who received EV-P in real world clinical setting. Methods: This retrospective study included patients with LA/mUC (07/2022-08/2024) at the Mayo Clinic who completed at least one cycle of EV-P. The best overall response (BOR) was evaluated using radiographic imaging and was adjudicated as complete response (CR), partial response (PR), stable disease (SD), mixed response (MR), or progressive disease (PD). The median progression-free survival (PFS), defined as the time from treatment initiation to disease progression or death, was estimated using the Kaplan-Meier method. Patients who had not experienced disease progression by their last follow-up were censored for the PFS analysis. Results: A total of 120 patients were included; 79 (65.8%) were males, 41 (34.1%) females, 109 (91%) White, and 119 (99%) were non-Hispanic/Latino. The median age was 72 years (IQR: 65-77). Of these, only 22 (18.3%) had locally advanced disease while the remaining had mUC; 62 (51.6%) had lower tract urothelial carcinoma (LUTC), 36 (30%) had upper tract urothelial carcinoma (UTUC). The median follow-up time was 7.1 months (IQR: 4.93-9.26). In terms of BOR, 56 patients (46.6%) experienced PR, 34 (28.3%) CR, 12 (10%) PD, 7 (5.8%) MR, and 8 (6.6%) experienced SD. 3 (2.5%) patients were not evaluable. The BOR across subgroups, LA and mUC, including LUTC and UTUC, is summarized in the table. Among the responders, the median number of treatment cycles to first response was 3 (IQR: 2.25–4). Within the follow-up, 28 patients (23.3%) experienced eventual radiographic disease progression; death was observed in 25 patients (20.8%). The median PFS was 12.7 months (95% CI: 9.8-NE), and OS was 25.1 months (16.3-NE). Conclusions: In real world setting, the combination of EV-P in LA/mUC demonstrated clinical efficacy compatible with the findings from the EV-302 trial. Our results showed comparable efficacy with EV-P in both UTUC and LUTC, highlighting it's broad therapeutic potential. However, clinical data at longer follow-up is required to fully evaluate overall survival and the durability of response. mUC LA Overall LUTC UTUC Total patients 98 62 36 22 BOR CR 23 (23.4%) 13 (20.9%) 10 (28.5%) 11 (50%) PR 49 (52.6%) 31 (50%) 18 (50%) 7 (31.8%) SD 6 (6.4%) 2 (3.2%) 4 (11.4%) 2 (9.0%) PD 10 (10.7%) 9 (15.5%) 1 (2.8%) 2 (9.0%) MR 7 (7.5%) 5 (8.6%) 2 (5.7%) 0 (0%)

136 Background: The ECOG performance status (PS) is a critical measure for evaluating functional capacity and determining clinical trial eligibility in patients with advanced prostate cancer. We aim to assess the capability of large language models (LLMs) to accurately infer ECOG PS from unstructured oncology notes. Methods: This retrospective study included patients with metastatic castration-resistant prostate cancer (mCRPC) receiving Lutetium-177–PSMA-617 ( 177 Lu) at the Mayo Clinic (2022-2023). Baseline ECOG PS scores were manually curated by a trained clinician from clinical notes recorded within four weeks prior to 177 Lu initiation. Structured zero-shot prompts were then used to prompt GPT-4 to infer ECOG PS scores from unstructured oncology notes. Total dataset was randomly sampled to prompt development (20%) and held-out test set (80%). Three experimental setups were employed to assess model performance: (i) utilizing unmasked clinical notes, (ii) clinical notes with only numerical ECOG PS scores masked while retaining narrative descriptive/assessment by oncologist of ECOG score, and (iii) clinical notes with both numerical scores and narrative ECOG descriptors masked. Final performance was assessed using evaluation metrics (precision, recall, F1 score) on the held-out test set. Results: A total of 240 patients (n: 50 prompt development; n: 190 test set) were included in the analysis. The median age was 70 (IQR: 66-76); a majority of patients were White (n: 229; 95%) and non-Hispanic (n: 233; 97%). The most prevalent ECOG PS score was 0 (n: 140; 58%), followed by 1 (n: 88; 37%), and 2 (n: 12; 5%). The performance was numerically higher when using unmasked notes (weighted average precision: 0.89; recall: 0.89; F1: 0.89), compared to using notes with masked scores only (weighted average precision: 0.76; recall: 0.79; F1: 0.75) and notes with both scores and narrative masked (weighted average precision: 0.57; recall: 0.53; F1: 0.48). Metrics by each ECOG PS score is outlined (Table). Conclusions: Large language models demonstrate significant potential in precisely extracting ECOG performance status from unstructured oncology notes, offering valuable support for treatment decision-making and clinical trial enrollment. However, model performance declines as reliance on implicit reporting within clinical notes increases. Future efforts focusing on prediction of ECOG using a series of notes over time will be the next step. ECOG Unmasked oncology notes ECOG Score masked only ECOG Score + ECOG narrative marked Precision Recall F1 Score Precision Recall F1 Score Precision Recall F1 Score 0 0.89 0.92 0.91 0.75 0.93 0.83 0.59 0.74 0.66 1 0.88 0.86 0.87 0.76 0.74 0.75 0.19 0.47 0.27 2 1.00 0.82 0.90 0.78 0.33 0.47 0.78 0.16 0.26