Singh, Parminder

e17005 Background: We previously developed a hybrid framework combining LLMs and rule-based algorithms (RBA) for automated risk stratification in localized prostate cancer (locPCa), achieving 89% accuracy. However, clinical T-stage (cTS) inference from narrative MRI reports remained the primary source of errors. Herein, we evaluated whether ensemble approaches using multiple state-of-the-art LLMs could improve cTS inference and overall NCCN risk stratification accuracy. Methods: This study included patients with locPCa presenting with at least one positive prostate biopsy and MRI report available. Building on our previously validated hybrid framework that extracts structured phenotypic variables (PSA, prostate volume, cTS, Gleason patterns, grade group, positive and examined cores) from unstructured reports using LLMs and applies NCCN-concordant RBA for risk stratification, we focused on improving cTS inference. Four HIPAA-compliant LLMs (GPT-4.1, GPT-5.1, Gemini-2.5-Pro, Gemini-2.5-Flash) were evaluated using structured zero-shot prompts iteratively refined through oncologist feedback. Majority voting was employed to generate consensus cTS predictions. Performance was evaluated using weighted accuracy. Results: A total of 358 patients were included. The median age at diagnosis was 65 years (IQR: 60-69); the majority were White (92%) and non-Hispanic (94%). cTS distribution was T2a (51%), T2c (20%), T3a (16%), T3b (7%), T2b (4%), and T4 (2%). The most prevalent risk category was unfavorable intermediate (38%) followed by favorable intermediate (20%), high risk (19%), very high risk (16%), low (5%), and very low (1%). Individual LLM accuracy for cTS inference ranged from 85-90%, with Gemini-2.5-Pro achieving highest accuracy (90%), followed by GPT-4.1 (88%), GPT-5.1 (87%), and Gemini-2.5-Flash (85%). The four-model ensemble reached consensus in 92% of cases, achieving 93% cTS inference accuracy. When integrated into NCCN risk stratification schema, single-model approaches achieved 89-92% accuracy, while ensemble consensus cases achieved 95% accuracy. Conclusions: The ensemble framework using multiple LLMs significantly improves cTS inference and achieves 95% NCCN risk stratification accuracy in consensus cases (92% of patients), representing a 6-percentage-point improvement over our previous single-model framework (89%). This strategy demonstrates that ensemble methods can overcome individual model limitations for complex clinical tasks and holds promise for reliable AI-assisted risk stratification.

e17003 Background: Accurate ascertainment of prostate cancer (PCa) disease states—biochemical recurrence (BCR), castration-sensitive prostate cancer (CSPC), and castration-resistant prostate cancer (CRPC)—is critical for automated clinical trial enrollment, quality improvement initiatives, and real-world evidence generation. Essential information for disease state ascertainment resides in unstructured EHR data, requiring labor-intensive manual abstraction that is inconsistent and not scalable. We developed an automated hybrid AI framework using large language models to identify PCa disease states from real-world EHR data. Methods: This retrospective study included patients with histopathologically confirmed PCa (2017-2022) and ≥3 years follow-up. Sequential EHR data—including PSA and testosterone levels, clinical notes, pathology and radiology reports—were retrieved across temporal points. BCR was identified using rule-based algorithms (post-radical prostatectomy [RP]: PSA ≥0.2 ng/mL ×2; post-radiation therapy [RT]: nadir + 2 ng/mL ×2). CRPC was determined using androgen deprivation therapy (ADT) timing, PSA kinetics, testosterone < 50 ng/dL, and radiographic progression per PCWG/EAU-ASCO criteria. GPT-4o with structured parametrized prompts extracted metastatic status and radiographic progression. Independent clinicians performed manual annotation for validation. The framework was developed on 10% of data with evaluation on 90%. External validation used a held-out cohort of clinical trial patients. Results: Among 200 PCa patients (median age 66 years [IQR 60-71]; 91% White, 93% non-Hispanic), the framework demonstrated high accuracy: PCa diagnosis (92%), RP dates (94%), RT dates (93%), and ADT initiation (92%). For disease state identification, BCR detection achieved 90% accuracy, castration status 93%, and metastatic state (M0/M1) 87%. External validation on trial-enrolled patients confirmed robust performance: 26/31 (84%) CRPC trial patients and 26/27 (96%) CSPC trial patients were correctly classified, validating the framework’s reliability for clinical trial matching. Conclusions: This hybrid LLM-based framework accurately identifies PCa disease states using real-world EHR data with minimal manual intervention. The system offers a scalable, reproducible approach for automated disease state identification, enabling streamlined clinical trial matching and enhanced real-world data curation in oncology.