Osorio, Mariana Borras

425 Background: Prostate cancer (PC) ranks among the most prevalent cancers in men, with incidence rising due to factors like aging populations and enhanced screening techniques. Effective data extraction from clinical records is crucial for advancing research and improving patient care; however, traditional manual methods can be time-consuming and susceptible to mistakes. This study seeks to evaluate RadOnc-GPT against human annotation for extracting essential clinical information from prostate cancer clinical records. Methods: RadOnc-GPT is a retrieval augmented generation (RAG)-based chatbot with access to a variety of patient data including treatment details, radiology reports, and clinical notes. RadOnc-GPT uses Open AI’s gpt-4o model. Patients who received curative intent radiotherapy as initial treatment for PC in our institution between 2014 and 2023 were randomly selected for evaluation. Relevant Clinical factors were extracted by a human expert annotator on our on-site RedCap database. The extraction of the same variables was performed using RadOnc-GPT. A comparison between the ground truth (human annotation) and RadOnc-GPT, to evaluate the performance of the AI system, was carried out including accuracy, recall and F1 score. Results: A total of 382 PC patients were included in the cohort for data extraction. The clinical variables extracted were the treatment intent, NCCN risk category, T and N stage, Gleason score, radiation volume, and Androgen Deprivation therapy (ADT) use. The overall performance of RadOnc-GPT was excellent, with the highest accuracy achieved for treatment intent (99.7%), Gleason score (98.4%), and Treatment Volume (90.1-99.5%). Conclusions: RadOnc-GPT performed excellent at extracting relevant clinical details from the electronic medical record and radiation oncology information systems with dramatic reduction in time and cost compared with manual review. LLMs could be highly useful for large-scale data extraction in PC research, offering a promising solution for automating the accurate data extraction process. Correct (n) Total (n) Accuracy Recall F1 score RT with curative intent 381 382 99.7 99.9 99.9 Prostate Cancer Risk Category 345 382 90.3 90.4 95.1 Primary tumor staging (T) at diagnosis 368 382 96.3 96.3 97.9 Regional nodal status (N) at diagnosis 352 382 92.1 92.2 95.9 Gleason Score Primary Secondary Total 376372376 382382382 98.497.498.4 98.597.498.5 99.298.799.2 Treated Areas Prostate Pelvic Nodes Para-Aortic Nodes Distant Site 380344378372 382382382382 99.590.19997.4 99.490.199.197.4 99.79599.598.7 ADT Use with RT 365 382 95.5 95.7 97.8

424 Background: Advances in prostate cancer (PC) care have led to a 98% 10-year survival rate. Prostate Specific Antigen (PSA) is the primary method of follow-up after definitive radiotherapy (RT), with benign PSA ‘bounce’ occurring in up to 30% of patients, causing uncertainty and anxiety. Distinguishing between PSA bounce and biochemical recurrence (BCR) remains challenging. The Phoenix Criteria is the standard definition of BCR since 2006, defined as PSA nadir + 2 ng/dl, but does not consider patient specific information. Our study aims to explore factors associated with bounce and BCR and develop a predictive tool to differentiate between these two events. Methods: PC patients who received RT at our institution between 2006 and 2022 were identified, including only patients with a PSA increase during follow-up. Exclusion criteria included palliative RT, prostatectomy, or insufficient follow-up information. Patients’ clinical data was collected, and PSA increase was classified as bounce or BCR. Variables were compared using Kruskal Wallis or Chi-square. Multivariate logistic regression was performed in the subset of patients with absolute PSA value >2 (n= 581). Results: Of 1783 patients reviewed due to PSA rise after treatment, 694 met the inclusion criteria. There were 213 (30.7%) with PSA bounce and 481 (69.3%) with recurrence. Prior to meeting the Phoenix Criteria, 20 (4.2%) had biopsy-proven recurrence and 18 (3.7%) imaging-proven recurrence. The median delta PSA was 0.8 ng/mL for bounce vs 5 ng/mL for recurrence (p<0.001). The median time from treatment to event was 14.4 months for bounce vs 45.3 months for BCR (p<0.001). Recurrence was significantly associated (p<0.001) with older age at diagnosis, N1 stage, higher risk category, Gleason score, and higher baseline PSA. In the subgroup of patients without ADT, recurrence was associated with a lower median first PSA post-RT (1.9 vs. 2.4, p=0.046) and a lower median nadir before event (0.6 vs. 1.8, p<0.001). Multivariate logistic regression achieved an accuracy of 0.94, with a sensitivity of 0.83, specificity of 0.98, and an area under the curve (AUC) of 0.97. Conclusions: Our results highlight significant differences in clinical factors between bounce and recurrence in post-RT patients. The multivariate logistic regression differentiated the events with 94% accuracy for patients with a rising PSA value >2 subgroup. We highlight the possible improvement that could be accomplished by personalizing a PSA threshold for recurrence versus the Phoenix definition. This tool offers patients and providers a highly specific tool to differentiate between PSA recurrence and bounce, and may reduce stress and unnecessary testing, or help diagnosis recurrence faster. Next steps include prospective validation of the model to assess the impact on clinical practice costs, patient experience, and cancer outcomes.