Citations

Abstract Large Language Models (LLMs) demonstrate strong capabilities in healthcare applications. However, recruiting human evaluators to assess usability and effectiveness becomes challenging in the large-scale LLM-generated outcomes. Existing metrics are often outdated for evaluating LLM-based healthcare technologies. This paper proposes a Research through Evaluation (RtE) approach that can refine metrics and improve assessments of LLM-generated outputs. We deploy RtE in a retrospective comparative study of prostate cancer patient inquiries, comparing responses from human clinical care teams with those generated by a GPT-4-based in-basket bot. Through three rounds of co-evaluations with clinical professionals and an LLM-as-graders study with four LLMs, the RtE method shows that starting with small, iterative evaluations helps human graders adjust metrics to suit their daily practices better. Additionally, in the LLM-as-graders study, LLMs exhibit similar behavior, preferring the in-basket bot's responses over the clinical care team. Our methodology and findings demonstrate that the RtE method can enhance the transparency and robustness of evaluation metrics for LLM-generated content, with the potential to generalize to evaluations in other clinical practices.

PURPOSE Cancer recurrence is a critical outcome for patients and physicians. Retrospective cancer recurrence data can evaluate recurrence-directed treatment and generate novel interventions targeting recurrent cancers. However, large cancer databases do not provide recurrence-related information, stymying study at scale, and consequently require significant manual record review. Automated evaluation of records may allow for the rapid generation of easily analyzed data sets, accelerating the evaluation of recurrence altogether. METHODS Patients treated with radiation therapy at one tertiary referral center from 2010 to 2018 with a verified cancer status (cancer recurrence v no cancer recurrence) were identified. Patients with recurrent disease were initially identified through manual record review, and the associated pathology report was collected. Google Automated Machine Learning with Natural Language Processing (AutoNLP) and Google Gemini 1.5 Pro were used to generate a model for binary classification, with comparison to the gold-standard manually developed data set. RESULTS A total of 7,054 patients were identified. 3,431 (48.6%) were female, with a median age of 64 years. Head and neck (1,482, 21%), breast (1,480, 21%), upper GI (1,307, 18.5%), and lung/thorax (1,107, 15.7%) were the most common disease sites. Recurrence was verified for 1,546 patients (21.9%) using pathology reports, of which 1,249 positive cases were paired with 651 negative pathology reports for model development. Google Gemini 1.5 Pro consistently outperformed AutoNLP across all measurements of accuracy, generating a greater absolute difference in precision, recall, negative predictive value, and specificity, and a higher likelihood of correct classification at the individual level, rendering Gemini superior in recurrence status extraction. CONCLUSION AutoNLP and Google Gemini 1.5 Pro are promising tools for identifying recurrence from pathology reports, with the latter demonstrating superior overall performance, making it particularly suitable for clinical translation.

Background/Objectives: We performed a systematic review of preclinical literature on the use of high-LET particle therapy, DDRi, and/or immunotherapy specifically in pMMR colorectal cancer. Methods: A systematic review of the literature published between 2014 and 2025 was conducted across major databases. Studies were included if they examined particle radiotherapy (e.g., proton, alpha, and carbon) or X-ray radiation either alone or in combination with DDRi and/or immune checkpoint inhibitors (ICIs) in pMMR colorectal cancer models. Results: In total, 131 studies met the inclusion criteria, including 70 preclinical studies. These studies consistently demonstrate that high-LET radiation amplifies immunogenic cell death, increases cGAS-STING pathway activation, and enhances tumor antigen presentation, thereby fostering greater immune infiltration and systemic antitumor responses. Concurrent irradiation with DDRi enhances persistent DNA damage and cytosolic DNA accumulation. In murine models, high-LET therapies show excellent local control, with manageable toxicity profiles. Combination regimens with ICIs exhibit improved local control and elicit systemic antitumor immune responses. Conclusions: High-LET particle radiation and/or the use of concurrent DDRi with ICI have significant preclinical evidence of immunostimulatory effects in pMMR rectal adenocarcinoma and increased response rates to immunotherapy. The clinical evidence will be reviewed in the companion manuscript.

Background/Objectives: We performed a systematic review of the current clinical and preclinical literature on the use of particle therapy, DDRi, and/or immunotherapy, specifically for pMMR colorectal cancer. Methods: A systematic review of the literature published between 2014 and 2025 was conducted across major databases. Studies were included if they examined particle radiotherapy (e.g., proton, alpha, carbon) either alone or in combination with DDRi and/or immune checkpoint inhibitors (ICI), or X-ray radiotherapy (XRT) in combination with DDRi/ICI. Results: In total, 133 studies met the inclusion criteria, including 62 clinical studies. Clinically, particle therapies show excellent local control and normal tissue sparing, with manageable toxicity profiles. Trials with any form of radiation and DDRi are few, potentially owing to toxicity concerns. ICI combinations showed promising efficacy with XRT, with no randomized trials comparing them to particle radiation. Conclusions: Particle radiation and/or DDRi have significant preclinical evidence of immunostimulatory effects in pMMR rectal adenocarcinoma and increase response rates to immunotherapy (presented in the companion manuscript). Despite strong preclinical evidence and rationale, clinical trials including all three modalities are scarce. Existing evidence suggests a potential benefit based on extrapolation from photon-based studies and supports prospective evaluation with careful attention to treatment-related toxicity, which remains a concern.

Abstract Generative artificial intelligence (AI) is rapidly populating medical records with synthetic or partially AI-generated content, creating a feedback loop where future models are increasingly at risk of training on uncurated AI-generated data. However, the clinical consequences of this AI-generated data contamination remain unexplored. Here, we show that in the absence of mandatory human verification, this self-referential cycle drives a rapid erosion of pathological variability and diagnostic reliability of medical data at population scale. By analysing more than 800,000 synthetic data points across clinical text generation, vision–language reporting, and medical image synthesis, we find that models progressively converge toward generic phenotypes regardless of the model architectures. Specifically, rare but critical findings, including pneumothorax and effusions, vanish from the synthetic content generated by AI models, while demographic representations skew heavily toward middle-aged male phenotypes. Crucially, this degradation is masked by false diagnostic confidence. Models continue to issue reassuring reports while failing to detect life-threatening pathology, with false reassurance rates tripling to 40%. Blinded physician evaluation confirms that this decoupling of confidence and accuracy renders AI-generated documentation clinically useless after just two generations. We systematically evaluate three mitigation strategies that can be easily integrated into existing clinical workflows, finding that while synthetic volume scaling fails to prevent collapse, mixing real data with quality-aware filtering effectively preserves diversity. Ultimately, our results suggest that without policy-mandated human oversight, the deployment of generative AI threatens to degrade the very healthcare data ecosystems it relies upon.

Abstract Generative artificial intelligence (AI) is rapidly populating medical records with synthetic or partially AI-generated content, creating a feedback loop where future models are increasingly at risk of training on uncurated AI-generated data. However, the clinical consequences of this AI-generated data contamination remain unexplored. Here, we show that in the absence of mandatory human verification, this self-referential cycle drives a rapid erosion of pathological variability and diagnostic reliability of medical data at population scale. By analysing more than 800,000 synthetic data points across clinical text generation, vision–language reporting, and medical image synthesis, we find that models progressively converge toward generic phenotypes regardless of the model architectures. Specifically, rare but critical findings, including pneumothorax and effusions, vanish from the synthetic content generated by AI models, while demographic representations skew heavily toward middle-aged male phenotypes. Crucially, this degradation is masked by false diagnostic confidence. Models continue to issue reassuring reports while failing to detect life-threatening pathology, with false reassurance rates tripling to 40%. Blinded physician evaluation confirms that this decoupling of confidence and accuracy renders AI-generated documentation clinically useless after just two generations. We systematically evaluate three mitigation strategies that can be easily integrated into existing clinical workflows, finding that while synthetic volume scaling fails to prevent collapse, mixing real data with quality-aware filtering effectively preserves diversity. Ultimately, our results suggest that without policy-mandated human oversight, the deployment of generative AI threatens to degrade the very healthcare data ecosystems it relies upon.

Abstract Poly ADP ribose polymerase inhibitors (PARPi) are being investigated in combination with photon and proton radiotherapy in clinical trials of pediatric and adult malignancies. The precise mechanisms of PARPi radiosensitization, and whether differences in photon and proton-mediated relative efficacy exist, have not been fully elucidated. To study mechanisms of PARPi radiosensitization we investigated the effects of PARPi in combination with photons and two different ionizing densities of protons delivered at the entrance (ENT) or the Bragg peak (BP) of the profile (dose-averaged LET [LETd] 2.2 and 7.0 keV/µm, respectively) of a 76.8 MeV proton beam after observing an extreme tumor and normal tissue response in an 18-year-old female patient with high grade glioma treated veliparib and proton therapy in a Children’s Oncology Group clinical trial. BRCA1/2 wild-type hTERT RPE-1, U2OS, and MDA-MB-231 cells were hypersensitive to the combination of veliparib and proton BP, with enhanced relative biological effectiveness (RBE) at 2 Gy values of 1.7, 1.4, and 1.3, respectively. Similar findings were observed with talazoparib and olaparib, suggesting that PARP trapping efficiency was not a major influence on the proton BP RBE. Further, we did not observe differences in transcription replication conflict between the different proton ionizing densities. Unexpectedly, DNA fiber assays revealed more elongated replication fork tracks in PARPi treated cells 60 minutes after proton BP compared with proton ENT, indicative of accelerated replication fork progression. To assess gap formation in the vicinity of nascent DNA we treated cells with the ssDNA-specific S1 nuclease and found that the lengths of CldU-labeled tracks were significantly reduced in cells treated with the PARPi/proton BP combination, suggestive of greater gap formation at replication forks. We postulated that more ssDNA gaps were being converted to DSBs during replication as we further observed that PARPi/proton BP resulted in increased γH2AX foci at 4- and 24-hours after treatment. Consistent with increased DSB formation, a higher proportion of cells arrested in the G2/M phase of the cell cycle at 24 hours following PARPi/proton BP. Despite being arrested in G2/M when homologous recombination is active, there was a significant increase in 53BP1 and a decrease in RAD51 recruitment, indicative of upregulated repair by more error-prone non-homologous end-joining (NHEJ) repair pathway. The enhanced proton BP RBE in PARPi treated cells was abrogated upon knockdown of XRCC4, a critical component of NHEJ repair pathway, suggesting that toxic NHEJ may be contributing to the enhanced proton BP RBE. In conclusion, PARPi renders cells hypersensitive to high LET BP proton irradiation. Enhancing the LET in combination with PARPi warrants further investigation to overcome therapeutic resistance. Citation Format: Yiqun Han, Qin Zhou, Anna Olson, Cameron M. Callaghan, Bin Chen, Nicholas B. Remmes, Meng Xu Welliver, Zheming Wu, Daniel K. Ebner, Jake A. Kloeber, Zhenkun Lou, Robert W. Mutter. Poly ADP ribose polymerase inhibitors potentiate proton therapy end-of-range relative biological effective dose by increasing replication stress [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2025; Part 1 (Regular Abstracts); 2025 Apr 25-30; Chicago, IL. Philadelphia (PA): AACR; Cancer Res 2025;85(8_Suppl_1):Abstract nr 4689.

PURPOSE Survival for patients with melanoma has recently improved. The propensity of melanoma to metastasize to the brain remains a common and serious feature of this disease. The purposes of this study were to evaluate prognostic factors for patients with newly diagnosed melanoma brain metastases (MBMs) in a large cohort treated with modern multimodal therapies, compare those results with those in prior eras, and update the Melanoma Graded Prognostic Assessment (GPA). METHODS Univariable and multivariable (MVA) analyses of prognostic factors and treatments associated with survival were performed on 1,796 patients with newly diagnosed MBM treated between January 01, 2015, and December 31, 2021, using a multi-institutional retrospective database. Multiple imputation was used to address missingness of potential predictors. Significant variables in combined MVA were used to update the Melanoma GPA. Comparisons were made with legacy cohorts. RESULTS Median survivals for cohorts A (1985-2007, n = 481), B (2006-2015, n = 823), and C (2015-2021, n = 1,796) were 6.7, 9.8, and 16.6 months and median follow-up times were 40.1, 43.6, and 48.8 months, respectively. In combined MVA, significant prognostic factors for survival were higher Karnofsky Performance Status, fewer MBMs, absence of extracranial metastases, lower serum lactate dehydrogenase, and no immunotherapy before MBM. These factors were incorporated into the updated Melanoma GPA. The combined median and 3-year survivals for patients with GPA 0-1, 1.5-2, and 2.5-4.0 were 5.4, 13.2, and 43.2 months and 12.4%, 28.8%, and 51.6%, respectively. CONCLUSION Prognostic factors have changed and survival has improved for patients with MBM but varies widely by GPA. The updated Melanoma GPA calculator (BrainMetGPA), available free online, can be used to estimate survival, individualize treatment, stratify clinical trials, guide surveillance, and augment clinical trial eligibility. Multidisciplinary treatment is essential. Trials are needed to elucidate the optimal sequencing of various therapeutic modalities.

Objectives/Goals: Our study’s objective is to evaluate RadOnc-GPT, a GPT-4o powered LLM, in generating responses to in-basket messages related to prostate cancer treatment in the Radiation Oncology department. By integrating it with electronic health record (EHR) systems, the goal is to assess its impact on clinician workload, response quality, and efficiency in healthcare communication. Methods/Study Population: RadOnc-GPT was integrated with patient EHRs from both hospital-wide and radiation-oncology-specific databases. The study examined 158 pre-recorded in-basket message interactions from 90 non-metastatic prostate cancer patients. Quantitative natural language processing analysis and two randomized single-blinded grading studies, involving four clinicians and four nurses, were conducted to evaluate RadOnc-GPT’s response quality in completeness, correctness, clarity, empathy, and estimated editing time. Response times were measured to estimate the time saved for clinicians and nurses. The study population included patient messages across all phases of care (pre-, during, and post-treatment) for those undergoing radiotherapy. Results/Anticipated Results: In the single-blinded grader study, clinician graders evaluated 316 responses (158 from human care teams and 158 from RadOnc-GPT). Results showed RadOnc-GPT outperformed human responses in empathy and clarity, while humans excelled in completeness and correctness. Sentiment analyses using TextBlob and VADER revealed RadOnc-GPT responses had a positive mean score of 0.25, whereas human responses clustered around neutral. VADER analysis indicated a high median score for RadOnc-GPT, nearing 1.0, reflecting predominantly positive sentiment, while human responses displayed a broader sentiment range, indicating sensitivity to context. Clinicians averaged 3.60 minutes (SD 1.44) to respond, compared to 6.39 minutes (SD 4.05) for nurses, highlighting RadOnc-GPT’s efficiency in generating timely responses. Discussion/Significance of Impact: RadOnc-GPT effectively generated responses to individualized patient in-basket messages, comparable to those from radiation oncologists and nurses. While human oversight is still necessary to avoid errors, RadOnc-GPT can speed up response times and reduce pressure on care teams, shifting their role from drafting to reviewing responses.

425 Background: Prostate cancer (PC) ranks among the most prevalent cancers in men, with incidence rising due to factors like aging populations and enhanced screening techniques. Effective data extraction from clinical records is crucial for advancing research and improving patient care; however, traditional manual methods can be time-consuming and susceptible to mistakes. This study seeks to evaluate RadOnc-GPT against human annotation for extracting essential clinical information from prostate cancer clinical records. Methods: RadOnc-GPT is a retrieval augmented generation (RAG)-based chatbot with access to a variety of patient data including treatment details, radiology reports, and clinical notes. RadOnc-GPT uses Open AI’s gpt-4o model. Patients who received curative intent radiotherapy as initial treatment for PC in our institution between 2014 and 2023 were randomly selected for evaluation. Relevant Clinical factors were extracted by a human expert annotator on our on-site RedCap database. The extraction of the same variables was performed using RadOnc-GPT. A comparison between the ground truth (human annotation) and RadOnc-GPT, to evaluate the performance of the AI system, was carried out including accuracy, recall and F1 score. Results: A total of 382 PC patients were included in the cohort for data extraction. The clinical variables extracted were the treatment intent, NCCN risk category, T and N stage, Gleason score, radiation volume, and Androgen Deprivation therapy (ADT) use. The overall performance of RadOnc-GPT was excellent, with the highest accuracy achieved for treatment intent (99.7%), Gleason score (98.4%), and Treatment Volume (90.1-99.5%). Conclusions: RadOnc-GPT performed excellent at extracting relevant clinical details from the electronic medical record and radiation oncology information systems with dramatic reduction in time and cost compared with manual review. LLMs could be highly useful for large-scale data extraction in PC research, offering a promising solution for automating the accurate data extraction process. Correct (n) Total (n) Accuracy Recall F1 score RT with curative intent 381 382 99.7 99.9 99.9 Prostate Cancer Risk Category 345 382 90.3 90.4 95.1 Primary tumor staging (T) at diagnosis 368 382 96.3 96.3 97.9 Regional nodal status (N) at diagnosis 352 382 92.1 92.2 95.9 Gleason Score Primary Secondary Total 376372376 382382382 98.497.498.4 98.597.498.5 99.298.799.2 Treated Areas Prostate Pelvic Nodes Para-Aortic Nodes Distant Site 380344378372 382382382382 99.590.19997.4 99.490.199.197.4 99.79599.598.7 ADT Use with RT 365 382 95.5 95.7 97.8