Ebner, Daniel

Abstract Large Language Models (LLMs) demonstrate strong capabilities in healthcare applications. However, recruiting human evaluators to assess usability and effectiveness becomes challenging in the large-scale LLM-generated outcomes. Existing metrics are often outdated for evaluating LLM-based healthcare technologies. This paper proposes a Research through Evaluation (RtE) approach that can refine metrics and improve assessments of LLM-generated outputs. We deploy RtE in a retrospective comparative study of prostate cancer patient inquiries, comparing responses from human clinical care teams with those generated by a GPT-4-based in-basket bot. Through three rounds of co-evaluations with clinical professionals and an LLM-as-graders study with four LLMs, the RtE method shows that starting with small, iterative evaluations helps human graders adjust metrics to suit their daily practices better. Additionally, in the LLM-as-graders study, LLMs exhibit similar behavior, preferring the in-basket bot's responses over the clinical care team. Our methodology and findings demonstrate that the RtE method can enhance the transparency and robustness of evaluation metrics for LLM-generated content, with the potential to generalize to evaluations in other clinical practices.

PURPOSE Cancer recurrence is a critical outcome for patients and physicians. Retrospective cancer recurrence data can evaluate recurrence-directed treatment and generate novel interventions targeting recurrent cancers. However, large cancer databases do not provide recurrence-related information, stymying study at scale, and consequently require significant manual record review. Automated evaluation of records may allow for the rapid generation of easily analyzed data sets, accelerating the evaluation of recurrence altogether. METHODS Patients treated with radiation therapy at one tertiary referral center from 2010 to 2018 with a verified cancer status (cancer recurrence v no cancer recurrence) were identified. Patients with recurrent disease were initially identified through manual record review, and the associated pathology report was collected. Google Automated Machine Learning with Natural Language Processing (AutoNLP) and Google Gemini 1.5 Pro were used to generate a model for binary classification, with comparison to the gold-standard manually developed data set. RESULTS A total of 7,054 patients were identified. 3,431 (48.6%) were female, with a median age of 64 years. Head and neck (1,482, 21%), breast (1,480, 21%), upper GI (1,307, 18.5%), and lung/thorax (1,107, 15.7%) were the most common disease sites. Recurrence was verified for 1,546 patients (21.9%) using pathology reports, of which 1,249 positive cases were paired with 651 negative pathology reports for model development. Google Gemini 1.5 Pro consistently outperformed AutoNLP across all measurements of accuracy, generating a greater absolute difference in precision, recall, negative predictive value, and specificity, and a higher likelihood of correct classification at the individual level, rendering Gemini superior in recurrence status extraction. CONCLUSION AutoNLP and Google Gemini 1.5 Pro are promising tools for identifying recurrence from pathology reports, with the latter demonstrating superior overall performance, making it particularly suitable for clinical translation.

Background/Objectives: We performed a systematic review of preclinical literature on the use of high-LET particle therapy, DDRi, and/or immunotherapy specifically in pMMR colorectal cancer. Methods: A systematic review of the literature published between 2014 and 2025 was conducted across major databases. Studies were included if they examined particle radiotherapy (e.g., proton, alpha, and carbon) or X-ray radiation either alone or in combination with DDRi and/or immune checkpoint inhibitors (ICIs) in pMMR colorectal cancer models. Results: In total, 131 studies met the inclusion criteria, including 70 preclinical studies. These studies consistently demonstrate that high-LET radiation amplifies immunogenic cell death, increases cGAS-STING pathway activation, and enhances tumor antigen presentation, thereby fostering greater immune infiltration and systemic antitumor responses. Concurrent irradiation with DDRi enhances persistent DNA damage and cytosolic DNA accumulation. In murine models, high-LET therapies show excellent local control, with manageable toxicity profiles. Combination regimens with ICIs exhibit improved local control and elicit systemic antitumor immune responses. Conclusions: High-LET particle radiation and/or the use of concurrent DDRi with ICI have significant preclinical evidence of immunostimulatory effects in pMMR rectal adenocarcinoma and increased response rates to immunotherapy. The clinical evidence will be reviewed in the companion manuscript.

Background/Objectives: We performed a systematic review of the current clinical and preclinical literature on the use of particle therapy, DDRi, and/or immunotherapy, specifically for pMMR colorectal cancer. Methods: A systematic review of the literature published between 2014 and 2025 was conducted across major databases. Studies were included if they examined particle radiotherapy (e.g., proton, alpha, carbon) either alone or in combination with DDRi and/or immune checkpoint inhibitors (ICI), or X-ray radiotherapy (XRT) in combination with DDRi/ICI. Results: In total, 133 studies met the inclusion criteria, including 62 clinical studies. Clinically, particle therapies show excellent local control and normal tissue sparing, with manageable toxicity profiles. Trials with any form of radiation and DDRi are few, potentially owing to toxicity concerns. ICI combinations showed promising efficacy with XRT, with no randomized trials comparing them to particle radiation. Conclusions: Particle radiation and/or DDRi have significant preclinical evidence of immunostimulatory effects in pMMR rectal adenocarcinoma and increase response rates to immunotherapy (presented in the companion manuscript). Despite strong preclinical evidence and rationale, clinical trials including all three modalities are scarce. Existing evidence suggests a potential benefit based on extrapolation from photon-based studies and supports prospective evaluation with careful attention to treatment-related toxicity, which remains a concern.

Abstract Generative artificial intelligence (AI) is rapidly populating medical records with synthetic or partially AI-generated content, creating a feedback loop where future models are increasingly at risk of training on uncurated AI-generated data. However, the clinical consequences of this AI-generated data contamination remain unexplored. Here, we show that in the absence of mandatory human verification, this self-referential cycle drives a rapid erosion of pathological variability and diagnostic reliability of medical data at population scale. By analysing more than 800,000 synthetic data points across clinical text generation, vision–language reporting, and medical image synthesis, we find that models progressively converge toward generic phenotypes regardless of the model architectures. Specifically, rare but critical findings, including pneumothorax and effusions, vanish from the synthetic content generated by AI models, while demographic representations skew heavily toward middle-aged male phenotypes. Crucially, this degradation is masked by false diagnostic confidence. Models continue to issue reassuring reports while failing to detect life-threatening pathology, with false reassurance rates tripling to 40%. Blinded physician evaluation confirms that this decoupling of confidence and accuracy renders AI-generated documentation clinically useless after just two generations. We systematically evaluate three mitigation strategies that can be easily integrated into existing clinical workflows, finding that while synthetic volume scaling fails to prevent collapse, mixing real data with quality-aware filtering effectively preserves diversity. Ultimately, our results suggest that without policy-mandated human oversight, the deployment of generative AI threatens to degrade the very healthcare data ecosystems it relies upon.

Abstract Generative artificial intelligence (AI) is rapidly populating medical records with synthetic or partially AI-generated content, creating a feedback loop where future models are increasingly at risk of training on uncurated AI-generated data. However, the clinical consequences of this AI-generated data contamination remain unexplored. Here, we show that in the absence of mandatory human verification, this self-referential cycle drives a rapid erosion of pathological variability and diagnostic reliability of medical data at population scale. By analysing more than 800,000 synthetic data points across clinical text generation, vision–language reporting, and medical image synthesis, we find that models progressively converge toward generic phenotypes regardless of the model architectures. Specifically, rare but critical findings, including pneumothorax and effusions, vanish from the synthetic content generated by AI models, while demographic representations skew heavily toward middle-aged male phenotypes. Crucially, this degradation is masked by false diagnostic confidence. Models continue to issue reassuring reports while failing to detect life-threatening pathology, with false reassurance rates tripling to 40%. Blinded physician evaluation confirms that this decoupling of confidence and accuracy renders AI-generated documentation clinically useless after just two generations. We systematically evaluate three mitigation strategies that can be easily integrated into existing clinical workflows, finding that while synthetic volume scaling fails to prevent collapse, mixing real data with quality-aware filtering effectively preserves diversity. Ultimately, our results suggest that without policy-mandated human oversight, the deployment of generative AI threatens to degrade the very healthcare data ecosystems it relies upon.