Citations

1615 Background: Immune checkpoint inhibitors (ICIs) have revolutionized lung cancer treatment but cause adverse events, including immune-related colitis (ir-colitis). Accurate identification of ir-colitis is critical for improving patient outcomes and clinical research. However, current methods relying on ICD codes have substantial limitations: non-specific categories that fail to capture the cases or distinguish cases from infectious or ischemic etiologies, as well as incomplete coding that leads to missed cases. Large language models (LLMs), with advanced natural language understanding capabilities, offer a scalable approach for automated phenotyping using unstructured clinical notes. We aimed to develop and evaluate an LLM-based approach for identifying ir-colitis in patients with lung cancer receiving ICIs and compare it with ICD code based method. Methods: We identified 5,278 lung cancer patients with ICI administration at Mayo Clinic (2012-2025). To investigate ir-colitis occurrence within two years after ICI initiation, patients were stratified into three ICD-based cohorts: (1) IR-Colitis, with clinician-assigned ir-colitis-specific diagnosis codes; (2) Non-IR Colitis, with colitis diagnosis codes excluding ir-colitis; and (3) No-Colitis, absent of any colitis-related codes. We used three open-source LLMs to process clinical notes : Llama4, MedGemma, and Qwen. Prompts were designed to extract both gold standard evidence (explicit ir-colitis terminology, such as "immune-mediated colitis" and "checkpoint inhibitor colitis") and supportive evidence (such as pathology and lab results). Results: Llama4 demonstrated superior accuracy among three LLMs in manual evaluation and was thus selected for all cohort analyses. Among ICD-based IR-Colitis patients (n = 339), 82.0% had confirmatory evidence while 18% lacked evidence, suggesting potential false-positive coding. In the Non-IR Colitis cohort (n = 297), LLM-based extraction identified 43.8% patients in fact had ir-colitis, demonstrating substantial misclassification. In the No-colitis cohort (n = 4,642, random sample of 996 for analysis), the LLM identified 10.7% with documented ir-colitis, revealing substantial missing by administrative coding. Conclusions: LLM-based phenotyping demonstrates superior sensitivity for identifying ir-colitis compared to ICD codes alone, successfully uncovering cases missed by administrative coding. This scalable approach enables more comprehensive case identification for immunotherapy safety research and clinical surveillance. ICD-Based Cohorts LLM-Identified ir-Colitis Gold Standard Evidence Supportive Evidence Any Evidence ir-Colitis (n = 339) 277/339 (81.7%) 275/339 (81.1%) 278/339 (82.0%) Non-ir-Colitis (n = 297) 129/297 (43.4%) 126/297 (42.4%) 130/297 (43.8%) No Colitis (n = 996) 106/996 (10.6%) 104/996 (10.4%) 107/996 (10.7%)

Abstract Objectives Precise literature recommendation and summarization are crucial for biomedical professionals. While the latest iteration of generative pretrained transformer (GPT) incorporates 2 distinct modes—real-time search and pretrained model utilization—it encounters challenges in dealing with these tasks. Specifically, the real-time search can pinpoint some relevant articles but occasionally provides fabricated papers, whereas the pretrained model excels in generating well-structured summaries but struggles to cite specific sources. In response, this study introduces RefAI, an innovative retrieval-augmented generative tool designed to synergize the strengths of large language models (LLMs) while overcoming their limitations. Materials and Methods RefAI utilized PubMed for systematic literature retrieval, employed a novel multivariable algorithm for article recommendation, and leveraged GPT-4 turbo for summarization. Ten queries under 2 prevalent topics (“cancer immunotherapy and target therapy” and “LLMs in medicine”) were chosen as use cases and 3 established counterparts (ChatGPT-4, ScholarAI, and Gemini) as our baselines. The evaluation was conducted by 10 domain experts through standard statistical analyses for performance comparison. Results The overall performance of RefAI surpassed that of the baselines across 5 evaluated dimensions—relevance and quality for literature recommendation, accuracy, comprehensiveness, and reference integration for summarization, with the majority exhibiting statistically significant improvements (P-values <.05). Discussion RefAI demonstrated substantial improvements in literature recommendation and summarization over existing tools, addressing issues like fabricated papers, metadata inaccuracies, restricted recommendations, and poor reference integration. Conclusion By augmenting LLM with external resources and a novel ranking algorithm, RefAI is uniquely capable of recommending high-quality literature and generating well-structured summaries, holding the potential to meet the critical needs of biomedical professionals in navigating and synthesizing vast amounts of scientific literature.

Background The rapid evolution of artificial intelligence (AI) in conjunction with recent updates in dual antiplatelet therapy (DAPT) management guidelines emphasizes the necessity for innovative models to predict ischemic or bleeding events after drug‐eluting stent implantation. Leveraging AI for dynamic prediction has the potential to revolutionize risk stratification and provide personalized decision support for DAPT management. Methods and Results We developed and validated a new AI‐based pipeline using retrospective data of drug‐eluting stent‐treated patients, sourced from the Cerner Health Facts data set (n=98 236) and Optum's de‐identified Clinformatics Data Mart Database (n=9978). The 36 months following drug‐eluting stent implantation were designated as our primary forecasting interval, further segmented into 6 sequential prediction windows. We evaluated 5 distinct AI algorithms for their precision in predicting ischemic and bleeding risks. Model discriminative accuracy was assessed using the area under the receiver operating characteristic curve, among other metrics. The weighted light gradient boosting machine stood out as the preeminent model, thus earning its place as our AI‐DAPT model. The AI‐DAPT demonstrated peak accuracy in the 30 to 36 months window, charting an area under the receiver operating characteristic curve of 90% [95% CI, 88%–92%] for ischemia and 84% [95% CI, 82%–87%] for bleeding predictions. Conclusions Our AI‐DAPT excels in formulating iterative, refined dynamic predictions by assimilating ongoing updates from patients' clinical profiles, holding value as a novel smart clinical tool to facilitate optimal DAPT duration management with high accuracy and adaptability.