Jia, Heling

ABSTRACT Background Mobility functional status is inconsistently recorded in electronic health records. Recent advances in Large Language Models (LLMs) enable automated extraction of functional information from unstructured clinical notes. Leveraging a validated functional LLM with high performance in mobility extraction, we aimed to study the association between LLM‐derived mobility functional status and clinical outcomes, including emergency department (ED) visits, hospitalizations, and mortality. Methods Individuals aged 60 years and older enrolled in the Mayo Clinic Study of Aging, with documented clinical notes in 2022 were included. Individuals without any identifiable functional information were excluded. The LLM‐derived mobility functional status categorized mobility as impaired or unimpaired, across the five mobility domains as classified by International Classification of Functioning, Disability, and Health. Clinical outcomes in 2023 were extracted from the Rochester Epidemiology Project database. Logistic regression models assessed associations between mobility impairment and outcomes, adjusted for age, sex, Elixhauser comorbidity index, and education. Results Three thousand three hundred thirty‐seven individuals were screened, of whom three hundred forty‐three had no functional information. Two thousand nine hundred ninety‐four individuals were analyzed. 79.4% had at least one mobility impairment. Higher mobility impairment is associated with older age and greater Elixhauser comorbidity. Individuals with 3 or more impairments had an increased adjusted odds of ED visits (Adjusted Odds Ratio (aOR): 2.18 (1.64, 2.90)) and hospitalizations (aOR: 2.35 (1.51, 3.65)) compared to unimpaired individuals ( p < 0.001), while mortality did not demonstrate a significant association with mobility impairment. Conclusion LLM‐derived mobility functional status assessment was associated with ED visits and hospitalization, but not with all‐cause mortality after adjustment. Prospective validation and evaluation of incremental discrimination, calibration, and clinical utility are needed before clinical implementation.

Abstract Objective This study evaluated sequential multi-site fine-tuning of large language models (LLMs), simulating incremental deployment across institutions for extracting mobility functional status from unstructured clinical notes. Materials and Methods We assembled a corpus of 600 clinical notes from 3 institutions, yielding 3810 annotated note sections (1200 training; 2610 testing). Building upon our previously established LLM mobility extraction pipeline, which utilized optimized prompt engineering, we applied parameter-efficient low-rank adaptation (LoRA) fine-tuning. We evaluated sequential multi-site fine-tuning, where the model is trained locally and only the updated LoRA weights are transferred to the next site. We compared this against joint fine-tuning, which requires pooling all multi-site patient data centrally before training. Performance was measured by micro F1-scores for 2 distinct subtasks: mobility extraction and impairment classification. Results Sequential multi-site fine-tuning yielded comparable results to centralized joint fine-tuning for mobility extraction (F1, 0.861 vs 0.894) and impairment classification (F1, 0.912 vs 0.900). Both sequential and joint fine-tuning outperformed an untuned 70B model. Additionally, single-site LoRA models showed robust cross-site generalization. Discussion Sequential multi-site fine-tuning offers a resource-friendly approach to adapt LLMs to clinical environments. By retaining previously acquired knowledge without severe interference, this strategy allows for local fine-tuning at each institution without the need to pool or share sensitive patient data required for joint fine-tuning. Conclusion Sequential multi-site fine-tuning provides a viable solution for the scalable, incremental deployment of LLMs across institutions, showing promising generalizability for clinical information extraction from electronic health record narratives.

Abstract Delirium is a state of acute confusion characterized by deficits in attention that occurs commonly in hospitalized older adults and is associated with adverse outcomes. Delirium may have significant implications for cognitive health, however the relationship of delirium to mild cognitive impairment (MCI) and dementia is not fully understood. To evaluate the impact of delirium on cognitive health trajectories, we leveraged the Mayo Clinic Study of Aging (MCSA), a population-based cohort study of cognitive health and aging in residents of Olmsted County, Minnesota, who are between the ages of 30 and 89 and without a previous diagnosis of dementia at enrollment. Emergency Department and hospital encounters were identified for all patients enrolled in the MCSA, and clinical notes following each encounter up to 30 days were analyzed using a previously validated natural language processing (NLP) algorithm for delirium detection. The association between delirium within 30 days of hospitalization and the development of MCI or dementia was assessed using Cox proportional hazards regression, adjusted for patient age, sex, and level of education. Initial MCSA assessment indicated normal cognition for all individuals in this cohort. We found that delirium was significantly associated with development of MCI and dementia, with 42 individuals experiencing delirium within 30 days of hospitalization; hazard ratio of 10.07 (95% CI 4.14, 24.49) for developing MCI and 19.10 (95% CI 8.40, 43.47) for dementia compared to those without delirium. Our results suggest that hospitalization with an episode of delirium can significantly impact cognitive trajectories in older adults.

Abstract Background Delirium is often underdiagnosed in clinical practice and is not routinely coded for billing. While manual chart review can identify delirium, it is labor-intensive and impractical for large-scale studies. Natural language processing (NLP) can analyze unstructured text in electronic health records (EHRs) to extract meaningful clinical information. Methods To support national integration of NLP for EHR-based delirium identification across different institutions, we launched the Delirium Interest Group within the national Evolve to Next-Gen Accrual to Clinical Trials (ENACT) NLP Working Group. This paper outlines our initial efforts to standardize, evaluate, and translate an NLP-based delirium detection model into the i2b2/ENACT platform. Results Multisite contextual inquiry identified several key challenges, including variations in local screening practices (eg, tools used, documentation frequency, and quality control), the need for harmonized definitions in the context of EHRs, and the complexity of modeling temporal logic. Multisite NLP evaluation revealed variable performance degradation driven by differences in delirium screening practices, clinical documentation patterns and semantics, and note syntactic structures. Conclusion Our work represents an important first step toward enabling scalable and standardized NLP-based delirium detection across institutions. By engaging diverse institutions through the ENACT NLP Working Group, we identified shared challenges and site-specific variations that impact model implementation and performance. Our collaborative approach enabled the development of a more robust framework for delirium identification across heterogeneous EHR systems. Future efforts will build on this foundation to enhance the validity, usability, and translational impact of delirium detection.