Jeon, Eunji

ABSTRACT Background Mobility functional status is inconsistently recorded in electronic health records. Recent advances in Large Language Models (LLMs) enable automated extraction of functional information from unstructured clinical notes. Leveraging a validated functional LLM with high performance in mobility extraction, we aimed to study the association between LLM‐derived mobility functional status and clinical outcomes, including emergency department (ED) visits, hospitalizations, and mortality. Methods Individuals aged 60 years and older enrolled in the Mayo Clinic Study of Aging, with documented clinical notes in 2022 were included. Individuals without any identifiable functional information were excluded. The LLM‐derived mobility functional status categorized mobility as impaired or unimpaired, across the five mobility domains as classified by International Classification of Functioning, Disability, and Health. Clinical outcomes in 2023 were extracted from the Rochester Epidemiology Project database. Logistic regression models assessed associations between mobility impairment and outcomes, adjusted for age, sex, Elixhauser comorbidity index, and education. Results Three thousand three hundred thirty‐seven individuals were screened, of whom three hundred forty‐three had no functional information. Two thousand nine hundred ninety‐four individuals were analyzed. 79.4% had at least one mobility impairment. Higher mobility impairment is associated with older age and greater Elixhauser comorbidity. Individuals with 3 or more impairments had an increased adjusted odds of ED visits (Adjusted Odds Ratio (aOR): 2.18 (1.64, 2.90)) and hospitalizations (aOR: 2.35 (1.51, 3.65)) compared to unimpaired individuals ( p < 0.001), while mortality did not demonstrate a significant association with mobility impairment. Conclusion LLM‐derived mobility functional status assessment was associated with ED visits and hospitalization, but not with all‐cause mortality after adjustment. Prospective validation and evaluation of incremental discrimination, calibration, and clinical utility are needed before clinical implementation.

Abstract Objective This study evaluated sequential multi-site fine-tuning of large language models (LLMs), simulating incremental deployment across institutions for extracting mobility functional status from unstructured clinical notes. Materials and Methods We assembled a corpus of 600 clinical notes from 3 institutions, yielding 3810 annotated note sections (1200 training; 2610 testing). Building upon our previously established LLM mobility extraction pipeline, which utilized optimized prompt engineering, we applied parameter-efficient low-rank adaptation (LoRA) fine-tuning. We evaluated sequential multi-site fine-tuning, where the model is trained locally and only the updated LoRA weights are transferred to the next site. We compared this against joint fine-tuning, which requires pooling all multi-site patient data centrally before training. Performance was measured by micro F1-scores for 2 distinct subtasks: mobility extraction and impairment classification. Results Sequential multi-site fine-tuning yielded comparable results to centralized joint fine-tuning for mobility extraction (F1, 0.861 vs 0.894) and impairment classification (F1, 0.912 vs 0.900). Both sequential and joint fine-tuning outperformed an untuned 70B model. Additionally, single-site LoRA models showed robust cross-site generalization. Discussion Sequential multi-site fine-tuning offers a resource-friendly approach to adapt LLMs to clinical environments. By retaining previously acquired knowledge without severe interference, this strategy allows for local fine-tuning at each institution without the need to pool or share sensitive patient data required for joint fine-tuning. Conclusion Sequential multi-site fine-tuning provides a viable solution for the scalable, incremental deployment of LLMs across institutions, showing promising generalizability for clinical information extraction from electronic health record narratives.

Abstract Clinical narrative text contains crucial patient information, yet reliable extraction remains challenging due to linguistic variability, documentation habits, and differences across care settings. Large language models (LLMs) have shown strong accuracy on clinical information extraction (IE), but their reproducibility (stability under repeated runs) and robustness (stability under small, natural prompt variations) are less consistently quantified, despite being central to clinical deployment. In this study, we evaluate three open-weight LLMs representing distinct modeling choices: a dense general-purpose model (Llama 3.3), a mixture-of-experts (MoE) general-purpose model (Llama 4), and a domain-tuned medical model (MedGemma). We focus on binary clinical IE aligned with four mobility classes from the International Classification of Functioning, Disability and Health (ICF) framework. Using a controlled experimental design, we quantify (1) intra-prompt reproducibility across repeated sampling and (2) inter-prompt robustness across paraphrased prompts. We jointly report predictive performance (F1-score) and stability (Fleiss’ Kappa). And we test factor effects using three-way ANOVA with post-hoc comparisons. Results show that increasing temperature generally degrades agreement, but the magnitude depends on model and task; furthermore, prompt paraphrasing can substantially reduce stability, with particularly large drops for the MoE model. Finally, we evaluate a practical mitigation, self-consistency via majority voting, which improves κ substantially and often improves or preserves F1-score, at the cost of additional inference. Together, these findings provide a reproducible framework and concrete recommendations for evaluating and improving LLM reliability in clinical IE.

Low self-esteem (LoST) is a latent yet critical psychosocial risk factor that predisposes individuals to depressive disorders. Although structured tools exist to assess self-esteem, their limited clinical adoption suggests that relevant indicators of LoST remain buried within unstructured clinical narratives. The scarcity of annotated clinical notes impedes the development of natural language processing (NLP) models for its detection. Manual chart reviews are labor-intensive and large language model (LLM)-driven (weak) labeling raises privacy concerns. Past studies demonstrate that NLP models trained on LLM-generated synthetic clinical notes achieve performance comparable to, and sometimes better than those trained on real notes. This highlights synthetic data’s utility for augmenting scarce clinical corpora while reducing privacy concerns. Prior efforts have leveraged social media data, such as Reddit, to identify linguistic markers of low self-esteem; however, the linguistic and contextual divergence between social media and clinical text limits the generalizability of these models. To address this gap, we present a novel framework that generates context-enhanced synthetic clinical notes from social media narratives and evaluates the utility of small language models for identifying expressions of low self-esteem. Our approach includes a mixed-method evaluation framework: (i) structure analysis, (ii) readability analysis, (iii) linguistic diversity, and (iv) contextual fidelity of LoST cues in source Reddit posts and synthetic notes. This work offers a scalable, privacy-preserving solution for synthetic data generation for early detection of psychosocial risks such as LoST and demonstrates a pathway for translating mental health signals in clinical notes into clinically actionable insights, thereby identifying patients at risk.