Rianon, Nahid

Abstract Error analysis is a critical step in evaluating and improving clinical concept extraction models, the most common clinical natural language processing (NLP) task. Unlike corpus annotation, which follows standardized protocols for creating gold-standard datasets, error analysis requires nuanced judgment grounded in both clinical expertise and NLP knowledge. This is especially important given the heterogeneity of clinical text, where variations in documentation style, note structure, and terminology can substantially influence model behavior. Despite its importance, there is currently no standardized, user-level framework to support systematic error analysis in clinical concept extraction task. In this study, we developed and validated MedError, a machine-assisted, human-in-the-loop framework designed to standardize and enhance error analysis for clinical concept extraction tasks. We collected and manually curated a corpus of 1,187 unique errors from a total of 4,237 notes across three different distinct hospitals. The error categories were defined using our previously validated error taxonomy and included 480 false negatives and 707 false positives across 25 error types and 48 clinical concept categories. We evaluated the performance of three proprietary and three open-source large language models (LLMs) in automatically classifying these errors into 26 and 15 predefined categories. We further developed a machine-assisted framework, MedError, which integrates best practices in error analysis, LLM-assisted classification and reasoning, and a user-friendly interface to enable more efficient, reproducible, and context-aware error analysis. The framework supports both single-site and federated multisite error analysis, facilitating the effective translation of clinical NLP systems into real-world settings.

Altered bone quality, caused by underlying metabolic changes of type 2 diabetes (T2D), has been hypothesized to cause altered bone strength and turnover leading to increased fracture risk in T2D patients. Current understanding about changes in bone turnover markers in T2D patients is mainly based on studies focused on Caucasian men and women. However, Hispanic populations have the highest prevalence of both T2D and osteoporosis in the US. We investigated associations of glycemic control (in terms of glycated hemoglobin [HbA1c]) and bone turnover rate in 69 older (≥50 years) Mexican American Cameron County Hispanic Cohort (CCHC) participants with T2D. Multivariable analyses were conducted to assess the associations between HbA1c (%), serum osteocalcin (OC), and serum sclerostin. In agreement with published reports from other racial/ethnic populations, our study found that lower bone turnover (indicated by lower serum OC) occurred in Mexican American men with T2D who had poorer glycemic control. For the women in our study, we found no significant association between glycemic control and OC. In contrast, HbA1c was positively associated with sclerostin for women, with near significance (p=0.07), while no association was found in men. We recommend screening Mexican American individuals with T2D, specifically those with poor glycemic control, for bone loss and fracture risk.