Citations

Abstract Background Delirium is often underdiagnosed in clinical practice and is not routinely coded for billing. While manual chart review can identify delirium, it is labor-intensive and impractical for large-scale studies. Natural language processing (NLP) can analyze unstructured text in electronic health records (EHRs) to extract meaningful clinical information. Methods To support national integration of NLP for EHR-based delirium identification across different institutions, we launched the Delirium Interest Group within the national Evolve to Next-Gen Accrual to Clinical Trials (ENACT) NLP Working Group. This paper outlines our initial efforts to standardize, evaluate, and translate an NLP-based delirium detection model into the i2b2/ENACT platform. Results Multisite contextual inquiry identified several key challenges, including variations in local screening practices (eg, tools used, documentation frequency, and quality control), the need for harmonized definitions in the context of EHRs, and the complexity of modeling temporal logic. Multisite NLP evaluation revealed variable performance degradation driven by differences in delirium screening practices, clinical documentation patterns and semantics, and note syntactic structures. Conclusion Our work represents an important first step toward enabling scalable and standardized NLP-based delirium detection across institutions. By engaging diverse institutions through the ENACT NLP Working Group, we identified shared challenges and site-specific variations that impact model implementation and performance. Our collaborative approach enabled the development of a more robust framework for delirium identification across heterogeneous EHR systems. Future efforts will build on this foundation to enhance the validity, usability, and translational impact of delirium detection.

Abstract Error analysis is a critical step in evaluating and improving clinical concept extraction models, the most common clinical natural language processing (NLP) task. Unlike corpus annotation, which follows standardized protocols for creating gold-standard datasets, error analysis requires nuanced judgment grounded in both clinical expertise and NLP knowledge. This is especially important given the heterogeneity of clinical text, where variations in documentation style, note structure, and terminology can substantially influence model behavior. Despite its importance, there is currently no standardized, user-level framework to support systematic error analysis in clinical concept extraction task. In this study, we developed and validated MedError, a machine-assisted, human-in-the-loop framework designed to standardize and enhance error analysis for clinical concept extraction tasks. We collected and manually curated a corpus of 1,187 unique errors from a total of 4,237 notes across three different distinct hospitals. The error categories were defined using our previously validated error taxonomy and included 480 false negatives and 707 false positives across 25 error types and 48 clinical concept categories. We evaluated the performance of three proprietary and three open-source large language models (LLMs) in automatically classifying these errors into 26 and 15 predefined categories. We further developed a machine-assisted framework, MedError, which integrates best practices in error analysis, LLM-assisted classification and reasoning, and a user-friendly interface to enable more efficient, reproducible, and context-aware error analysis. The framework supports both single-site and federated multisite error analysis, facilitating the effective translation of clinical NLP systems into real-world settings.

Abstract Current discussion surrounding the clinical capabilities of generative language models (GLMs) predominantly center around multiple-choice question-answer (MCQA) benchmarks derived from clinical licensing examinations. While accepted for human examinees, characteristics unique to GLMs bring into question the validity of such benchmarks. Here, we validate four benchmarks using eight GLMs, ablating for parameter size and reasoning capabilities, validating via prompt permutation three key assumptions that underpin the generalizability of MCQA-based assessments: that knowledge is applied, not memorized, that semantic consistency will lead to consistent answers, and that situations with no answers can be recognized. While large models are more resilient to our perturbations compared to small models, we globally invalidate these assumptions, with implications for reasoning models. Additionally, despite retaining the knowledge, small models are prone to answer memorization. All models exhibit significant failure in null-answer scenarios. We then suggest several adaptations for more robust benchmark designs more reflective of real-world conditions.

Abstract Objective: Electronic Health Record (EHR) data are critical for advancing translational research and AI technologies. The ENACT network offers access to structured EHR data across 57 CTSA hubs. However, substantial information is contained in clinical narratives, requiring natural language processing (NLP) for research. The ENACT NLP Working Group was formed to make NLP-derived clinical information accessible and queryable across the network. Methods: We established the ENACT NLP Working Group with 13 sites selected based on criteria including clinical notes access, IT infrastructure, NLP expertise, and institutional support. We divided sites into five focus groups targeting clinical tasks within disease contexts. Each focus group consisted of two development sites and two validation sites. We extended the ENACT ontology to standardize NLP-derived data and conducted multisite evaluations using the Open Health Natural Language Processing (OHNLP) Toolkit. Results: The working group achieved 100% site retention and deployed NLP infrastructure across all sites. We developed and validated NLP algorithms for rare disease phenotyping, social determinants of health, opioid use disorder, sleep phenotyping, and delirium phenotyping. Performance varied across sites (F1 scores 0.53–0.96), highlighting data heterogeneity impacts. We extended the ENACT common data model and ontology to incorporate NLP-derived data while maintaining Shared Health Research Informatics NEtwork (SHRINE) compatibility. Conclusion: This demonstrates feasibility of deploying NLP infrastructure across large, federated networks. The focus group approach proved more practical than general-purpose approaches. Key lessons include the challenge of data heterogeneity and importance of collaborative governance. This work also provides a foundation that other networks can build on to implement NLP capabilities for translational research.

ABSTRACT Background Preparation of the right ventricular outflow tract (RVOT) using covered stent placement is a common preparatory step for transcatheter pulmonary valve replacement (TPVR) to reduce the risk of disruption associated with dilating calcified RVOT conduits. Aim We present a novel method employing a self‐expanding covered aortic endoprosthesis and discuss its advantages as an alternative strategy. Methods Patients with congenital heart disease undergoing RVOT conduit preparation with the GORE TAG Thoracic Branch Endoprosthesis Extender (W.L. Gore & Associates, Newark, DE) followed by TPVR, were prospectively enrolled in a dedicated database. To date, 14 patients have been included. Data on demographics, diagnoses, procedural details, hemodynamic parameters, and outcomes were analyzed. Results Fourteen patients (median age: 35 years [range: 18–79]; 57% female) successfully underwent endograft placement immediately before TPVR. Treated conduits included pulmonary homografts ( n = 8), surgical bioprostheses ( n = 2), Hancock conduit ( n = 1), Contegra conduit ( n = 1), native RVOT ( n = 1), and Melody valve within a surgical bioprosthesis ( n = 1). Most patients presented with mixed stenosis and regurgitation ( n = 9), with the remaining having stenosis ( n = 4) or regurgitation ( n = 1). Valves implanted post‐preparation included the Sapien S3 valve (Edwards Lifesciences, Irvine, CA) ( n = 10) and the Melody valve (Medtronic, Minneapolis, MN) ( n = 5). Significant reductions in RVOT systolic gradient were observed (median 34 mmHg pre‐procedure to 3 mmHg post‐procedure), along with marked conduit expansion (median diameter 13 mm pre‐procedure to 25.5 mm post‐procedure). No complications were reported. Conclusions This study highlights the feasibility of a self‐expanding aortic endoprosthesis for RVOT preparation before TPVR. Further comparative study is warranted. Procedural success was achieved in all cases.

ABSTRACT Introduction Improving organ preservation is essential for expanding the donor pool and enhancing outcomes for pediatric transplant recipients. This report presents three pediatric patients who received donor hearts preserved using the Organ Care System (OCS). All donor hearts were prepared, placed, and monitored on OCS according to standard protocols. Heart function and myocardial edema were visually assessed during monitoring. Case Reports Patient 1: A 17‐year‐old female patient with hypertrophic cardiomyopathy utilized OCS technology due to donor distance. She experienced an uncomplicated post‐transplant course and was discharged on post‐transplant day 10. Patient 2: A 14‐year‐old male patient with complex congenital heart disease, palliated with Fontan circulation, underwent orthotopic heart transplantation (OHT) with OCS support due to both the complexity of the procedure and donor heart distance. His post‐transplant course was complicated by a revision of the ascending aorta anastomosis and prolonged milrinone infusion for diastolic dysfunction. He was discharged on post‐transplant day 31. Patient 3: A 16‐year‐old male patient with hypoplastic left heart syndrome, also palliated with Fontan circulation, underwent OHT using OCS technology due to the complexity of the surgical procedure. His post‐transplant recovery was unremarkable, and he was discharged on post‐transplant day 15. Conclusions All three pediatric patients demonstrated favorable short‐term outcomes using OCS technology. The OCS system facilitated shorter ischemic times despite the need for extended transport distances or complex surgical procedures. However, further studies with larger patient cohorts and longer follow‐up are needed to better understand the long‐term impact of OCS on pediatric heart transplantation outcomes.

Solitary myofibromas are rare, benign, mesenchymal tumors typically found in the head and neck regions. We report an exceptionally rare case of a solitary cardiac myofibroma and its short-term management and outcome. A two-month-old infant was referred due to a murmur. Echocardiography revealed a 2 × 1 cm solitary mass originating from a mitral papillary muscle, causing severe regurgitation. Magnetic resonance imaging showed the mass to be homogeneous on T1, hyperintense on T2, and hypointense on first-pass perfusion imaging with no enhancement on delayed postcontrast imaging. The mass was surgically excised partially due to its involvement with the mitral valve. Postoperative recovery was uneventful, with echocardiography four months later showing no tumor recurrence and trivial mitral regurgitation. This report provides valuable imaging and pathology examples to aid in the recognition of this rare tumor and demonstrates that successful surgical excision is possible, even with significant cardiac involvement.

Abstract Background Error analysis plays a crucial role in clinical concept extraction, a fundamental subtask within clinical natural language processing (NLP). The process typically involves a manual review of error types, such as contextual and linguistic factors contributing to their occurrence, and the identification of underlying causes to refine the NLP model and improve its performance. Conducting error analysis can be complex, requiring a combination of NLP expertise and domain-specific knowledge. Due to the high heterogeneity of electronic health record (EHR) settings across different institutions, challenges may arise when attempting to standardize and reproduce the error analysis process. Objectives This study aims to facilitate a collaborative effort to establish common definitions and taxonomies for capturing diverse error types, fostering community consensus on error analysis for clinical concept extraction tasks. Materials and Methods We iteratively developed and evaluated an error taxonomy based on existing literature, standards, real-world data, multisite case evaluations, and community feedback. The finalized taxonomy was released in both .dtd and .owl formats at the Open Health Natural Language Processing Consortium. The taxonomy is compatible with several different open-source annotation tools, including MAE, Brat, and MedTator. Results The resulting error taxonomy comprises 43 distinct error classes, organized into 6 error dimensions and 4 properties, including model type (symbolic and statistical machine learning), evaluation subject (model and human), evaluation level (patient, document, sentence, and concept), and annotation examples. Internal and external evaluations revealed strong variations in error types across methodological approaches, tasks, and EHR settings. Key points emerged from community feedback, including the need to enhancing clarity, generalizability, and usability of the taxonomy, along with dissemination strategies. Conclusion The proposed taxonomy can facilitate the acceleration and standardization of the error analysis process in multi-site settings, thus improving the provenance, interpretability, and portability of NLP models. Future researchers could explore the potential direction of developing automated or semi-automated methods to assist in the classification and standardization of error analysis.

Abstract Objective Transformer-based language models are prevailing in the clinical domain due to their excellent performance on clinical NLP tasks. The generalizability of those models is usually ignored during the model development process. This study evaluated the generalizability of CancerBERT, a Transformer-based clinical NLP model, along with classic machine learning models, i.e., conditional random field (CRF), bi-directional long short-term memory CRF (BiLSTM-CRF), across different clinical institutes through a breast cancer phenotype extraction task. Materials and methods Two clinical corpora of breast cancer patients were collected from the electronic health records from the University of Minnesota (UMN) and Mayo Clinic (MC), and annotated following the same guideline. We developed three types of NLP models (i.e., CRF, BiLSTM-CRF and CancerBERT) to extract cancer phenotypes from clinical texts. We evaluated the generalizability of models on different test sets with different learning strategies (model transfer vs locally trained). The entity coverage score was assessed with their association with the model performances. Results We manually annotated 200 and 161 clinical documents at UMN and MC. The corpora of the two institutes were found to have higher similarity between the target entities than the overall corpora. The CancerBERT models obtained the best performances among the independent test sets from two clinical institutes and the permutation test set. The CancerBERT model developed in one institute and further fine-tuned in another institute achieved reasonable performance compared to the model developed on local data (micro-F1: 0.925 vs 0.932). Conclusions The results indicate the CancerBERT model has superior learning ability and generalizability among the three types of clinical NLP models for our named entity recognition task. It has the advantage to recognize complex entities, e.g., entities with different labels.