Citations

Abstract Background Error analysis plays a crucial role in clinical concept extraction, a fundamental subtask within clinical natural language processing (NLP). The process typically involves a manual review of error types, such as contextual and linguistic factors contributing to their occurrence, and the identification of underlying causes to refine the NLP model and improve its performance. Conducting error analysis can be complex, requiring a combination of NLP expertise and domain-specific knowledge. Due to the high heterogeneity of electronic health record (EHR) settings across different institutions, challenges may arise when attempting to standardize and reproduce the error analysis process. Objectives This study aims to facilitate a collaborative effort to establish common definitions and taxonomies for capturing diverse error types, fostering community consensus on error analysis for clinical concept extraction tasks. Materials and Methods We iteratively developed and evaluated an error taxonomy based on existing literature, standards, real-world data, multisite case evaluations, and community feedback. The finalized taxonomy was released in both .dtd and .owl formats at the Open Health Natural Language Processing Consortium. The taxonomy is compatible with several different open-source annotation tools, including MAE, Brat, and MedTator. Results The resulting error taxonomy comprises 43 distinct error classes, organized into 6 error dimensions and 4 properties, including model type (symbolic and statistical machine learning), evaluation subject (model and human), evaluation level (patient, document, sentence, and concept), and annotation examples. Internal and external evaluations revealed strong variations in error types across methodological approaches, tasks, and EHR settings. Key points emerged from community feedback, including the need to enhancing clarity, generalizability, and usability of the taxonomy, along with dissemination strategies. Conclusion The proposed taxonomy can facilitate the acceleration and standardization of the error analysis process in multi-site settings, thus improving the provenance, interpretability, and portability of NLP models. Future researchers could explore the potential direction of developing automated or semi-automated methods to assist in the classification and standardization of error analysis.

Abstract Background Multimorbidity and functional limitation are associated with poor outcomes in heart failure (HF). However, the individual and combined effect of these on health‐related quality of life in patients with HF is not well understood. Methods Patients aged ≥30 years with two or more HF diagnostic codes and one or more HF‐related prescription drugs from four U.S. institutions were mailed a survey to measure patient‐centric factors including functional status (activities of daily living [ADLs]) and health‐related quality of life (PROMIS‐29 Health Profile). Patients with HF from January 1, 2013 to February 1, 2018 were included. Multimorbidity was defined as ≥2 non‐cardiovascular comorbidities; functional limitation as any limitation in at least one of eight ADLs. Patients were categorized into four groups by multimorbidity (Yes/No) and functional limitation (Yes/No). We dichotomized the PROMIS‐29 sub‐scale scores at the median and calculated odd ratios for the four multimorbidity/functional limitation groups. Results A total of 3330 patients with HF returned the survey (response rate 31%); 3020 completed the questions of interest and were retained. Among these patients (45% female; mean age 73 [standard deviation: 12] years), 29% had neither multimorbidity nor functional limitation, 24% had multimorbidity only, 22% had functional limitation only, and 25% had both. After adjustment, having functional limitation only was associated with higher anxiety (odds ratio [OR]: 3.44, 95% confidence interval [CI]: 2.66–4.45), depression (OR: 3.11, 95% CI: 2.39–4.06), and fatigue (OR: 4.19, 95% CI: 3.25–5.40); worse sleep (OR: 2.14, 95% CI: 1.69–2.72) and pain (OR: 6.73, 95% CI: 5.15–8.78); and greater difficulty with social activities (OR: 9.40, 95% CI: 7.19–12.28) compared with having neither. Results were similar for having both multimorbidity and functional limitation. Conclusion Patients with only functional limitation have similar poor health‐related quality of life scores as those with both multimorbidity and functional limitation, underscoring the important role that physical functioning plays in the well‐being of patients with HF.

55 Background: Nearly one-half of deficient mismatch repair (d-MMR) metastatic colorectal cancers (mCRC) demonstrate durable responses to immune checkpoint inhibitors (ICIs). Given preclinical data indicating that liver metastases sequester activated CD8+ T cells from systemic circulation, we examined clinical outcome by metastatic site. Methods: In a retrospective cohort of patients with d-MMR mCRCs treated at multiple centers in France (n=66), we sought to validate data from a U.S. cohort, and then performed pooled analysis (n=104). All patients received first-line ICI monotherapy. Metastatic site was analyzed in relationship to tumor response (RECIST version 1.1), and progression-free survival (PFS) by multivariable stratified Cox regression after adjustment for covariates. Results: Objective responses were achieved in 38/66 (58%) patients in the validation cohort. Best tumor response included 13 (20%) complete responses (CR), 25 (38%) partial responses (PR), 16 (25%) stable disease, and 11 (17%) progressive disease (PD). One-year and 5-year PFS rates were 73% and 67%, respectively; 18 (27%) patients progressed during immunotherapy. Best tumor response was attenuated in patients with liver metastasis ( P=0.03). Presence of liver metastasis, but not other sites, was associated with significantly poorer PFS after adjustment for covariates (HRadj 2.82; 95%CI, 1.08-7.39; Padj=0.03). In a pooled analysis, liver metastasis remained significantly and independently associated with poorer PFS (HRadj 3.18; 95%CI, 1.52-6.67; Padj=0.002) and with attenuated tumor best response (P=0.01). Conclusions: Metastasis to the liver, but not other sites, was validated as an independent factor associated with poorer response and survival after ICI treatment in d-MMR mCRCs. These data underscore the need for novel therapeutic strategies in these patients. [Table: see text]