Liu, Hongfang

Abstract Background Error analysis plays a crucial role in clinical concept extraction, a fundamental subtask within clinical natural language processing (NLP). The process typically involves a manual review of error types, such as contextual and linguistic factors contributing to their occurrence, and the identification of underlying causes to refine the NLP model and improve its performance. Conducting error analysis can be complex, requiring a combination of NLP expertise and domain-specific knowledge. Due to the high heterogeneity of electronic health record (EHR) settings across different institutions, challenges may arise when attempting to standardize and reproduce the error analysis process. Objectives This study aims to facilitate a collaborative effort to establish common definitions and taxonomies for capturing diverse error types, fostering community consensus on error analysis for clinical concept extraction tasks. Materials and Methods We iteratively developed and evaluated an error taxonomy based on existing literature, standards, real-world data, multisite case evaluations, and community feedback. The finalized taxonomy was released in both .dtd and .owl formats at the Open Health Natural Language Processing Consortium. The taxonomy is compatible with several different open-source annotation tools, including MAE, Brat, and MedTator. Results The resulting error taxonomy comprises 43 distinct error classes, organized into 6 error dimensions and 4 properties, including model type (symbolic and statistical machine learning), evaluation subject (model and human), evaluation level (patient, document, sentence, and concept), and annotation examples. Internal and external evaluations revealed strong variations in error types across methodological approaches, tasks, and EHR settings. Key points emerged from community feedback, including the need to enhancing clarity, generalizability, and usability of the taxonomy, along with dissemination strategies. Conclusion The proposed taxonomy can facilitate the acceleration and standardization of the error analysis process in multi-site settings, thus improving the provenance, interpretability, and portability of NLP models. Future researchers could explore the potential direction of developing automated or semi-automated methods to assist in the classification and standardization of error analysis.

Abstract Advances in genetic technology have led to the increasing use of genomic panels in precision oncology practice, with panels ranging from a couple to hundreds of genes. However, the clinical utilization and utility of oncology genomic panels, especially among vulnerable populations, is unclear. We examined the association of panel size with socioeconomic status and clinical trial matching. We retrospectively identified 9,886 eligible adult subjects in the Mayo Clinic Health System who underwent genomic testing between January 1, 2016 and June 30, 2020. Patient data were retrieved from structured and unstructured data sources of institutional collections, including cancer registries, clinical data warehouses, and clinical notes. Socioeconomic surrogates were approximated using the Area Deprivation Index (ADI) corresponding to primary residence addresses. Logistic regression was performed to analyze relationships between ADI or rural/urban status and (i) use of genomic test by panel size; (ii) clinical trial matching status. Compared with patients from the most affluent areas, patients had a lower odds of receiving a panel test (vs. a single-gene test) if from areas of higher socioeconomic deprivation [OR (95% confidence interval (CI): 0.71 (0.61–0.83), P < 0.01] or a rural area [OR (95% CI): 0.85 (0.76–0.96), P < 0.01]. Patients in areas of higher socioeconomic deprivation were less likely to be matched to clinical trials if receiving medium panel tests [(OR) (95% CI): 0.69 (0.49–0.97), P = 0.03]; however, there was no difference among patients receiving large panel tests (P > 0.05) and rural patients were almost 2x greater odds of being matched if receiving a large panel test [(OR) (95% CI): 1.76 (1.21–2.55), P < 0.01]. Significance: We identified socioeconomic and rurality disparities in the use of genomic tests and trial matching by panel size, which may have implications for equal access to targeted therapies. The lack of association between large panel tests and clinical trial matching by socioeconomic status, suggests a potential health equity impact, while removing barriers in access to large panels for rural patients may improve access to trials. However, further research is needed.

AbstractTo address the current and long‐term unmet health needs of the growing population of non‐Hodgkin lymphoma (NHL) patients, we established the Lymphoma Epidemiology of Outcomes (LEO) cohort study (NCT02736357; https://leocohort.org/). A total of 7735 newly diagnosed patients aged 18 years and older with NHL were prospectively enrolled from 7/1/2015 to 5/31/2020 at 8 academic centers in the United States. The median age at diagnosis was 62 years (range, 18–99). Participants came from 49 US states and included 538 Black/African‐Americans (AA), 822 Hispanics (regardless of race), 3386 women, 716 age <40 years, and 1513 rural residents. At study baseline, we abstracted clinical, pathology, and treatment data; banked serum/plasma (N = 5883, 76.0%) and germline DNA (N = 5465, 70.7%); constructed tissue microarrays for four major NHL subtypes (N = 1189); and collected quality of life (N = 5281, 68.3%) and epidemiologic risk factor (N = 4489, 58.0%) data. Through August 2022, there were 1492 deaths. Compared to population‐based SEER data (2015–2019), LEO participants had a similar distribution of gender, AA race, Hispanic ethnicity, and NHL subtype, while LEO was underrepresented for patients who were Asian and aged 80 years and above. Observed overall survival rates for LEO at 1 and 2 years were similar to population‐based SEER rates for indolent B‐cell (follicular and marginal zone) and T‐cell lymphomas, but were 10%–15% higher than SEER rates for aggressive B‐cell subtypes (diffuse large B‐cell and mantle cell). The LEO cohort is a robust and comprehensive national resource to address the role of clinical, tumor, host genetic, epidemiologic, and other biologic factors in NHL prognosis and survivorship.

Abstract Objective Transformer-based language models are prevailing in the clinical domain due to their excellent performance on clinical NLP tasks. The generalizability of those models is usually ignored during the model development process. This study evaluated the generalizability of CancerBERT, a Transformer-based clinical NLP model, along with classic machine learning models, i.e., conditional random field (CRF), bi-directional long short-term memory CRF (BiLSTM-CRF), across different clinical institutes through a breast cancer phenotype extraction task. Materials and methods Two clinical corpora of breast cancer patients were collected from the electronic health records from the University of Minnesota (UMN) and Mayo Clinic (MC), and annotated following the same guideline. We developed three types of NLP models (i.e., CRF, BiLSTM-CRF and CancerBERT) to extract cancer phenotypes from clinical texts. We evaluated the generalizability of models on different test sets with different learning strategies (model transfer vs locally trained). The entity coverage score was assessed with their association with the model performances. Results We manually annotated 200 and 161 clinical documents at UMN and MC. The corpora of the two institutes were found to have higher similarity between the target entities than the overall corpora. The CancerBERT models obtained the best performances among the independent test sets from two clinical institutes and the permutation test set. The CancerBERT model developed in one institute and further fine-tuned in another institute achieved reasonable performance compared to the model developed on local data (micro-F1: 0.925 vs 0.932). Conclusions The results indicate the CancerBERT model has superior learning ability and generalizability among the three types of clinical NLP models for our named entity recognition task. It has the advantage to recognize complex entities, e.g., entities with different labels.

Abstract Despite recent methodology advancements in clinical natural language processing (NLP), the adoption of clinical NLP models within the translational research community remains hindered by process heterogeneity and human factor variations. Concurrently, these factors also dramatically increase the difficulty in developing NLP models in multi-site settings, which is necessary for algorithm robustness and generalizability. Here, we reported on our experience developing an NLP solution for Coronavirus Disease 2019 (COVID-19) signs and symptom extraction in an open NLP framework from a subset of sites participating in the National COVID Cohort (N3C). We then empirically highlight the benefits of multi-site data for both symbolic and statistical methods, as well as highlight the need for federated annotation and evaluation to resolve several pitfalls encountered in the course of these efforts.