Citations

Abstract Objectives The surge in patient portal messages (PPMs) with increasing needs and workloads for efficient PPM triage in healthcare settings has spurred the exploration of AI-driven solutions to streamline the healthcare workflow processes, ensuring timely responses to patients to satisfy their healthcare needs. However, there has been less focus on isolating and understanding patient primary concerns in PPMs—a practice which holds the potential to yield more nuanced insights and enhances the quality of healthcare delivery and patient-centered care. Materials and Methods We propose a fusion framework to leverage pretrained language models (LMs) with different language advantages via a Convolution Neural Network for precise identification of patient primary concerns via multi-class classification. We examined 3 traditional machine learning models, 9 BERT-based language models, 6 fusion models, and 2 ensemble models. Results The outcomes of our experimentation underscore the superior performance achieved by BERT-based models in comparison to traditional machine learning models. Remarkably, our fusion model emerges as the top-performing solution, delivering a notably improved accuracy score of 77.67 ± 2.74% and an F1 score of 74.37 ± 3.70% in macro-average. Discussion This study highlights the feasibility and effectiveness of multi-class classification for patient primary concern detection and the proposed fusion framework for enhancing primary concern detection. Conclusions The use of multi-class classification enhanced by a fusion of multiple pretrained LMs not only improves the accuracy and efficiency of patient primary concern identification in PPMs but also aids in managing the rising volume of PPMs in healthcare, ensuring critical patient communications are addressed promptly and accurately.

Abstract Background Error analysis plays a crucial role in clinical concept extraction, a fundamental subtask within clinical natural language processing (NLP). The process typically involves a manual review of error types, such as contextual and linguistic factors contributing to their occurrence, and the identification of underlying causes to refine the NLP model and improve its performance. Conducting error analysis can be complex, requiring a combination of NLP expertise and domain-specific knowledge. Due to the high heterogeneity of electronic health record (EHR) settings across different institutions, challenges may arise when attempting to standardize and reproduce the error analysis process. Objectives This study aims to facilitate a collaborative effort to establish common definitions and taxonomies for capturing diverse error types, fostering community consensus on error analysis for clinical concept extraction tasks. Materials and Methods We iteratively developed and evaluated an error taxonomy based on existing literature, standards, real-world data, multisite case evaluations, and community feedback. The finalized taxonomy was released in both .dtd and .owl formats at the Open Health Natural Language Processing Consortium. The taxonomy is compatible with several different open-source annotation tools, including MAE, Brat, and MedTator. Results The resulting error taxonomy comprises 43 distinct error classes, organized into 6 error dimensions and 4 properties, including model type (symbolic and statistical machine learning), evaluation subject (model and human), evaluation level (patient, document, sentence, and concept), and annotation examples. Internal and external evaluations revealed strong variations in error types across methodological approaches, tasks, and EHR settings. Key points emerged from community feedback, including the need to enhancing clarity, generalizability, and usability of the taxonomy, along with dissemination strategies. Conclusion The proposed taxonomy can facilitate the acceleration and standardization of the error analysis process in multi-site settings, thus improving the provenance, interpretability, and portability of NLP models. Future researchers could explore the potential direction of developing automated or semi-automated methods to assist in the classification and standardization of error analysis.

Abstract Advances in genetic technology have led to the increasing use of genomic panels in precision oncology practice, with panels ranging from a couple to hundreds of genes. However, the clinical utilization and utility of oncology genomic panels, especially among vulnerable populations, is unclear. We examined the association of panel size with socioeconomic status and clinical trial matching. We retrospectively identified 9,886 eligible adult subjects in the Mayo Clinic Health System who underwent genomic testing between January 1, 2016 and June 30, 2020. Patient data were retrieved from structured and unstructured data sources of institutional collections, including cancer registries, clinical data warehouses, and clinical notes. Socioeconomic surrogates were approximated using the Area Deprivation Index (ADI) corresponding to primary residence addresses. Logistic regression was performed to analyze relationships between ADI or rural/urban status and (i) use of genomic test by panel size; (ii) clinical trial matching status. Compared with patients from the most affluent areas, patients had a lower odds of receiving a panel test (vs. a single-gene test) if from areas of higher socioeconomic deprivation [OR (95% confidence interval (CI): 0.71 (0.61–0.83), P < 0.01] or a rural area [OR (95% CI): 0.85 (0.76–0.96), P < 0.01]. Patients in areas of higher socioeconomic deprivation were less likely to be matched to clinical trials if receiving medium panel tests [(OR) (95% CI): 0.69 (0.49–0.97), P = 0.03]; however, there was no difference among patients receiving large panel tests (P > 0.05) and rural patients were almost 2x greater odds of being matched if receiving a large panel test [(OR) (95% CI): 1.76 (1.21–2.55), P < 0.01]. Significance: We identified socioeconomic and rurality disparities in the use of genomic tests and trial matching by panel size, which may have implications for equal access to targeted therapies. The lack of association between large panel tests and clinical trial matching by socioeconomic status, suggests a potential health equity impact, while removing barriers in access to large panels for rural patients may improve access to trials. However, further research is needed.

AbstractTo address the current and long‐term unmet health needs of the growing population of non‐Hodgkin lymphoma (NHL) patients, we established the Lymphoma Epidemiology of Outcomes (LEO) cohort study (NCT02736357; https://leocohort.org/). A total of 7735 newly diagnosed patients aged 18 years and older with NHL were prospectively enrolled from 7/1/2015 to 5/31/2020 at 8 academic centers in the United States. The median age at diagnosis was 62 years (range, 18–99). Participants came from 49 US states and included 538 Black/African‐Americans (AA), 822 Hispanics (regardless of race), 3386 women, 716 age <40 years, and 1513 rural residents. At study baseline, we abstracted clinical, pathology, and treatment data; banked serum/plasma (N = 5883, 76.0%) and germline DNA (N = 5465, 70.7%); constructed tissue microarrays for four major NHL subtypes (N = 1189); and collected quality of life (N = 5281, 68.3%) and epidemiologic risk factor (N = 4489, 58.0%) data. Through August 2022, there were 1492 deaths. Compared to population‐based SEER data (2015–2019), LEO participants had a similar distribution of gender, AA race, Hispanic ethnicity, and NHL subtype, while LEO was underrepresented for patients who were Asian and aged 80 years and above. Observed overall survival rates for LEO at 1 and 2 years were similar to population‐based SEER rates for indolent B‐cell (follicular and marginal zone) and T‐cell lymphomas, but were 10%–15% higher than SEER rates for aggressive B‐cell subtypes (diffuse large B‐cell and mantle cell). The LEO cohort is a robust and comprehensive national resource to address the role of clinical, tumor, host genetic, epidemiologic, and other biologic factors in NHL prognosis and survivorship.

Abstract Objective Transformer-based language models are prevailing in the clinical domain due to their excellent performance on clinical NLP tasks. The generalizability of those models is usually ignored during the model development process. This study evaluated the generalizability of CancerBERT, a Transformer-based clinical NLP model, along with classic machine learning models, i.e., conditional random field (CRF), bi-directional long short-term memory CRF (BiLSTM-CRF), across different clinical institutes through a breast cancer phenotype extraction task. Materials and methods Two clinical corpora of breast cancer patients were collected from the electronic health records from the University of Minnesota (UMN) and Mayo Clinic (MC), and annotated following the same guideline. We developed three types of NLP models (i.e., CRF, BiLSTM-CRF and CancerBERT) to extract cancer phenotypes from clinical texts. We evaluated the generalizability of models on different test sets with different learning strategies (model transfer vs locally trained). The entity coverage score was assessed with their association with the model performances. Results We manually annotated 200 and 161 clinical documents at UMN and MC. The corpora of the two institutes were found to have higher similarity between the target entities than the overall corpora. The CancerBERT models obtained the best performances among the independent test sets from two clinical institutes and the permutation test set. The CancerBERT model developed in one institute and further fine-tuned in another institute achieved reasonable performance compared to the model developed on local data (micro-F1: 0.925 vs 0.932). Conclusions The results indicate the CancerBERT model has superior learning ability and generalizability among the three types of clinical NLP models for our named entity recognition task. It has the advantage to recognize complex entities, e.g., entities with different labels.