Citations

Background/Objectives: Lung cancer is a devastating disease with the highest mortality rate among cancer types. Over 60% of non-small cell lung cancer (NSCLC) patients, accounting for 87% of lung cancer diagnoses, require radiation therapy. Rapid treatment initiation significantly increases the patient’s survival rate and reduces the mortality rate. Accurate tumor segmentation is a critical step in diagnosing and treating NSCLC. Manual segmentation is time- and labor-consuming and causes delays in treatment initiation. Although many lung nodule detection methods, including deep learning-based models, have been proposed. Most of these methods still have a long-standing problem of high false positives (FPs). Methods: Here, we developed an electronic health record (EHR)-guided lung tumor auto-segmentation called EXACT-Net (EHR-enhanced eXACtitude in Tumor segmentation), where the extracted information from EHRs using a pre-trained large language model (LLM) was used to remove the FPs and keep the TP nodules only. Results: The auto-segmentation model was trained on NSCLC patients’ computed tomography (CT), and the pre-trained LLM was used with the zero-shot learning approach. Our approach resulted in a 250% boost in successful nodule detection using the data from ten NSCLC patients treated in our institution. Conclusions: We demonstrated that combining vision-language information in EXACT-Net multi-modal AI framework greatly enhances the performance of vision only models, paving the road to multimodal AI framework for medical image processing.

Abstract Background Error analysis plays a crucial role in clinical concept extraction, a fundamental subtask within clinical natural language processing (NLP). The process typically involves a manual review of error types, such as contextual and linguistic factors contributing to their occurrence, and the identification of underlying causes to refine the NLP model and improve its performance. Conducting error analysis can be complex, requiring a combination of NLP expertise and domain-specific knowledge. Due to the high heterogeneity of electronic health record (EHR) settings across different institutions, challenges may arise when attempting to standardize and reproduce the error analysis process. Objectives This study aims to facilitate a collaborative effort to establish common definitions and taxonomies for capturing diverse error types, fostering community consensus on error analysis for clinical concept extraction tasks. Materials and Methods We iteratively developed and evaluated an error taxonomy based on existing literature, standards, real-world data, multisite case evaluations, and community feedback. The finalized taxonomy was released in both .dtd and .owl formats at the Open Health Natural Language Processing Consortium. The taxonomy is compatible with several different open-source annotation tools, including MAE, Brat, and MedTator. Results The resulting error taxonomy comprises 43 distinct error classes, organized into 6 error dimensions and 4 properties, including model type (symbolic and statistical machine learning), evaluation subject (model and human), evaluation level (patient, document, sentence, and concept), and annotation examples. Internal and external evaluations revealed strong variations in error types across methodological approaches, tasks, and EHR settings. Key points emerged from community feedback, including the need to enhancing clarity, generalizability, and usability of the taxonomy, along with dissemination strategies. Conclusion The proposed taxonomy can facilitate the acceleration and standardization of the error analysis process in multi-site settings, thus improving the provenance, interpretability, and portability of NLP models. Future researchers could explore the potential direction of developing automated or semi-automated methods to assist in the classification and standardization of error analysis.

Abstract Objective Transformer-based language models are prevailing in the clinical domain due to their excellent performance on clinical NLP tasks. The generalizability of those models is usually ignored during the model development process. This study evaluated the generalizability of CancerBERT, a Transformer-based clinical NLP model, along with classic machine learning models, i.e., conditional random field (CRF), bi-directional long short-term memory CRF (BiLSTM-CRF), across different clinical institutes through a breast cancer phenotype extraction task. Materials and methods Two clinical corpora of breast cancer patients were collected from the electronic health records from the University of Minnesota (UMN) and Mayo Clinic (MC), and annotated following the same guideline. We developed three types of NLP models (i.e., CRF, BiLSTM-CRF and CancerBERT) to extract cancer phenotypes from clinical texts. We evaluated the generalizability of models on different test sets with different learning strategies (model transfer vs locally trained). The entity coverage score was assessed with their association with the model performances. Results We manually annotated 200 and 161 clinical documents at UMN and MC. The corpora of the two institutes were found to have higher similarity between the target entities than the overall corpora. The CancerBERT models obtained the best performances among the independent test sets from two clinical institutes and the permutation test set. The CancerBERT model developed in one institute and further fine-tuned in another institute achieved reasonable performance compared to the model developed on local data (micro-F1: 0.925 vs 0.932). Conclusions The results indicate the CancerBERT model has superior learning ability and generalizability among the three types of clinical NLP models for our named entity recognition task. It has the advantage to recognize complex entities, e.g., entities with different labels.

Abstract Despite recent methodology advancements in clinical natural language processing (NLP), the adoption of clinical NLP models within the translational research community remains hindered by process heterogeneity and human factor variations. Concurrently, these factors also dramatically increase the difficulty in developing NLP models in multi-site settings, which is necessary for algorithm robustness and generalizability. Here, we reported on our experience developing an NLP solution for Coronavirus Disease 2019 (COVID-19) signs and symptom extraction in an open NLP framework from a subset of sites participating in the National COVID Cohort (N3C). We then empirically highlight the benefits of multi-site data for both symbolic and statistical methods, as well as highlight the need for federated annotation and evaluation to resolve several pitfalls encountered in the course of these efforts.

Importance Numerous studies have established the association of the common APOE ε2 and APOE ε4 alleles with Alzheimer disease (AD) risk across ancestries. Studies of the interaction of these alleles with other amino acid changes on APOE in non-European ancestries are lacking and may improve ancestry-specific risk prediction. Objective To determine whether APOE amino acid changes specific to individuals of African ancestry modulate AD risk. Design, Setting, and Participants Case-control study including 31 929 participants and using a sequenced discovery sample (Alzheimer Disease Sequencing Project; stage 1) followed by 2 microarray imputed data sets derived from the Alzheimer Disease Genetic Consortium (stage 2, internal replication) and the Million Veteran Program (stage 3, external validation). This study combined case-control, family-based, population-based, and longitudinal AD cohorts, which recruited participants (1991-2022) in primarily US-based studies with 1 US/Nigerian study. Across all stages, individuals included in this study were of African ancestry. Exposures Two APOE missense variants (R145C and R150H) were assessed, stratified by APOE genotype. Main Outcomes and Measures The primary outcome was AD case-control status, and secondary outcomes included age at AD onset. Results Stage 1 included 2888 cases (median age, 77 [IQR, 71-83] years; 31.3% male) and 4957 controls (median age, 77 [IQR, 71-83] years; 28.0% male). In stage 2, across multiple cohorts, 1201 cases (median age, 75 [IQR, 69-81] years; 30.8% male) and 2744 controls (median age, 80 [IQR, 75-84] years; 31.4% male) were included. In stage 3, 733 cases (median age, 79.4 [IQR, 73.8-86.5] years; 97.0% male) and 19 406 controls (median age, 71.9 [IQR, 68.4-75.8] years; 94.5% male) were included. In ε3/ε4-stratified analyses of stage 1, R145C was present in 52 individuals with AD (4.8%) and 19 controls (1.5%); R145C was associated with an increased risk of AD (odds ratio [OR], 3.01; 95% CI, 1.87-4.85; P = 6.0 × 10 −6 ) and was associated with a reported younger age at AD onset (β, −5.87 years; 95% CI, −8.35 to −3.4 years; P = 3.4 × 10 −6 ). Association with increased AD risk was replicated in stage 2 (R145C was present in 23 individuals with AD [4.7%] and 21 controls [2.7%]; OR, 2.20; 95% CI, 1.04-4.65; P = .04) and was concordant in stage 3 (R145C was present in 11 individuals with AD [3.8%] and 149 controls [2.7%]; OR, 1.90; 95% CI, 0.99-3.64; P = .051). Association with earlier AD onset was replicated in stage 2 (β, −5.23 years; 95% CI, −9.58 to −0.87 years; P = .02) and stage 3 (β, −10.15 years; 95% CI, −15.66 to −4.64 years; P = 4.0 × 10 −4 ). No significant associations were observed in other APOE strata for R145C or in any APOE strata for R150H. Conclusions and Relevance In this exploratory analysis, the APOE ε3[R145C] missense variant was associated with an increased risk of AD among individuals of African ancestry with the ε3/ε4 genotype. With additional external validation, these findings may inform AD genetic risk assessment in individuals of African ancestry.

Abstract Objective Accurate extraction of breast cancer patients’ phenotypes is important for clinical decision support and clinical research. This study developed and evaluated cancer domain pretrained CancerBERT models for extracting breast cancer phenotypes from clinical texts. We also investigated the effect of customized cancer-related vocabulary on the performance of CancerBERT models. Materials and Methods A cancer-related corpus of breast cancer patients was extracted from the electronic health records of a local hospital. We annotated named entities in 200 pathology reports and 50 clinical notes for 8 cancer phenotypes for fine-tuning and evaluation. We kept pretraining the BlueBERT model on the cancer corpus with expanded vocabularies (using both term frequency-based and manually reviewed methods) to obtain CancerBERT models. The CancerBERT models were evaluated and compared with other baseline models on the cancer phenotype extraction task. Results All CancerBERT models outperformed all other models on the cancer phenotyping NER task. Both CancerBERT models with customized vocabularies outperformed the CancerBERT with the original BERT vocabulary. The CancerBERT model with manually reviewed customized vocabulary achieved the best performance with macro F1 scores equal to 0.876 (95% CI, 0.873–0.879) and 0.904 (95% CI, 0.902–0.906) for exact match and lenient match, respectively. Conclusions The CancerBERT models were developed to extract the cancer phenotypes in clinical notes and pathology reports. The results validated that using customized vocabulary may further improve the performances of domain specific BERT models in clinical NLP tasks. The CancerBERT models developed in the study would further help clinical decision support.

Background . There is growing evidence that social and behavioral determinants of health (SBDH) play a substantial effect in a wide range of health outcomes. Electronic health records (EHRs) have been widely employed to conduct observational studies in the age of artificial intelligence (AI). However, there has been limited review into how to make the most of SBDH information from EHRs using AI approaches. Methods . A systematic search was conducted in six databases to find relevant peer-reviewed publications that had recently been published. Relevance was determined by screening and evaluating the articles. Based on selected relevant studies, a methodological analysis of AI algorithms leveraging SBDH information in EHR data was provided. Results . Our synthesis was driven by an analysis of SBDH categories, the relationship between SBDH and healthcare-related statuses, natural language processing (NLP) approaches for extracting SBDH from clinical notes, and predictive models using SBDH for health outcomes. Discussion . The associations between SBDH and health outcomes are complicated and diverse; several pathways may be involved. Using NLP technology to support the extraction of SBDH and other clinical ideas simplifies the identification and extraction of essential concepts from clinical data, efficiently unlocks unstructured data, and aids in the resolution of unstructured data-related issues. Conclusion . Despite known associations between SBDH and diseases, SBDH factors are rarely investigated as interventions to improve patient outcomes. Gaining knowledge about SBDH and how SBDH data can be collected from EHRs using NLP approaches and predictive models improves the chances of influencing health policy change for patient wellness, ultimately promoting health and health equity.

AbstractObjectiveWith COVID-19, there was a need for a rapidly scalable annotation system that facilitated real-time integration with clinical decision support systems (CDS). Current annotation systems suffer from a high-resource utilization and poor scalability limiting real-world integration with CDS. A potential solution to mitigate these issues is to use the rule-based gazetteer developed at our institution.Materials and MethodsPerformance, resource utilization, and runtime of the rule-based gazetteer were compared with five annotation systems: BioMedICUS, cTAKES, MetaMap, CLAMP, and MedTagger.ResultsThis rule-based gazetteer was the fastest, had a low resource footprint, and similar performance for weighted microaverage and macroaverage measures of precision, recall, and f1-score compared to other annotation systems.DiscussionOpportunities to increase its performance include fine-tuning lexical rules for symptom identification. Additionally, it could run on multiple compute nodes for faster runtime.ConclusionThis rule-based gazetteer overcame key technical limitations facilitating real-time symptomatology identification for COVID-19 and integration of unstructured data elements into our CDS. It is ideal for large-scale deployment across a wide variety of healthcare settings for surveillance of acute COVID-19 symptoms for integration into prognostic modeling. Such a system is currently being leveraged for monitoring of postacute sequelae of COVID-19 (PASC) progression in COVID-19 survivors. This study conducted the first in-depth analysis and developed a rule-based gazetteer for COVID-19 symptom extraction with the following key features: low processor and memory utilization, faster runtime, and similar weighted microaverage and macroaverage measures for precision, recall, and f1-score compared to industry-standard annotation systems.

Abstract IntroductionRacially and ethnically diverse minorities often experience the disease burden of sexually transmitted infections or diseases (STD) more often than their White counterparts. Yet, little is known about the connection of STD systematic discrimination, racism, and social and behavioral determinants. Plus, little to no details exists related to how this information is recorded in their Electronic Health Records (EHRs). The objective of this study is to assess the completeness of social and behavioral determinants of health (SDOH) data in the EHRs of a minority cohort with STD.Materials and Methods2,993 minority patients diagnosed with a STD at the Mayo Clinic were identified for this study. A natural language processing (NLP) algorithm was applied on the Patient-Provided Information (PPI) in their EHRs to extract SDOH information in six domains that are associated with STD, namely alcohol use, substance use, sexual activity, sexual orientation, housing status, and employment status. The completeness of SDOH was assessed in terms of documentation, breadth, and density.ResultsOur study indicates that nearly half of 2,993 patients did not have SDOH-related PPI records in their EHRs whereas the patients who had SDOH-related PPI records had well-documented records for five out of six SDOH domains, including alcohol use, substance use, sexual activity, housing status, and employment status, except for sexual orientation. A total of 1,504 patients had PPI in their EHRs for at least one of six SDOH domains, which is about 50.3% of the study cohort. Most SDOH domains have a short time span of 1 year, with up to 18 years of record data. Our analysis also indicated that education and age have a significant impact on the recording of SDOH-related PPI records. Patients that are female, older, and higher educated tend to have more SDOH information available in their records.Discussion and ConclusionWe assessed the completeness of SDOH information recorded in the PPI from patients’ EHRs. Due to large amounts of missing SDOH information in the PPI, future research is needed to integrate accurate and robust SDOH related data for downstream research and the impact of systematic discrimination on how this information is collected and interpreted in the EHRs.