Citations

Abstract Objective Transformer-based language models are prevailing in the clinical domain due to their excellent performance on clinical NLP tasks. The generalizability of those models is usually ignored during the model development process. This study evaluated the generalizability of CancerBERT, a Transformer-based clinical NLP model, along with classic machine learning models, i.e., conditional random field (CRF), bi-directional long short-term memory CRF (BiLSTM-CRF), across different clinical institutes through a breast cancer phenotype extraction task. Materials and methods Two clinical corpora of breast cancer patients were collected from the electronic health records from the University of Minnesota (UMN) and Mayo Clinic (MC), and annotated following the same guideline. We developed three types of NLP models (i.e., CRF, BiLSTM-CRF and CancerBERT) to extract cancer phenotypes from clinical texts. We evaluated the generalizability of models on different test sets with different learning strategies (model transfer vs locally trained). The entity coverage score was assessed with their association with the model performances. Results We manually annotated 200 and 161 clinical documents at UMN and MC. The corpora of the two institutes were found to have higher similarity between the target entities than the overall corpora. The CancerBERT models obtained the best performances among the independent test sets from two clinical institutes and the permutation test set. The CancerBERT model developed in one institute and further fine-tuned in another institute achieved reasonable performance compared to the model developed on local data (micro-F1: 0.925 vs 0.932). Conclusions The results indicate the CancerBERT model has superior learning ability and generalizability among the three types of clinical NLP models for our named entity recognition task. It has the advantage to recognize complex entities, e.g., entities with different labels.

Abstract Despite recent methodology advancements in clinical natural language processing (NLP), the adoption of clinical NLP models within the translational research community remains hindered by process heterogeneity and human factor variations. Concurrently, these factors also dramatically increase the difficulty in developing NLP models in multi-site settings, which is necessary for algorithm robustness and generalizability. Here, we reported on our experience developing an NLP solution for Coronavirus Disease 2019 (COVID-19) signs and symptom extraction in an open NLP framework from a subset of sites participating in the National COVID Cohort (N3C). We then empirically highlight the benefits of multi-site data for both symbolic and statistical methods, as well as highlight the need for federated annotation and evaluation to resolve several pitfalls encountered in the course of these efforts.

Abstract Objective Social determinants of health (SDoH) play critical roles in health outcomes and well-being. Understanding the interplay of SDoH and health outcomes is critical to reducing healthcare inequalities and transforming a “sick care” system into a “health-promoting” system. To address the SDOH terminology gap and better embed relevant elements in advanced biomedical informatics, we propose an SDoH ontology (SDoHO), which represents fundamental SDoH factors and their relationships in a standardized and measurable way. Material and Methods Drawing on the content of existing ontologies relevant to certain aspects of SDoH, we used a top-down approach to formally model classes, relationships, and constraints based on multiple SDoH-related resources. Expert review and coverage evaluation, using a bottom-up approach employing clinical notes data and a national survey, were performed. Results We constructed the SDoHO with 708 classes, 106 object properties, and 20 data properties, with 1,561 logical axioms and 976 declaration axioms in the current version. Three experts achieved 0.967 agreement in the semantic evaluation of the ontology. A comparison between the coverage of the ontology and SDOH concepts in 2 sets of clinical notes and a national survey instrument also showed satisfactory results. Discussion SDoHO could potentially play an essential role in providing a foundation for a comprehensive understanding of the associations between SDoH and health outcomes and paving the way for health equity across populations. Conclusion SDoHO has well-designed hierarchies, practical objective properties, and versatile functionalities, and the comprehensive semantic and coverage evaluation achieved promising performance compared to the existing ontologies relevant to SDoH.

Abstract Background: Clinical information retrieval (IR) plays a vital role in modern healthcare by facilitating efficient access and analysis of medical literature for clinicians and researchers. This scoping review aims to offer a comprehensive overview of the current state of clinical IR research and identify gaps and potential opportunities for future studies in this field. Objectives: The main objective of this review is to identify and analyze published research on clinical IR, including the methods, techniques, and tools used to retrieve and analyze clinical information from various sources. We aim to provide a comprehensive overview of the current state of clinical IR research and guide future research efforts in this field. Methods: We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines and conducted a thorough search of multiple databases, including Ovid Embase, Ovid Cochrane Central Register of Controlled Trials, Ovid Cochrane Database of Systematic Reviews, Scopus, ACM Digital Library, IEEE Xplore, and Web of Science, from January 1st, 2012, to January 4th, 2023. The screening process involved multiple reviewers, and we included 184 papers for the final review. Results: We conducted a detailed analysis and discussion of various aspects of clinical IR research, including publication year, data sources, methods, techniques, evaluation metrics, shared tasks, and applications. Our analysis revealed key research areas in clinical IR, such as indexing, ranking, and query expansion, and identified opportunities for future research in these areas.

Multiplex immunofluorescence (MxIF) images provide detailed information of cell composition and spatial context for biomedical research. However, compromised data quality could lead to research biases. Comprehensive image quality checking (QC) is essential for reliable downstream analysis. As a reliable and specific staining of cell nuclei, 4′,6-diamidino-2-phenylindole (DAPI) signals were used as references for tissue localization and auto-focusing across MxIF staining–scanning–bleaching iterations and could potentially be reused for QC. To confirm the feasibility of using DAPI as QC reference, pixel-level DAPI values were extracted to calculate signal fluctuations and tissue content similarities in staining–scanning–bleaching iterations for identifying quality issues. Concordance between automatic quantification and human experts’ annotations were evaluated on a data set consisting of 348 fields of view (FOVs) with 45 immune and tumor cell markers. Cell distribution differences between subsets of QC-pass vs QC-failed FOVs were compared to investigate the downstream effects. Results showed that 87.3% FOVs with tissue damage and 73.4% of artifacts were identified. QC-failed FOVs showed elevated regional gathering in cellular feature space compared with the QC-pass FOVs. Our results supported that DAPI signals could be used as references for MxIF image QC, and low-quality FOVs identified by our method must be cautiously considered for downstream analyses.

Importance The effectiveness of triplet therapy compared with androgen pathway inhibitor (API) doublets in a heterogeneous patient population with metastatic castration-sensitive prostate cancer (mCSPC) is unknown. Objective To assess the comparative effectiveness of contemporary systemic treatment options for patients with mCSPC across clinically relevant subgroups. Data Sources For this systematic review and meta-analysis, Ovid MEDLINE and Embase were searched from each database’s inception (MEDLINE, 1946; Embase, 1974) through June 16, 2021. Subsequently, a “living” auto search was created with weekly updates to identify new evidence as it became available. Study Selection Phase 3 randomized clinical trials (RCTs) assessing first-line treatment options for mCSPC. Data Extraction and Synthesis Two independent reviewers extracted data from eligible RCTs. The comparative effectiveness of different treatment options was assessed with a fixed-effect network meta-analysis. Data were analyzed on July 10, 2022. Main Outcomes and Measures Outcomes of interest included overall survival (OS), progression-free survival (PFS), grade 3 or higher adverse events, and health-related quality of life. Results This report included 10 RCTs with 11 043 patients and 9 unique treatment groups. Median ages of the included population ranged from 63 to 70 years. Current evidence for the overall population suggests that the darolutamide (DARO) triplet (DARO + docetaxel [D] + androgen deprivation therapy [ADT]; hazard ratio [HR], 0.68; 95% CI, 0.57-0.81), as well as the abiraterone (AAP) triplet (AAP + D + ADT; HR, 0.75; 95% CI, 0.59-0.95), are associated with improved OS compared with D doublet (D + ADT) but not compared with API doublets. Among patients with high-volume disease, AAP + D + ADT may improve OS compared with D + ADT (HR, 0.72; 95% CI, 0.55-0.95) but not compared with AAP + ADT, enzalutamide (E) + ADT, and apalutamide (APA) + ADT. For patients with low-volume disease, AAP + D + ADT may not improve OS compared with APA + ADT, AAP + ADT, E + ADT, and D + ADT. Conclusions and Relevance The potential benefit observed with triplet therapy must be interpreted with careful accounting for the volume of disease and the choice of doublet comparisons used in the clinical trials. These findings suggest an equipoise to how triplet regimens compare with API doublet combinations and provide direction for future clinical trials.

Abstract Introduction: Despite the decline in overall mortality and incidence of cancer in the US population, disparities in cancer care still largely exist within certain groups. The proportion of racial and ethnic minorities recruited to participate in cancer research is persistently lower than the US population. The rapid adoption of electronic health records (EHRs) systems has enabled computational cohort identification and eligibility screening for facilitating cancer research. However, EHR data is known to suffer from data quality issues due to numerous factors, such as care access. Patients with limited care access due to low socioeconomic status would naturally be sparsely represented in EHR data relative to those of higher status. Prospective enrollees may be disqualified due to biased or incomplete EHR data at the screening stage. Methods: To systematically identify data quality variabilities in EHRs, we implemented an information score (i-score) to reflect the density, variability, and irregularity of patients’ data. The pattern can be measured by estimating the variability of the time gaps between observations. For a patient observed n times, the encountered observation times are represented as x(1)…x(n). The relative time interval g for each observed encounter can be defined as g1 = [x(i+1)-x(i)]/[x(n)-x(1)], for I = 1…n - 1. The average amount of information of each observation is then defined as I, between 0 and 1, calculated as: I = 2/n+(n-2)/n[1-sqrt((n-1)Var{gi;i=1…n-1})]. An equality-spaced observation would receive a high i-score. The lower the i-score, the higher the data irregularity. We conducted a case study in a cohort of de novo stage IV breast cancer (DNIV) assembled by a previously validated phenotyping algorithm (Wang AACR 2020). The cohort contains 1,918 DNIV cases between 2004 and 2018 at Mayo Clinic Rochester. We used the i-score to assess patient visits, breast cancer diagnosis codes, and narrative documentation, achieved by the extraction of stage, surgery, de novo, and metastasis information based on the phenotyping definitions. The summary statistics of the i-scores were compared among four racial groups: African American (3%), Asian (1%), Caucasian (87%), and Other (9%). ANOVA F test was used to compare the group differences. Results and Conclusion: The mean i-scores for visit, diagnosis and narrative documentation were 0.712, 0.200, and 0.056 for African American, 0.412, 0.301, and 0.073 for Asian, 0.438, 0.278, and 0.056 for Caucasian, and 0.519, 0.343, and 0.043 for the other category, respectively. The F test indicated significant differences between visit (p<0.001) and diagnosis (p=0.0267) data. No difference was found in narrative documentation (p=0. 8591). In conclusion, there was a substantial i-score variation discovered, especially within the African American group, suggesting a strong need to address EHR quality disparity for cancer research. Citation Format: Sunyang Fu, Liwei Wang, Folakemi T. Odedina, Hongfang Liu. Assessment of EHR data quality variabilities among different racial groups in the cohort of de novo stage IV breast cancer [abstract]. In: Proceedings of the 15th AACR Conference on the Science of Cancer Health Disparities in Racial/Ethnic Minorities and the Medically Underserved; 2022 Sep 16-19; Philadelphia, PA. Philadelphia (PA): AACR; Cancer Epidemiol Biomarkers Prev 2022;31(1 Suppl):Abstract nr A020.