Fu, Sunyang

Background: Multiple algorithms with variable performance have been developed to identify dementia using combinations of billing codes and medication data that are widely available from electronic health records (EHR). If the characteristics of misclassified patients are clearly identified, modifying existing algorithms to improve performance may be possible. Objective: To examine the performance of a code-based algorithm to identify dementia cases in the population-based Mayo Clinic Study of Aging (MCSA) where dementia diagnosis (i.e., reference standard) is actively assessed through routine follow-up and describe the characteristics of persons incorrectly categorized. Methods: There were 5,316 participants (age at baseline (mean (SD)): 73.3 (9.68) years; 50.7% male) without dementia at baseline and available EHR data. ICD-9/10 codes and prescription medications for dementia were extracted between baseline and one year after an MCSA dementia diagnosis or last follow-up. Fisher’s exact or Kruskal-Wallis tests were used to compare characteristics between groups. Results: Algorithm sensitivity and specificity were 0.70 (95% CI: 0.67, 0.74) and 0.95 (95% CI: 0.95, 0.96). False positives (i.e., participants falsely diagnosed with dementia by the algorithm) were older, with higher Charlson comorbidity index, more likely to have mild cognitive impairment (MCI), and longer follow-up (versus true negatives). False negatives (versus true positives) were older, more likely to have MCI, or have more functional limitations. Conclusions: We observed a moderate-high performance of the code-based diagnosis method against the population-based MCSA reference standard dementia diagnosis. Older participants and those with MCI at baseline were more likely to be misclassified.

Abstract Despite recent methodology advancements in clinical natural language processing (NLP), the adoption of clinical NLP models within the translational research community remains hindered by process heterogeneity and human factor variations. Concurrently, these factors also dramatically increase the difficulty in developing NLP models in multi-site settings, which is necessary for algorithm robustness and generalizability. Here, we reported on our experience developing an NLP solution for Coronavirus Disease 2019 (COVID-19) signs and symptom extraction in an open NLP framework from a subset of sites participating in the National COVID Cohort (N3C). We then empirically highlight the benefits of multi-site data for both symbolic and statistical methods, as well as highlight the need for federated annotation and evaluation to resolve several pitfalls encountered in the course of these efforts.

Abstract Objective Social determinants of health (SDoH) play critical roles in health outcomes and well-being. Understanding the interplay of SDoH and health outcomes is critical to reducing healthcare inequalities and transforming a “sick care” system into a “health-promoting” system. To address the SDOH terminology gap and better embed relevant elements in advanced biomedical informatics, we propose an SDoH ontology (SDoHO), which represents fundamental SDoH factors and their relationships in a standardized and measurable way. Material and Methods Drawing on the content of existing ontologies relevant to certain aspects of SDoH, we used a top-down approach to formally model classes, relationships, and constraints based on multiple SDoH-related resources. Expert review and coverage evaluation, using a bottom-up approach employing clinical notes data and a national survey, were performed. Results We constructed the SDoHO with 708 classes, 106 object properties, and 20 data properties, with 1,561 logical axioms and 976 declaration axioms in the current version. Three experts achieved 0.967 agreement in the semantic evaluation of the ontology. A comparison between the coverage of the ontology and SDOH concepts in 2 sets of clinical notes and a national survey instrument also showed satisfactory results. Discussion SDoHO could potentially play an essential role in providing a foundation for a comprehensive understanding of the associations between SDoH and health outcomes and paving the way for health equity across populations. Conclusion SDoHO has well-designed hierarchies, practical objective properties, and versatile functionalities, and the comprehensive semantic and coverage evaluation achieved promising performance compared to the existing ontologies relevant to SDoH.

Abstract Introduction: Despite the decline in overall mortality and incidence of cancer in the US population, disparities in cancer care still largely exist within certain groups. The proportion of racial and ethnic minorities recruited to participate in cancer research is persistently lower than the US population. The rapid adoption of electronic health records (EHRs) systems has enabled computational cohort identification and eligibility screening for facilitating cancer research. However, EHR data is known to suffer from data quality issues due to numerous factors, such as care access. Patients with limited care access due to low socioeconomic status would naturally be sparsely represented in EHR data relative to those of higher status. Prospective enrollees may be disqualified due to biased or incomplete EHR data at the screening stage. Methods: To systematically identify data quality variabilities in EHRs, we implemented an information score (i-score) to reflect the density, variability, and irregularity of patients’ data. The pattern can be measured by estimating the variability of the time gaps between observations. For a patient observed n times, the encountered observation times are represented as x(1)…x(n). The relative time interval g for each observed encounter can be defined as g1 = [x(i+1)-x(i)]/[x(n)-x(1)], for I = 1…n - 1. The average amount of information of each observation is then defined as I, between 0 and 1, calculated as: I = 2/n+(n-2)/n[1-sqrt((n-1)Var{gi;i=1…n-1})]. An equality-spaced observation would receive a high i-score. The lower the i-score, the higher the data irregularity. We conducted a case study in a cohort of de novo stage IV breast cancer (DNIV) assembled by a previously validated phenotyping algorithm (Wang AACR 2020). The cohort contains 1,918 DNIV cases between 2004 and 2018 at Mayo Clinic Rochester. We used the i-score to assess patient visits, breast cancer diagnosis codes, and narrative documentation, achieved by the extraction of stage, surgery, de novo, and metastasis information based on the phenotyping definitions. The summary statistics of the i-scores were compared among four racial groups: African American (3%), Asian (1%), Caucasian (87%), and Other (9%). ANOVA F test was used to compare the group differences. Results and Conclusion: The mean i-scores for visit, diagnosis and narrative documentation were 0.712, 0.200, and 0.056 for African American, 0.412, 0.301, and 0.073 for Asian, 0.438, 0.278, and 0.056 for Caucasian, and 0.519, 0.343, and 0.043 for the other category, respectively. The F test indicated significant differences between visit (p<0.001) and diagnosis (p=0.0267) data. No difference was found in narrative documentation (p=0. 8591). In conclusion, there was a substantial i-score variation discovered, especially within the African American group, suggesting a strong need to address EHR quality disparity for cancer research. Citation Format: Sunyang Fu, Liwei Wang, Folakemi T. Odedina, Hongfang Liu. Assessment of EHR data quality variabilities among different racial groups in the cohort of de novo stage IV breast cancer [abstract]. In: Proceedings of the 15th AACR Conference on the Science of Cancer Health Disparities in Racial/Ethnic Minorities and the Medically Underserved; 2022 Sep 16-19; Philadelphia, PA. Philadelphia (PA): AACR; Cancer Epidemiol Biomarkers Prev 2022;31(1 Suppl):Abstract nr A020.

Background Covert cerebrovascular disease (CCD) has been shown to be associated with dementia in population‐based studies with magnetic resonance imaging (MRI) screening, but dementia risk associated with incidentally discovered CCD is not known. Methods and Results Individuals aged ≥50 years enrolled in the Kaiser Permanente Southern California health system receiving head computed tomography (CT) or MRI for nonstroke indications from 2009 to 2019, without prior ischemic stroke/transient ischemic attack, dementia/Alzheimer disease, or visit reason/scan indication suggestive of cognitive decline or stroke were included. Natural language processing identified incidentally discovered covert brain infarction (id‐CBI) and white matter disease (id‐WMD) on the neuroimage report; white matter disease was characterized as mild, moderate, severe, or undetermined. We estimated risk of dementia associated with id‐CBI and id‐WMD. Among 241 050 qualified individuals, natural language processing identified 69 931 (29.0%) with id‐WMD and 11 328 (4.7%) with id‐CBI. Dementia incidence rates (per 1000 person‐years) were 23.5 (95% CI, 22.9–24.0) for patients with id‐WMD, 29.4 (95% CI, 27.9–31.0) with id‐CBI, and 6.0 (95% CI, 5.8–6.2) without id‐CCD. The association of id‐WMD with future dementia was stronger in younger (aged <70 years) versus older (aged ≥70 years) patients and for CT‐ versus MRI‐discovered lesions. For patients with versus without id‐WMD on CT, the adjusted HR was 2.87 (95% CI, 2.58–3.19) for younger and 1.87 (95% CI, 1.79–1.95) for older patients. For patients with versus without id‐WMD on MRI, the adjusted HR for dementia risk was 2.28 (95% CI, 1.99–2.62) for younger and 1.48 (95% CI, 1.32–1.66) for older patients. The adjusted HR for id‐CBI was 2.02 (95% CI, 1.70–2.41) for younger and 1.22 (95% CI, 1.15–1.30) for older patients for either modality. Dementia risk was strongly correlated with id‐WMD severity; adjusted HRs compared with patients who were negative for id‐WMD by MRI ranged from 1.41 (95% CI, 1.25–1.60) for those with mild disease on MRI to 4.11 (95% CI, 3.58–4.72) for those with severe disease on CT. Conclusions Incidentally discovered CCD is common and associated with a high risk of dementia, representing an opportunity for prevention. The association is strengthened when discovered at younger age, by increasing id‐WMD severity, and when id‐WMD is detected by CT scan rather than MRI.

AbstractAn increasing number of studies have reported using natural language processing (NLP) to assist observational research by extracting clinical information from electronic health records (EHRs). Currently, no standardized reporting guidelines for NLP‐assisted observational studies exist. The absence of detailed reporting guidelines may create ambiguity in the use of NLP‐derived content, knowledge gaps in the current research reporting practices, and reproducibility challenges. To address these issues, we conducted a scoping review of NLP‐assisted observational clinical studies and examined their reporting practices, focusing on NLP methodology and evaluation. Through our investigation, we discovered a high variation regarding the reporting practices, such as inconsistent use of references for measurement studies, variation in the reporting location (reference, appendix, and manuscript), and different granularity of NLP methodology and evaluation details. To promote the wide adoption and utilization of NLP solutions in clinical research, we outline several perspectives that align with the six principles released by the World Health Organization (WHO) that guide the ethical use of artificial intelligence for health.