Fu, Sunyang

Abstract Internal validation is the most popular evaluation strategy used for drug–target predictive models. The simple random shuffling in the cross-validation, however, is not always ideal to handle large, diverse and copious datasets as it could potentially introduce bias. Hence, these predictive models cannot be comprehensively evaluated to provide insight into their general performance on a variety of use-cases (e.g. permutations of different levels of connectiveness and categories in drug and target space, as well as validations based on different data sources). In this work, we introduce a benchmark, BETA, that aims to address this gap by (i) providing an extensive multipartite network consisting of 0.97 million biomedical concepts and 8.5 million associations, in addition to 62 million drug–drug and protein–protein similarities and (ii) presenting evaluation strategies that reflect seven cases (i.e. general, screening with different connectivity, target and drug screening based on categories, searching for specific drugs and targets and drug repurposing for specific diseases), a total of seven Tests (consisting of 344 Tasks in total) across multiple sampling and validation strategies. Six state-of-the-art methods covering two broad input data types (chemical structure- and gene sequence-based and network-based) were tested across all the developed Tasks. The best-worst performing cases have been analyzed to demonstrate the ability of the proposed benchmark to identify limitations of the tested methods for running over the benchmark tasks. The results highlight BETA as a benchmark in the selection of computational strategies for drug repurposing and target discovery.

AbstractSummaryBuilding a high-quality annotation corpus requires expenditure of considerable time and expertise, particularly for biomedical and clinical research applications. Most existing annotation tools provide many advanced features to cover a variety of needs where the installation, integration and difficulty of use present a significant burden for actual annotation tasks. Here, we present MedTator, a serverless annotation tool, aiming to provide an intuitive and interactive user interface that focuses on the core steps related to corpus annotation, such as document annotation, corpus summarization, annotation export and annotation adjudication.Availability and implementationMedTator and its tutorial are freely available from https://ohnlp.github.io/MedTator. MedTator source code is available under the Apache 2.0 license: https://github.com/OHNLP/MedTator.Supplementary informationSupplementary data are available at Bioinformatics online.

Background: Real world studies of silent brain infarction (SBI) and white matter disease (WMD) are impeded by challenges in cohort identification. Natural language processing (NLP) from imaging reports may facilitate future studies. However, electronic health records can be heterogeneous and the process of interpreting neuroimages and generating reports can vary. Understanding knowledge representation and relationships between neuroimages and imaging reports is crucial for using NLP to facilitate disease management and cohort identification. Methods: A balanced sample of head neuroimages (CT, MRI) of patients >50 years without clinical histories of symptomatic stroke, TIA, or dementia were obtained at Mayo Clinic and Tufts Medical Center. A team of 4 radiology residents performed report interpretation (RI) on 1000 reports according to a standardized protocol for the presence of SBIs, the presence of WMD, and WMD grade. A random subsample of 400 was doubly read for interrater reliability. For benchmarking, a team of 4 neuroradiologists directly reviewed and described findings on a subsample of 182 images, each doubly read. We assessed interrater reliability for direct review (DR) and RI, and agreement between these 2 information sources. An NLP algorithm was developed to review and extract findings from 1000 imaging reports. Results: For DR, interrater reliability was moderate for SBIs and WMD (k = 0.53, 95% CI 0.43-0.64 and k = 0.47, 95% CI 0.33-0.61) and good for WMD grade (Spearman 0.71, p<0.001). For RI, interrater reliability for SBIs, WMD and WMD grade was good (k = 0.88, 95% CI 0.80-0.97; k = 0.98, 95% CI 0.97-1.00; and Spearman = 0.985, p<0.001, respectively). Agreement between DR and RI was good for SBIs (k = 0.77, 95% CI 0.67-0.86) and WMD (k = 0.65, 95% CI 0.54-0.77). Spearman rank correlations comparing WMD grade interpretation DR to RI was 0.60 (p<0.001). In identifying the presence of SBIs and WMDs, the accuracy of the NLP algorithm was 0.991 and 0.994, respectively. Conclusion: For the presence of SBI and WMD, and WMD grade, agreement between RI and DR was similar to agreement between 2 neuroradiologists directly reviewing neuroimages. It is highly feasible to use NLP to identify patients with SBIs and WMDs for clinical studies.

Background . Patients increasingly use asynchronous communication platforms to converse with care teams. Natural language processing (NLP) to classify content and automate triage of these messages has great potential to enhance clinical efficiency. We characterize the contents of a corpus of portal messages generated by patients using NLP methods. We aim to demonstrate descriptive analyses of patient text that can contribute to the development of future sophisticated NLP applications. Methods . We collected approximately 3,000 portal messages from the cardiology, dermatology, and gastroenterology departments at Mayo Clinic. After labeling these messages as either Active Symptom, Logistical, Prescription, or Update, we used NER (named entity recognition) to identify medical concepts based on the UMLS library. We hierarchically analyzed the distribution of these messages in terms of departments, message types, medical concepts, and keywords therewithin. Results . Active Symptom and Logistical content types comprised approximately 67% of the message cohort. The “Findings” medical concept had the largest number of keywords across all groupings of content types and departments. “Anatomical Sites” and “Disorders” keywords were more prevalent in Active Symptom messages, while “Drugs” keywords were most prevalent in Prescription messages. Logistical messages tended to have the lower proportions of “Anatomical Sites,”, “Disorders,”, “Drugs,”, and “Findings” keywords when compared to other message content types. Conclusions . This descriptive corpus analysis sheds light on the content and foci of portal messages. The insight into the content and differences among message themes can inform the development of more robust NLP models.

Background: White matter disease (WMD) and silent brain infarction (SBI) are known to be risk markers for stroke. Nevertheless, the predictive value of these changes when seen incidentally on routinely-obtained neuroimages is unknown. Methods: In this retrospective cohort study, Kaiser Permanente-Southern California health plan enrollees aged ≥ 50 years old with a brain CT or MRI scan between 2009-2019 and without a prior history of ischemic stroke, transient ischemic attack, or dementia were identified. Natural language processing (NLP) was used to identify patients with SBI and WMD on the index neuroimaging report. We used Cox proportional hazards to estimate the risk of future ischemic stroke associated with the presence of SBI and of WMD, controlling for major stroke risk factors. Results: Among 262,875 individuals receiving brain neuroimaging, 13,154 (5.0%) and 78,330 (29.8%) had SBI and WMD, respectively. The Table below summarizes the crude stroke incidence rates. The crude hazard ratio (HR) was 3.40 (95% CI 3.25-3.56) for SBI and 2.63 (95% CI 2.54-2.71) for WMD. In the multivariable model controlling for all major stroke risk factors, the effect of SBI was found to be stronger in younger versus older patients and for MRI- versus CT-discovered lesions. With MRI, the average adjusted HR over time was 2.95 (95% CI 2.53-3.44) for those < age 65 and 2.15 (95% CI 1.91-2.41) for those ≥ age 65. With CT scan, the average adjusted HR over time was 2.48 (95%CI 2.19-2.81) for those < age 65 and 1.81 (95% CI 1.71-1.91) for those ≥ age 65. The adjusted HR associated with a finding of WMD was 1.76 (95% CI 1.69-1.82) and was not modified by age or imaging modality. The effect of SBI decreased gradually over time, while the effect of WMD remained constant. Conclusion: Incidentally-discovered SBI and WMD are common in patients ≥ age 50 and are associated with substantial increases in the risk of subsequent symptomatic stroke. The findings may represent an opportunity for stroke prevention.

Background: White matter disease (WMD) and silent brain infarction (SBI) are known to be risk markers for dementia. Nevertheless, the effects of incidentally-discovered WMD and SBI on routinely-obtained neuroimages is unknown. Methods: In the retrospective cohort study, Kaiser Permanente-Southern California (KPSC) health plan enrollees aged ≥ 50 years old with a brain cross-sectional image (CT or MRI) between 2009-2019 and without a prior history of ischemic stroke, transient ischemic attack, or dementia were identified. Natural language processing (NLP) was used to identify patients with SBI and WMD on the first qualifying neuroimaging report. Cox proportional hazards regression model was applied to estimate the effects of WMD and SBI on dementia, controlling for major cardiovascular risk factors. Results: Among 262,875 individuals receiving brain neuroimaging, 13,154 (5.0%) and 78,330 (29.8%) had SBI and WMD, respectively. The table below summarizes the crude dementia incidence rates. The crude hazard ratio (HR) was 3.69 (95% CI 3.58-3.80) for WMD and 2.80 (95% CI 2.68-2.93) for SBI. In the multivariable model controlling for all major cardiovascular risk factors, the effect of WMD on dementia risk was found to be stronger in younger compared to older patients and for CT- compared to MRI-discovered lesions. With CT, the adjusted HR was 3.08 (95% CI 2.84-3.34) for those < age 70 and 1.90 (95% CI 1.83-1.97) for those ≥ age 70. With MRI, the adjusted HR was 2.58 (95% CI 2.31-2.87) for those < age 70 and 1.59 (95% CI 1.45-1.75) for those ≥ age 70. The adjusted HR associated with a finding of SBI was 2.01 (95% CI 1.78-2.32) for patients age <70 and 1.26 (95% CI 1.20-1.32) for patients age ≥70, and was not modified by imaging modality. Conclusion: Incidentally-discovered WMD and SBI are common in patients ≥ age 50 and are associated with substantial increase in the risk of subsequent dementia. The findings may represent an opportunity for dementia prevention.

Introduction: Silent brain infarcts (SBIs) and white matter disease (WMD) are highly prevalent and associated with increased risk of ischemic stroke in patients with traditional stroke risk factors (RFs) in prospective cohort studies. Their frequency and associations with stroke RFs have not been well described in real world populations. Methods: This was a cross-sectional study of patients age ≥ 50 in the Kaiser Permanente-Southern California (KPSC) health system between 2009-2019 with a head CT or MRI for non-stroke indications and no history of ischemic stroke, transient ischemic attack, or dementia. A natural language processing (NLP) algorithm developed at Mayo Clinic and Tufts Medical Center was applied to the KPSC EHR to identify individuals with reported SBIs or WMD. Multivariable Poisson regression with robust error variance was used to estimate risk ratios of demographics, stroke RFs (from the Framingham Stroke Risk Score), and scan modality on the presence of SBIs or WMD. Results: Among 262,875 individuals, the NLP identified 13,154 (5.0%) with SBIs and 78,330 (29.8%) with WMD. Stroke RFs were highly prevalent in this cohort. The majority underwent CTs (74.8%) instead of MRIs as their initial neuroimaging. After adjustment for demographics and RFs, advanced age demonstrated a strong association with increased risk of SBIs and WMD (table). MRI was associated with a reduced risk of reported SBIs (ARR: 0.87, 95% CI 0.83-0.91) and an increased risk of reported WMD (ARR 2.86, 95% CI 2.83-2.90). Despite being prevalent, traditional stroke RFs had weak associations with increased risk of SBIs or increased risk of WMD. Conclusions: Advanced age is strongly associated with incidentally discovered SBIs and WMD on neuroimaging studies obtained in routine care. The development of SBIs and WMD may not be fully attributable to traditional stroke RFs.

Abstract Background Delirium is underdiagnosed in clinical practice and is not routinely coded for billing. Manual chart review can be used to identify the occurrence of delirium; however, it is labor-intensive and impractical for large-scale studies. Natural language processing (NLP) has the capability to process raw text in electronic health records (EHRs) and determine the meaning of the information. We developed and validated NLP algorithms to automatically identify the occurrence of delirium from EHRs. Methods This study used a randomly selected cohort from the population-based Mayo Clinic Biobank (N = 300, age ≥65). We adopted the standardized evidence-based framework confusion assessment method (CAM) to develop and evaluate NLP algorithms to identify the occurrence of delirium using clinical notes in EHRs. Two NLP algorithms were developed based on CAM criteria: one based on the original CAM (NLP-CAM; delirium vs no delirium) and another based on our modified CAM (NLP-mCAM; definite, possible, and no delirium). The sensitivity, specificity, and accuracy were used for concordance in delirium status between NLP algorithms and manual chart review as the gold standard. The prevalence of delirium cases was examined using International Classification of Diseases, 9th Revision (ICD-9), NLP-CAM, and NLP-mCAM. Results NLP-CAM demonstrated a sensitivity, specificity, and accuracy of 0.919, 1.000, and 0.967, respectively. NLP-mCAM demonstrated sensitivity, specificity, and accuracy of 0.827, 0.913, and 0.827, respectively. The prevalence analysis of delirium showed that the NLP-CAM algorithm identified 12 651 (9.4%) delirium patients, the NLP-mCAM algorithm identified 20 611 (15.3%) definite delirium cases, and 10 762 (8.0%) possible cases. Conclusions NLP algorithms based on the standardized evidence-based CAM framework demonstrated high performance in delineating delirium status in an expeditious and cost-effective manner.

Background: The focus of most existing phenotyping algorithms based on electronic health record (EHR) data has been to accurately identify cases and non-cases of specific diseases. However, a more challenging task is to accurately identify disease incidence, as identifying the first occurrence of disease is more important for efficient and valid clinical and epidemiological research. Moreover, stroke is a challenging phenotype due to diagnosis difficulty and common miscoding. This task generally requires utilization of multiple types of EHR data (e.g., diagnoses and procedure codes, unstructured clinical notes) and a more robust algorithm integrating both natural language processing and machine learning. In this study, we developed and validated an EHR-based classifier to accurately identify stroke incidence among a cohort of atrial fibrillation (AF) patients Methods: We developed a stroke phenotyping algorithm using International Classification of Diseases, Ninth Revision (ICD-9) codes, Current Procedural Terminology (CPT) codes, and expert-provided keywords as model features. Structured data was extracted from Rochester Epidemiology Project (REP) database. Natural Language Processing (NLP) was used to extract and validate keyword occurrence in clinical notes. A window of ±30 days was considered when including/excluding keywords/codes into the input vector. Frequencies of keywords/codes were used as input feature sets for model training. Multiple competing models were trained using various combinations of feature sets and two machine learning algorithms: logistic regression and random forest. Training data were provided by two nurse abstractors and included validated stroke incidences from a previously established atrial fibrillation cohort. Precision, recall, and F-score of the algorithm were calculated to assess and compare model performances. Results: Among 4,914 patients with atrial fibrillation, 1,773 patients were screened. 3,141 patients had no stroke-related codes or keywords and were presumed to be free of stroke during follow-up. Among the screened patients, 740 had validated strokes and 1,033 did not have a stroke based on review of the EHR by trained nurse abstractors. The best performing stroke incidence phenotyping classifier utilized Keywords+ICD-9+CPT features using a random forest classifier, achieving a precision of 0.942, recall of 0.943, and F-score of 0.943. Conclusion: In conclusion, we developed and validated a stroke algorithm that performed well for identifying stroke incidence in an enriched population (AF cohort), which extends beyond the typical binary case/non-case stroke identification problem. Future work will involve testing the generalizability of this algorithm in a general population.