Citations

335 Background: Systematic reviews are outdated quickly when the evidence is rapidly evolving as the process is laborious and there is little incentive for primary author team of an index SRMA to update the evidence. Consequently, there is an epidemic of redundant SRMAs performed by different teams—sometimes with conflicted results—for treatment of first line mRCC. Methods: We have created a living, interactive systematic review (LISR)and network meta-analysis(LINMA) for the treatment of first line mRCC using an Artificial intelligence (AI) assisted framework for evidence synthesis (Living, Interactive evidence synthesis framework) (LIvE) . The framework is implemented in five-layered architecture (application layer, shared module layer, core service layer, middleware layer, and storage layer) which work together to automate the identification of new studies and analysis and semi-automate the screening and data extraction. Dynamic features such as interactive tables, figures and evidence maps are enabled using Python and JavaScript programming languages. Results: We have maintained a living, interactive evidence profile for the first line treatment mRCC since September 2019 ( LIVING WEBSITE) . Living search strategy identifies new studies as they become available. As of October 13, 2020 LISR, includes data 14 clinical trials ( PRISMA ). Baseline characteristics are summarized in an interactive table ( TABLE) . Cabozantinib& Nivolumab (Cabo-Nivo) is the highest ranked drug for improving Overall Response (OR), Progression Free Survival (PFS) and Overall Survival (OS) whereas Ipilimumab in combination with Nivolumab (Ipi-Nivo) is highest ranked drug for achieving complete response (CR). Ipi-Nivo and Atezolizumab & Bevacizumab (Ate-Bev) ranked highest and Cabo-Nivo ranked lowest for treatment related Adverse events (TRAEs). Results of network meta-analysis are summarized as interactive tables and plots ( NMA ), summary of findings tables ( MULTIPLE COMAPRISONS ) and evidence maps ( MAP ). Conclusions: LISRs can potentially reduce redundancy, increase transparency, reproducibility, enable shared-decision making (at a guideline level, or in a patient-clinician dyad) and support living guidelines.

Abstract Background: DNIV accounts for 6%–10% of newly diagnosed breast cancer cases. Despite widespread mammography screening, its incidence is increasing in the United States and survival of this disease has only modestly improved since the late 1970s. As patient data accumulates in EHR, it’s promising to generate practice-based evidence through utilization of observational data sources. However, assembly of a DNIV cohort based on EHR data is challenging, as key pathologic and staging information are stored in unstructured clinical narratives and not available as structured data. In this study, we developed a rule-based algorithm to phenotype DNIV using natural language processing (NLP) techniques, and implemented the algorithm on our institutional EHR to extract potential DNIV cases. Methods and Results: We defined DNIV as those with either (1) M1 disease identified at time of initial presentation or M1 disease identified within 4 months after definitive surgery. We first developed a reference case list of DNIV verified by physician chart review of the EHR. We next refined the algorithm on a training dataset containing 51 positive and 38 negative reference cases. Next we tested the performance on the testing data containing 23 positive and 55 negative cases. The phenotyping algorithm identified key data elements using NLP, i.e., stage IV breast cancer, definitive surgery, stage 0-III, recurrent breast cancer and associated dates. To identify DNIV cases, phenotyping algorithm integrated temporal relations among the key data elements. The following steps were conducted in the following sequential order: (1) Identification of patients with breast cancer diagnosis using ICD-9 and ICD-10 codes. (2) Patients are positive cases if there are explicit mentions that delineate DNIV from recurrent metastatic breast cancer, such as “de novo stage IV” or “primary intact” detected by NLP at time of diagnosis. (3) Otherwise, patients with definitive surgery are selected if stage IV was within 5 years before definitive surgery or within 4 months after definitive surgery, along with patients with stage IV but without definitive surgery. (4) We further excluded patients with stage 0-III detected within 5 years after stage IV. (5) We further excluded patients with recurrent breast cancer detected before or at time of detection of stage IV. The remaining patients were left as positive cases. Precision of the algorithm was 70%, recall was 87% and F1, the weighted average of precision and recall, was 77%. We implemented our algorithm to interrogate 10 million clinical documents in a cohort of 56,548 patients with breast cancer diagnosis codes who presented to our institution between 2004 and 2018 and had research authorization. We identified 1918 potential DNIV cases. Conclusion: Our future focus is on algorithm refinement. An algorithm-generated cohort could serve as a data source for further study on outcomes related to DNIV and ideally as automated data abstractor and staging process. Citation Format: Liwei Wang, Karthik Giridhar, Kimberly Corbin, Brenda Ernst, Sadia Choudhery, Emmanuel Gabriel, Feichen Shen, Hongfang Liu, James Jakub. Identifying de novo stage IV breast cancer (DNIV) cases in Electronic Health Records (EHR) using natural language processing [abstract]. In: Proceedings of the AACR Virtual Special Conference on Artificial Intelligence, Diagnosis, and Imaging; 2021 Jan 13-14. Philadelphia (PA): AACR; Clin Cancer Res 2021;27(5_Suppl):Abstract nr PO-050.

Abstract Background: Next-generation sequencing provides comprehensive information about individuals’ genetic makeup and is commonplace in precision oncology practice. Due to the heterogeneity of individual patient’s disease conditions and treatment journeys, not all targeted therapies were initiated despite actionable mutations. To better understand and support the clinical decision-making process in precision oncology, there is a need to examine real-world associations of patients’ genetic information and treatment choice.Methods: To fill the gap of insufficient use of real-world data (RWD) in electronic health records (EHRs), we generated a single Resource Description Framework (RDF) resource, called PO2RDF (precision oncology to RDF) by integrating information regarding gene, variant, disease, and drug from genetic reports and EHRs. Results: There are total 2,309,014 triples contained in the PO2RDF. Among them 32,815 triples are related to Gene, 34,695 triples are related to Variant, 8,787 triples are related to Disease, 26,154 triples are related to Drug. We performed one use case analysis to demonstrate the usability of the PO2RDF: we examined real-world associations between EGFR mutations and targeted therapies to confirm existing knowledge and detect off-label use. Conclusions: In conclusion, our work proposed to use RDF to organize and distribute clinical RWD that is otherwise inaccessible externally. Our work serves as a pilot study that will lead to new clinical applications and could ultimately stimulate progress in the field of precision oncology.

Objective: To determine whether a bi-directional relationship exists between depression and HF within a single population of individuals receiving primary care services, using longitudinal electronic health records (EHRs). Methods: This retrospective cohort study utilized EHRs for adults who received primary care services within a large healthcare system in 2006. Validated EHR-based algorithms identified 10,649 people with depression (depression cohort) and 5,911 people with HF (HF cohort) between January 1, 2006 and December 31, 2018. Each person with depression or HF was matched 1:1 with an unaffected referent on age, sex, and outpatient service use. Each cohort (with their matched referents) was followed up electronically to identify newly diagnosed HF (in the depression cohort) and depression (in the HF cohort) that occurred after the index diagnosis of depression or HF, respectively. The risks of these outcomes were compared (vs. referents) using marginal Cox proportional hazard models adjusted for 16 comorbid chronic conditions. Results: 2,024 occurrences of newly diagnosed HF were observed in the depression cohort and 944 occurrences of newly diagnosed depression were observed in the HF cohort over approximately 4–6 years of follow-up. People with depression had significantly increased risk for developing newly diagnosed HF (HR 2.08, 95% CI 1.89–2.28) and people with HF had a significantly increased risk of newly diagnosed depression (HR 1.34, 95% CI 1.17–1.54) after adjusting for all 16 comorbid chronic conditions. Conclusion: These results provide evidence of a bi-directional relationship between depression and HF independently of age, sex, and multimorbidity from chronic illnesses.

Abstract Background: Next-generation sequencing provides comprehensive information about individuals’ genetic makeup and is commonplace in oncology clinical practice. However, the utility of genetic information in clinical decision-making process has not been examined extensively from a real-world, data-driven perspective. Through mining real-world data (RWD) from clinical notes, we could extract patients’ genetic information and further associate treatment decisions with genetic information.Methods: We proposed a real-world evidence (RWE) study framework that incorporates context-based natural language processing (NLP) methods and data quality examination before final association analysis. The framework was demonstrated in a Foundation-tested women cancer cohort (N=196). Upon retrieval of patients’ genetic information using NLP system, we assessed the completeness of genetic data captured in unstructured clinical notes according to a genetic data-model. We examined the distribution of different topics regarding BRCA1/2 throughout patients’ treatment process, and then analyzed the association between BRCA1/2 mutation status and the discussion/prescription of targeted therapy. Results: We identified seven topics in clinical context of genetic mentions including: Information, Evaluation, Insurance, Order, Negative, Positive, and Variants of unknown significance (VUS). Our rule-based system achieved a precision of 0.87, recall of 0.93 and F-measure of 0.91. Our machine learning system achieved a precision of 0.901, recall of 0.899 and F-measure of 0.9 for four-topic classification and a precision of 0.833, recall of 0.823 and F-measure of 0.82 for seven-topic classification. We found in result-containing sentences, capture of BRCA1/2 mutation information was 75%, but detailed variant information (e.g. variant types) is largely missing. Using cleaned RWD, significant associations were found between BRCA1/2 positive mutation and targeted therapies.Conclusions: In conclusion, we demonstrated a framework to generate RWE using RWD from different clinical sources. Rule-based NLP system achieved the best performance for resolving contextual variability when extracting RWD from unstructured clinical notes. Data quality issues such as incompleteness and discrepancies exist thus manual data cleaning is needed before further analysis can be performed. Finally, we were able to use cleaned RWD to evaluate real-world utility of genetic information to initiate a prescription of targeted therapy.

Abstract The development and implementation of clinical decision support (CDS) that trains itself and adapts its algorithms based on new data—here referred to as Adaptive CDS—present unique challenges and considerations. Although Adaptive CDS represents an expected progression from earlier work, the activities needed to appropriately manage and support the establishment and evolution of Adaptive CDS require new, coordinated initiatives and oversight that do not currently exist. In this AMIA position paper, the authors describe current and emerging challenges to the safe use of Adaptive CDS and lay out recommendations for the effective management and monitoring of Adaptive CDS.

Abstract Background Delirium is underdiagnosed in clinical practice and is not routinely coded for billing. Manual chart review can be used to identify the occurrence of delirium; however, it is labor-intensive and impractical for large-scale studies. Natural language processing (NLP) has the capability to process raw text in electronic health records (EHRs) and determine the meaning of the information. We developed and validated NLP algorithms to automatically identify the occurrence of delirium from EHRs. Methods This study used a randomly selected cohort from the population-based Mayo Clinic Biobank (N = 300, age ≥65). We adopted the standardized evidence-based framework confusion assessment method (CAM) to develop and evaluate NLP algorithms to identify the occurrence of delirium using clinical notes in EHRs. Two NLP algorithms were developed based on CAM criteria: one based on the original CAM (NLP-CAM; delirium vs no delirium) and another based on our modified CAM (NLP-mCAM; definite, possible, and no delirium). The sensitivity, specificity, and accuracy were used for concordance in delirium status between NLP algorithms and manual chart review as the gold standard. The prevalence of delirium cases was examined using International Classification of Diseases, 9th Revision (ICD-9), NLP-CAM, and NLP-mCAM. Results NLP-CAM demonstrated a sensitivity, specificity, and accuracy of 0.919, 1.000, and 0.967, respectively. NLP-mCAM demonstrated sensitivity, specificity, and accuracy of 0.827, 0.913, and 0.827, respectively. The prevalence analysis of delirium showed that the NLP-CAM algorithm identified 12 651 (9.4%) delirium patients, the NLP-mCAM algorithm identified 20 611 (15.3%) definite delirium cases, and 10 762 (8.0%) possible cases. Conclusions NLP algorithms based on the standardized evidence-based CAM framework demonstrated high performance in delineating delirium status in an expeditious and cost-effective manner.

Abstract Background : Next- generation sequencing provides comprehensive information about individuals’ genetic makeup and is commonplace in oncology clinical practice. However, the utility of genetic information in clinical decision-making process has not been examined extensively from a real-world, data-driven perspective. Through mining real-world data (RWD) from clinical notes, we could extract patients’ genetic information and further associate treatment decisions with genetic information. Methods : We proposed a real-world evidence (RWE) study framework that incorporates context-based natural language processing (NLP) methods and data quality examination before final association analysis. The framework was demonstrated in a Foundation-tested women cancer cohort (N=196). Upon retrieval of patients’ genetic information using NLP system, we assessed the completeness of genetic data captured in unstructured clinical notes according to a genetic data-model. We examined the distribution of different topics regarding BRCA1/2 throughout patients’ treatment process, and then analyzed the association between BRCA1/2 mutation status and the discussion/prescription of targeted therapy . Results: We identified seven topics in clinical context of genetic mentions including: Information, Evaluation, Insurance, Order, Negative, Positive, and Variants of unknown significance (VUS) . O ur rule-based system achieved a precision of 0.87, recall of 0.93 and F- measure of 0.91. Our machine learning system achieved a precision of 0.901, recall of 0.899 and F- measure of 0.9 for four-topic classification and a precision of 0.833, recall of 0.823 and F- measure of 0.82 for seven-topic classification. We found in result-containing sentences, capture of BRCA1/2 mutation information was 75%, but detailed variant information (e.g. variant types) is largely missing. Using cleaned RWD, significant associations were found between BRCA1/2 positive mutation and targeted therapies. Conclusions : In conclusion, we demonstrated a framework to generate RWE using RWD from different clinical sources. Rule-based NLP system achieved the best performance for resolving contextual variability when extracting RWD from unstructured clinical notes . Data quality issues such as incompleteness and discrepancies exist thus manual data cleaning is needed before further analysis can be performed. Finally, we were able to use cleaned RWD to evaluate real-world utility of genetic information to initiate a prescription of targeted therapy.

Abstract Objective Growing numbers of academic medical centers offer patient cohort discovery tools to their researchers, yet the performance of systems for this use case is not well understood. The objective of this research was to assess patient-level information retrieval methods using electronic health records for different types of cohort definition retrieval. Materials and Methods We developed a test collection consisting of about 100 000 patient records and 56 test topics that characterized patient cohort requests for various clinical studies. Automated information retrieval tasks using word-based approaches were performed, varying 4 different parameters for a total of 48 permutations, with performance measured using B-Pref. We subsequently created structured Boolean queries for the 56 topics for performance comparisons. In addition, we performed a more detailed analysis of 10 topics. Results The best-performing word-based automated query parameter settings achieved a mean B-Pref of 0.167 across all 56 topics. The way a topic was structured (topic representation) had the largest impact on performance. Performance not only varied widely across topics, but there was also a large variance in sensitivity to parameter settings across the topics. Structured queries generally performed better than automated queries on measures of recall and precision but were still not able to recall all relevant patients found by the automated queries. Conclusion While word-based automated methods of cohort retrieval offer an attractive solution to the labor-intensive nature of this task currently used at many medical centers, we generally found suboptimal performance in those approaches, with better performance obtained from structured Boolean queries. Future work will focus on using the test collection to develop and evaluate new approaches to query structure, weighting algorithms, and application of semantic methods.