Liu, Sijia

Abstract Despite recent methodology advancements in clinical natural language processing (NLP), the adoption of clinical NLP models within the translational research community remains hindered by process heterogeneity and human factor variations. Concurrently, these factors also dramatically increase the difficulty in developing NLP models in multi-site settings, which is necessary for algorithm robustness and generalizability. Here, we reported on our experience developing an NLP solution for Coronavirus Disease 2019 (COVID-19) signs and symptom extraction in an open NLP framework from a subset of sites participating in the National COVID Cohort (N3C). We then empirically highlight the benefits of multi-site data for both symbolic and statistical methods, as well as highlight the need for federated annotation and evaluation to resolve several pitfalls encountered in the course of these efforts.

Introduction: Prior AF screening trials demonstrated low yield, highlighting the need for more targeted approaches. An AI algorithm was developed to identify ECG signatures of AF risk during normal sinus rhythm, which has been validated in diverse external populations. Hypothesis: An AI-guided, targeted screening approach could improve the diagnosis of AF. Methods We conducted a pragmatic decentralized trial to prospectively recruit patients with stroke risk factors but no prior AF. The AI algorithm was applied to the ECGs performed in routine practice and divided patients into high- or low-risk groups. The primary endpoint was AF lasting ≥ 30 seconds on a subsequent 30-day continuous cardiac rhythm monitor. In a secondary analysis, trial participants were 1:1 propensity score-matched to real-world controls derived from the eligible but unenrolled population. Results A total of 1,003 patients from 40 U.S. states completed the study, with a mean age of 74 [SD 8.8] years. Over a mean of 22.3 days of continuous monitoring, AF was detected in 6 (1.6%) of low-risk patients and 48 (7.6%) of high-risk patients (OR 4.98 [2.11-11.75], p<0.001). Compared to usual care, AI-guided AF screening was associated with increased detection of AF (high-risk group: 4.2% vs. 11.1%, p<0.001; low-risk group: 0.9% vs. 2.4%, p=0.12) over a median follow-up of 10 months. Conclusions A prospective pragmatic study found that the AI a can risk-stratify a relatively uniform population (i.e., older adults at risk for stroke) to detect AF during short-term cardiac monitoring. Furthermore, when compared with usual care, AI-guided cardiac monitoring was associated with increased AF detection. As such, an AI-guided AF screening approach, leveraging existing EHR data and infrastructure, could be effective, patient-centered, and massively scalable, thereby reducing unnecessary health utilization and diagnosis-related anxiety (Clinicaltrials.gov: NCT04208971).

PURPOSEThe advancement of natural language processing (NLP) has promoted the use of detailed textual data in electronic health records (EHRs) to support cancer research and to facilitate patient care. In this review, we aim to assess EHR for cancer research and patient care by using the Minimal Common Oncology Data Elements (mCODE), which is a community-driven effort to define a minimal set of data elements for cancer research and practice. Specifically, we aim to assess the alignment of NLP-extracted data elements with mCODE and review existing NLP methodologies for extracting said data elements.METHODSPublished literature studies were searched to retrieve cancer-related NLP articles that were written in English and published between January 2010 and September 2020 from main literature databases. After the retrieval, articles with EHRs as the data source were manually identified. A charting form was developed for relevant study analysis and used to categorize data including four main topics: metadata, EHR data and targeted cancer types, NLP methodology, and oncology data elements and standards.RESULTSA total of 123 publications were selected finally and included in our analysis. We found that cancer research and patient care require some data elements beyond mCODE as expected. Transparency and reproductivity are not sufficient in NLP methods, and inconsistency in NLP evaluation exists.CONCLUSIONWe conducted a comprehensive review of cancer NLP for research and patient care using EHRs data. Issues and barriers for wide adoption of cancer NLP were identified and discussed.

AbstractPurposeRural populations are disproportionately affected by the COVID‐19 pandemic. We characterized urban‐rural disparities in patient portal messaging utilization for COVID‐19, and, of those who used the portal during its early stage in the Midwest.MethodsWe collected over 1 million portal messages generated by midwestern Mayo Clinic patients from February to August 2020. We analyzed patient‐generated messages (PGMs) on COVID‐19 by urban‐rural locality and incorporated patients’ sociodemographic factors into the analysis.FindingsThe urban‐rural ratio of portal users, message senders, and COVID‐19 message senders was 1.18, 1.31, and 1.79, indicating greater use among urban patients. The urban‐rural ratio (1.69) of PGMs on COVID‐19 was higher than that (1.43) of general PGMs. The urban‐rural ratios of messaging were 1.72‐1.85 for COVID‐19‐related care and 1.43‐1.66 for other health care issues on COVID‐19. Compared with urban patients, rural patients sent fewer messages for COVID‐19 diagnosis and treatment but more messages for other reasons related to COVID‐19‐related health care (eg, isolation and anxiety). The frequent senders of COVID‐19‐related messages among rural patients were 40+ years old, women, married, and White.ConclusionsIn this Midwest health system, rural patients were less likely to use patient online services during a pandemic and their reasons for its use differ from urban patients. Results suggest opportunities for increasing equity in rural patient engagement in patient portals (in particular, minority populations) for COVID‐19. Public health intervention strategies could target reasons why rural patients might seek health care in a pandemic, such as social isolation and anxiety.

AbstractSummaryBuilding a high-quality annotation corpus requires expenditure of considerable time and expertise, particularly for biomedical and clinical research applications. Most existing annotation tools provide many advanced features to cover a variety of needs where the installation, integration and difficulty of use present a significant burden for actual annotation tasks. Here, we present MedTator, a serverless annotation tool, aiming to provide an intuitive and interactive user interface that focuses on the core steps related to corpus annotation, such as document annotation, corpus summarization, annotation export and annotation adjudication.Availability and implementationMedTator and its tutorial are freely available from https://ohnlp.github.io/MedTator. MedTator source code is available under the Apache 2.0 license: https://github.com/OHNLP/MedTator.Supplementary informationSupplementary data are available at Bioinformatics online.

AbstractObjectiveWith COVID-19, there was a need for a rapidly scalable annotation system that facilitated real-time integration with clinical decision support systems (CDS). Current annotation systems suffer from a high-resource utilization and poor scalability limiting real-world integration with CDS. A potential solution to mitigate these issues is to use the rule-based gazetteer developed at our institution.Materials and MethodsPerformance, resource utilization, and runtime of the rule-based gazetteer were compared with five annotation systems: BioMedICUS, cTAKES, MetaMap, CLAMP, and MedTagger.ResultsThis rule-based gazetteer was the fastest, had a low resource footprint, and similar performance for weighted microaverage and macroaverage measures of precision, recall, and f1-score compared to other annotation systems.DiscussionOpportunities to increase its performance include fine-tuning lexical rules for symptom identification. Additionally, it could run on multiple compute nodes for faster runtime.ConclusionThis rule-based gazetteer overcame key technical limitations facilitating real-time symptomatology identification for COVID-19 and integration of unstructured data elements into our CDS. It is ideal for large-scale deployment across a wide variety of healthcare settings for surveillance of acute COVID-19 symptoms for integration into prognostic modeling. Such a system is currently being leveraged for monitoring of postacute sequelae of COVID-19 (PASC) progression in COVID-19 survivors. This study conducted the first in-depth analysis and developed a rule-based gazetteer for COVID-19 symptom extraction with the following key features: low processor and memory utilization, faster runtime, and similar weighted microaverage and macroaverage measures for precision, recall, and f1-score compared to industry-standard annotation systems.

AbstractBackgroundA subset of patients with hypertrophic cardiomyopathy (HCM) is at high risk of sudden cardiac death (SCD). Practice guidelines endorse use of a risk calculator, which requires entry of left atrial (LA) diameter. However, American Society of Echocardiography (ASE) guidelines recommend the use of LA volume index (LAVI) for routine quantification of LA size. The aims of this study were to (a) develop a model to estimate LA diameter from LAVI and (b) evaluate whether substitution of measured LA diameter by estimated LA diameter derived from LAVI reclassifies HCM‐SCD risk.MethodsThe study cohort was comprised of 500 randomly selected HCM patients who underwent transthoracic echocardiography (TTE). LA diameter and LAVI were measured offline using digital clips from TTE. Linear regression models were developed to estimate LA diameter from LAVI. A European Society of Cardiology endorsed equation estimated SCD risk, which was measured using LA diameter and estimated LA diameter derived from LAVI.ResultsThe mean LAVI was 48.5 ± 18.8 mL/m2. The derived LA diameter was 45.1 mm (SD: 5.5 mm), similar to the measured LA diameter (45.1 mm, SD: 7.1 mm). Median SCD risk at 5 years estimated by measured LA diameter was 2.22% (interquartile range (IQR): 1.39, 3.56), while median risk calculated by estimated LA diameter was 2.18% (IQR: 1.44, 3.52). 476/500 (95%) patients maintained the same risk classification regardless of whether the measured or estimated LA diameter was used.ConclusionsSubstitution of measured LA diameter by estimated LA diameter in the HCM‐SCD calculator did not reclassify risk.

Abstract Objective Growing numbers of academic medical centers offer patient cohort discovery tools to their researchers, yet the performance of systems for this use case is not well understood. The objective of this research was to assess patient-level information retrieval methods using electronic health records for different types of cohort definition retrieval. Materials and Methods We developed a test collection consisting of about 100 000 patient records and 56 test topics that characterized patient cohort requests for various clinical studies. Automated information retrieval tasks using word-based approaches were performed, varying 4 different parameters for a total of 48 permutations, with performance measured using B-Pref. We subsequently created structured Boolean queries for the 56 topics for performance comparisons. In addition, we performed a more detailed analysis of 10 topics. Results The best-performing word-based automated query parameter settings achieved a mean B-Pref of 0.167 across all 56 topics. The way a topic was structured (topic representation) had the largest impact on performance. Performance not only varied widely across topics, but there was also a large variance in sensitivity to parameter settings across the topics. Structured queries generally performed better than automated queries on measures of recall and precision but were still not able to recall all relevant patients found by the automated queries. Conclusion While word-based automated methods of cohort retrieval offer an attractive solution to the labor-intensive nature of this task currently used at many medical centers, we generally found suboptimal performance in those approaches, with better performance obtained from structured Boolean queries. Future work will focus on using the test collection to develop and evaluate new approaches to query structure, weighting algorithms, and application of semantic methods.