Wang, Yanshan

Abstract IntroductionRacially and ethnically diverse minorities often experience the disease burden of sexually transmitted infections or diseases (STD) more often than their White counterparts. Yet, little is known about the connection of STD systematic discrimination, racism, and social and behavioral determinants. Plus, little to no details exists related to how this information is recorded in their Electronic Health Records (EHRs). The objective of this study is to assess the completeness of social and behavioral determinants of health (SDOH) data in the EHRs of a minority cohort with STD.Materials and Methods2,993 minority patients diagnosed with a STD at the Mayo Clinic were identified for this study. A natural language processing (NLP) algorithm was applied on the Patient-Provided Information (PPI) in their EHRs to extract SDOH information in six domains that are associated with STD, namely alcohol use, substance use, sexual activity, sexual orientation, housing status, and employment status. The completeness of SDOH was assessed in terms of documentation, breadth, and density.ResultsOur study indicates that nearly half of 2,993 patients did not have SDOH-related PPI records in their EHRs whereas the patients who had SDOH-related PPI records had well-documented records for five out of six SDOH domains, including alcohol use, substance use, sexual activity, housing status, and employment status, except for sexual orientation. A total of 1,504 patients had PPI in their EHRs for at least one of six SDOH domains, which is about 50.3% of the study cohort. Most SDOH domains have a short time span of 1 year, with up to 18 years of record data. Our analysis also indicated that education and age have a significant impact on the recording of SDOH-related PPI records. Patients that are female, older, and higher educated tend to have more SDOH information available in their records.Discussion and ConclusionWe assessed the completeness of SDOH information recorded in the PPI from patients’ EHRs. Due to large amounts of missing SDOH information in the PPI, future research is needed to integrate accurate and robust SDOH related data for downstream research and the impact of systematic discrimination on how this information is collected and interpreted in the EHRs.

Abstract Objective Growing numbers of academic medical centers offer patient cohort discovery tools to their researchers, yet the performance of systems for this use case is not well understood. The objective of this research was to assess patient-level information retrieval methods using electronic health records for different types of cohort definition retrieval. Materials and Methods We developed a test collection consisting of about 100 000 patient records and 56 test topics that characterized patient cohort requests for various clinical studies. Automated information retrieval tasks using word-based approaches were performed, varying 4 different parameters for a total of 48 permutations, with performance measured using B-Pref. We subsequently created structured Boolean queries for the 56 topics for performance comparisons. In addition, we performed a more detailed analysis of 10 topics. Results The best-performing word-based automated query parameter settings achieved a mean B-Pref of 0.167 across all 56 topics. The way a topic was structured (topic representation) had the largest impact on performance. Performance not only varied widely across topics, but there was also a large variance in sensitivity to parameter settings across the topics. Structured queries generally performed better than automated queries on measures of recall and precision but were still not able to recall all relevant patients found by the automated queries. Conclusion While word-based automated methods of cohort retrieval offer an attractive solution to the labor-intensive nature of this task currently used at many medical centers, we generally found suboptimal performance in those approaches, with better performance obtained from structured Boolean queries. Future work will focus on using the test collection to develop and evaluate new approaches to query structure, weighting algorithms, and application of semantic methods.

Abstract Defining patient-to-patient similarity is essential for the development of precision medicine in clinical care and research. Conceptually, the identification of similar patient cohorts appears straightforward; however, universally accepted definitions remain elusive. Simultaneously, an explosion of vendors and published algorithms have emerged and all provide varied levels of functionality in identifying patient similarity categories. To provide clarity and a common framework for patient similarity, a workshop at the American Medical Informatics Association 2019 Annual Meeting was convened. This workshop included invited discussants from academics, the biotechnology industry, the FDA, and private practice oncology groups. Drawing from a broad range of backgrounds, workshop participants were able to coalesce around 4 major patient similarity classes: (1) feature, (2) outcome, (3) exposure, and (4) mixed-class. This perspective expands into these 4 subtypes more critically and offers the medical informatics community a means of communicating their work on this important topic.

Abstract Objective The n2c2/UMass Lowell spin-off shared task focused on medical concept normalization (MCN) in clinical records. This task aimed to assess state-of-the-art methods for matching salient medical concepts from clinical records to a controlled vocabulary. We describe the task and the dataset used, compare the participating systems, and identify the strengths and limitations of the current approaches and directions for future research. Materials and Methods Participating teams were asked to link preselected text spans in discharge summaries (henceforth referred to as concept mentions) to the corresponding concepts in the SNOMED CT (Systematized Nomenclature of Medicine Clinical Terms) and RxNorm vocabularies from the Unified Medical Language System. The shared task used the MCN corpus created by the organizers, which maps all mentions of problems, treatments, and tests in the 2010 i2b2/VA challenge data to the Unified Medical Language System concepts. Submitted systems represented 4 broad categories of approaches: cascading dictionary matching, cosine distance, deep learning, and retrieve-and-rank systems. Disambiguation modules were common across all approaches. Results A total of 33 teams participated in the shared task. The best-performing team achieved an accuracy of 0.8526. The median and mean performances among all teams were 0.7733 and 0.7426, respectively. Conclusions Overall performance among the top 10 teams was high. However, particularly challenging for all teams were mentions requiring disambiguation of misspelled words, acronyms, abbreviations, and mentions with more than 1 possible semantic type. Complex mentions of long, multiword terms were also challenging and, in the future, will require better methods for learning contextualized representations of concept mentions and better use of domain knowledge.

Update This article was updated on December 6, 2019, because of a previous error. On page 1936, in Table VII, “Performance of the Bearing Surface Algorithm,” the row that had read “Bearing surface predicted by algorithm” now reads “Bearing surface predicted by algorithm*.” An erratum has been published: J Bone Joint Surg Am. 2020 Jan 2;102(1):e4. Update This article was updated on March 31, 2020, because of a previous error. On page 1934, in Table IV (“THA Bearing Surface-Related Keywords in Operative Notes”), the row that had read “Femoral stem; stem; HFx-stem; femoral component; femoral component/stem; permanent prosthesis; stem fem cemented” now reads “Femoral head; ball; delta head; delta ceramic head; ion treated; BIOLOX delta; ceramic head; ceramic femoral head; ceramic offset head; ceramic (size) head; alumina ceramic head; alumina prosthetic head; alumna ceramic head; BIOLOX ceramic head; BIOLOX delta head; BIOLOX femoral head; BIOLOX delta ceramic head.” An erratum has been published: J Bone Joint Surg Am. 2020 May 6;102(9):e43. Background: Manual chart review is labor-intensive and requires specialized knowledge possessed by highly trained medical professionals. Natural language processing (NLP) tools are distinctive in their ability to extract critical information from raw text in electronic health records (EHRs). As a proof of concept for the potential application of this technology, we examined the ability of NLP to correctly identify common elements described by surgeons in operative notes for total hip arthroplasty (THA). Methods: We evaluated primary THAs that had been performed at a single academic institution from 2000 to 2015. A training sample of operative reports was randomly selected to develop prototype NLP algorithms, and additional operative reports were randomly selected as the test sample. Three separate algorithms were created with rules aimed at capturing (1) the operative approach, (2) the fixation method, and (3) the bearing surface category. The algorithms were applied to operative notes to evaluate the language used by 29 different surgeons at our center and were applied to EHR data from outside facilities to determine external validity. Accuracy statistics were calculated with use of manual chart review as the gold standard. Results: The operative approach algorithm demonstrated an accuracy of 99.2% (95% confidence interval [CI], 97.1% to 99.9%). The fixation technique algorithm demonstrated an accuracy of 90.7% (95% CI, 86.8% to 93.8%). The bearing surface algorithm demonstrated an accuracy of 95.8% (95% CI, 92.7% to 97.8%). Additionally, the NLP algorithms applied to operative reports from other institutions yielded comparable performance, demonstrating external validity. Conclusions: NLP-enabled algorithms are a promising alternative to the current gold standard of manual chart review for identifying common data elements from orthopaedic operative notes. The present study provides a proof of concept for use of NLP techniques in clinical research studies and registry-development endeavors to reliably extract data of interest in an expeditious and cost-effective manner.

AbstractObjectivesTo create test collections for evaluating clinical information retrieval (IR) systems and advancing clinical IR research.Materials and MethodsElectronic health record (EHR) data, including structured and free-text data, from 45 000 patients who are a part of the Mayo Clinic Biobank cohort was retrieved from the clinical data warehouse. The clinical IR system indexed a total of 42 million free-text EHR documents. The search queries consisted of 56 topics developed through a collaboration between Mayo Clinic and Oregon Health & Science University. We described the creation of test collections, including a to-be-evaluated document pool using five retrieval models, and human assessment guidelines. We analyzed the relevance judgment results in terms of human agreement and time spent, and results of three levels of relevance, and reported performance of five retrieval models.ResultsThe two judges had a moderate overall agreement with a Kappa value of 0.49, spent a consistent amount of time judging the relevance, and were able to identify easy and difficult topics. The conventional retrieval model performed best on most topics while a concept-based retrieval model had better performance on the topics requiring conceptual level retrieval.DiscussionIR can provide an alternate approach to leveraging clinical narratives for patient information discovery as it is less dependent on semantics. Our study showed the feasibility of test collections along with a few challenges.ConclusionThe conventional test collections for evaluating the IR system show potential for successfully evaluating clinical IR systems with a few challenges to be investigated.