Kaggal, Vinod C.

Objective To compare the diagnostic accuracy of four available automated electronic medical record (EMR) retrieval methods, including a large language model (LLM)-assisted workflow, against manual chart adjudication for identifying cardiovascular events. Design Retrospective diagnostic accuracy study. Setting Three sites within a single US tertiary health system. Participants Two adult cohorts with previously adjudicated cardiovascular outcomes were included. Cohort 1 included 2258 patients treated with immune checkpoint inhibitors, and Cohort 2 included 1426 patients who underwent transcatheter aortic valve replacement. Primary and secondary outcome measures The reference standard was clinician manual chart adjudication. Outcomes included ischaemic stroke or transient ischaemic attack, myocardial infarction (MI), heart failure (HF) exacerbation or hospitalisation and a composite major adverse cardiovascular events (MACE) outcome. Automated retrieval methods included International Classification of Diseases (ICD) codes, primary diagnosis, problem list and a zero-shot LLM workflow. Area under the (receiver operating characteristic) curve (AUC), sensitivity, specificity and net reclassification improvement were assessed. Results In Cohort 1, the LLM achieved the highest AUC for stroke (0.920; 95% CI 0.881 to 0.958), MI (0.938; 95% CI 0.905 to 0.971) and composite MACE (0.880; 95% CI 0.854 to 0.907), whereas ICD-based retrieval had a higher AUC for HF (0.882; 95% CI 0.845 to 0.918 vs 0.873; 95% CI 0.831 to 0.914). In Cohort 2, the LLM achieved the highest AUC for all evaluated outcomes: stroke (0.915; 95% CI 0.862 to 0.968), MI (0.928; 95% CI 0.839 to 1.000), HF (0.844; 95% CI 0.803 to 0.884) and composite MACE (0.862; 95% CI 0.829 to 0.895). In Cohort 1, differences in AUC between the LLM and ICD methods were not statistically significant across outcomes, whereas in Cohort 2 the LLM showed significantly higher AUC for stroke and composite MACE. Conclusion In this multisite retrospective validation study, the LLM-assisted workflow showed strong but context-dependent performance for identifying cardiovascular events from the EMR. Performance varied by outcome and cohort, and ICD-based retrieval remained competitive for some use cases. These findings support a complementary role for LLM-assisted extraction in retrospective cardiovascular outcomes research.

Abstract Background: Germline Pathogenic Variants (PVs) in ATM, BRCA1, BRCA2, CHEK2, and PALB2 are known to increase breast cancer (BCa) risk, but the effect of these PVs on tumor biology is not well-understood. Ki-67 is an established prognostic marker in hormone receptor (HR) positive breast cancer but the distribution of Ki-67 in breast cancer associated with germline PVs is not known. Methods: The study included patients with loco-regional (Stage I-III) HR-positive (ER+ or PR+), HER2-negative invasive ductal or lobular breast cancer evaluated at Mayo Clinic for curative intent treatment between 2012 and 2021. PV carrier status was ascertained in more than two-thirds of cases from germline whole exome sequencing (Regeneron) of patients who participated in a prospective registry at Mayo Clinic. Additional PV carriers were identified through a review of the results of clinical germline genetic testing in electronic medical records aided by natural language processing. Clinically performed Ki-67 values were abstracted from electronic medical records and analyzed as continuous and categorical variables (low: < 20% vs. high: ≥20%). Throughout the timeline of the study, a consistent method for assessment of Ki-67 was utilized at Mayo Clinic, Rochester: Immunohistochemical staining of the Ki-67 antigen in formalin-fixed paraffin-embedded tissue was performed using MIB-1 clone, the scanned slides were reviewed by a technologist and traced areas further analyzed for Ki-67 using Aperio Software. Patients who received neoadjuvant therapy prior to the assessment of Ki-67, with metastatic disease at diagnosis, with PVs in two or more genes of interest, or unknown germline PV or Ki-67 status were excluded from the analysis. Multiple primary tumors in the same patient diagnosed within one year of each other were analyzed independently. The control (non-carrier) group included randomly selected patients from the prospective registry who tested negative for PVs in the five genes over the same time period. Ki-67 values were compared between germline PV carriers and non-carriers in a multivariable logistic regression analysis adjusting for sex, race, age at diagnosis, stage at presentation, and histology of the tumor. Results: A total of 113 HR+/HER2- tumors from 111 PV carriers (ATM: 27, BRCA1: 24, BRCA2: 28, CHEK2: 22, and PALB2:12) along with 431 HR+/HER2- tumors from 422 randomly selected non-carriers were evaluated. The majority (>95%) of patients were non-Hispanic White. Germline PV carriers in each gene were diagnosed at a significantly younger age (P< 0.05) compared to non-carriers. The median Ki-67 value was 12 (Range: 1 to 93) for non-carriers, 16 (Range: 0 to 79) for ATM, 31 (Range: 4 to 86) for BRCA1, 30 (Range: 6 to 80) for BRCA2, 13 (Range: 1 to 38) for CHEK2, and 20 (Range: 3 to 46) for PALB2 PV carriers. In multivariable analysis, compared to non-carriers, we observed significantly higher odds of a high Ki67 value (≥20%) for tumors arising from germline PV carriers in BRCA1 (Odds Ratio (OR): 7.3, 95%CI: 2.3 – 23.3, p< 0.001), BRCA2 (OR: 3.0, 95%CI: 1.2 – 7.4, p=0.02) and PALB2 (OR: 3.9, 95%CI: 1.1 – 14.5, p=0.04). No such relationship was observed with either ATM (OR: 1.1, 95%CI: 0.4 – 2.9, p=0.81) or CHEK2 (OR: 0.8, 95%CI: 0.3 – 2.2, p=0.63). Conclusion: This study reveals that HR+/HER2- tumors in BRCA1, BRCA2, and PALB2 PV carriers have a significantly higher Ki-67 expression compared to non-carriers. These findings have implications for understanding prognosis and endocrine response PV with HR+ breast cancer. Further exploration of the relationships between Ki67 and germline PV status is necessary to personalize systemic therapy options in germline PV carriers. Acknowledgments: Thank you to Regeneron Genetics Center. The study was funded in part by the NIH Specialized Program of Research Excellence in Breast Cancer [P50CA116201] to Mayo Clinic, Mayo Clinic Department of Oncology Small Grants Program, and the Paul Calabresi Program at Mayo Clinic Citation Format: Robert Scheel, Grace Choong, Tara Rao, Nicholas Boddicker, Chunling Hu, Jie Na, Vinod Kaggal, Sean Murphy, Karthik Giridhar, Saba Yasir, Matthew Goetz, Fergus Couch, Siddhartha Yadav. Ki-67 Expression in Breast Cancer associated with ATM, BRCA1, BRCA2, CHEK2 and PALB2 Pathogenic Variants [abstract]. In: Proceedings of the 2023 San Antonio Breast Cancer Symposium; 2023 Dec 5-9; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2024;84(9 Suppl):Abstract nr PO5-08-07.

Introduction: Predictive algorithms using multiple clinical variables can identify patients at higher risk for abdominal aortic aneurysms (AAA), however they are cumbersome and cannot used directly by patients. A simple, cheap, noninvasive, and readily available (virtual) screening tool for identification of AAA could aid in screening algorithms. Aims: We sought to determine if a machine learning algorithm could predict the presence of an AAA by using a image of the face Methods: Diagnostic imaging studies were extracted from the electronic health record (EHR) from 5/7/2018 through 1/1/2023 and analyzed by a natural language processing algorithm previously validated to identify abdominal aortic aneurysms. Patient EHR profile pictures were extracted and matched to imaging studies. Various deep neural network (DNN) architectures were explored, all trained as classifiers on the face images along with clinical variables to predict clinical outcomes. All models used were CNNs with an EfficientNet architecture. Model performance was evaluated by standard metrics such as the area under the curve (AUC), specificities (Sp), and sensitivities (Sn). All performance metrics reported results from the test subset of the primary dataset and did not contain any observations considered during training. Results: A total of 5522 patients with facial images and AAA diagnostic studies were analyzed, among whom 1314 (23.8%) had a AAA. The optimal model had an AUC of 0.72 (Sn 0.44, Sp 0.83) for predicting AAA and remained similar in women (AUC 0.7), men (AUC 0.65), age >65 (AUC 0.63), and age <=65 (AUC 0.72) (Table). Conclusions: Using only an image of a patients face, the presence of an AAA could be identified with moderate performance in men, women, and in those less than 65. The model’s performance in demographics not typically screened for AAA may allow for targeted screening programs in these groups.

Abstract Hypertrophic cardiomyopathy (HCM) is a heritable, phenotypically diverse, and often asymptomatic heart muscle disease which is a major cause of sudden cardiac death (SCD) in young adults. The gold-standard for the diagnosis of HCM is echocardiography (echo), which is an ultrasound-based cardiac imaging modality. Across all sites of the Mayo Clinic enterprise, echo images and measurement data are reviewed, interpreted, and reported via the Echo Information Management System (EIMS). The objective of this paper is to develop a machine learning model for the identification of HCM from cardiac measurements obtained by the echo. We developed a novel machine learning model on patient demographic information and echo measurements that were retrieved from the EIMS digital data registry and selected by cardiologists. Random forest (RF) was utilized to investigate the predictive performance of these features on the identification of HCM patients. The HCM cohort consists of 3,548 patients with at least one HCM diagnostic billing code (ICD-9 or ICD-10), from 2014 to 2019. The class labels “HCM yes” and “HCM no” were assigned by manual review of medical records as well as the outcomes of the gold standard imaging tests for HCM diagnosis. The developed model performed well in finding HCM patients with an accuracy of 95%, recall of 99%, and precision of 97%. The F1 score was 98 %, while 4% of patients were misclassified. This model will be translated into clinical practice for a clinical decision support system in EIMS to assist providers in the accurate diagnosis of HCM from echo data automatically while ensuring high-quality echo interpretation.

Background Automated individualized risk prediction tools linked to electronic health records ( EHR s) are not available for management of patients with peripheral arterial disease. The goal of this study was to create a prognostic tool for patients with peripheral arterial disease using data elements automatically extracted from an EHR to enable real‐time and individualized risk prediction at the point of care. Methods and Results A previously validated phenotyping algorithm was deployed to an EHR linked to the Rochester Epidemiology Project to identify peripheral arterial disease cases from Olmsted County, MN, for the years 1998 to 2011. The study cohort was composed of 1676 patients: 593 patients died over 5‐year follow‐up. The c‐statistic for survival in the overall data set was 0.76 (95% confidence interval [CI], 0.74–0.78), and the c‐statistic across 10 cross‐validation data sets was 0.75 (95% CI, 0.73–0.77). Stratification of cases demonstrated increasing mortality risk by subgroup (low: hazard ratio, 0.35 [95% CI, 0.21–0.58]; intermediate‐high: hazard ratio, 2.98 [95% CI, 2.37–3.74]; high: hazard ratio, 8.44 [95% CI, 6.66–10.70], all P <0.0001 versus the reference subgroup). An equation for risk calculation was derived from Cox model parameters and β estimates. Big data infrastructure enabled deployment of the real‐time risk calculator to the point of care via the EHR . Conclusions This study demonstrates that electronic tools can be deployed to EHR s to create automated real‐time risk calculators to predict survival of patients with peripheral arterial disease. Moreover, the prognostic model developed may be translated to patient care as an automated and individualized real‐time risk calculator deployed at the point of care.

The concept of optimizing health care by understanding and generating knowledge from previous evidence, ie, the Learning Health-care System (LHS), has gained momentum and now has national prominence. Meanwhile, the rapid adoption of electronic health records (EHRs) enables the data collection required to form the basis for facilitating LHS. A prerequisite for using EHR data within the LHS is an infrastructure that enables access to EHR data longitudinally for health-care analytics and real time for knowledge delivery. Additionally, significant clinical information is embedded in the free text, making natural language processing (NLP) an essential component in implementing an LHS. Herein, we share our institutional implementation of a big data-empowered clinical NLP infrastructure, which not only enables health-care analytics but also has real-time NLP processing capability. The infrastructure has been utilized for multiple institutional projects including the MayoExpertAdvisor, an individualized care recommendation solution for clinical care. We compared the advantages of big data over two other environments. Big data infrastructure significantly outperformed other infrastructure in terms of computing speed, demonstrating its value in making the LHS a possibility in the near future.

Abstract Objective The objective of the Strategic Health IT Advanced Research Project area four (SHARPn) was to develop open-source tools that could be used for the normalization of electronic health record (EHR) data for secondary use—specifically, for high throughput phenotyping. We describe the role of Intermountain Healthcare’s Clinical Element Models ([CEMs] Intermountain Healthcare Health Services, Inc, Salt Lake City, Utah) as normalization “targets” within the project. Materials and Methods Intermountain’s CEMs were either repurposed or created for the SHARPn project. A CEM describes “valid” structure and semantics for a particular kind of clinical data. CEMs are expressed in a computable syntax that can be compiled into implementation artifacts. The modeling team and SHARPn colleagues agilely gathered requirements and developed and refined models. Results Twenty-eight “statement” models (analogous to “classes”) and numerous “component” CEMs and their associated terminology were repurposed or developed to satisfy SHARPn high throughput phenotyping requirements. Model (structural) mappings and terminology (semantic) mappings were also created. Source data instances were normalized to CEM-conformant data and stored in CEM instance databases. A model browser and request site were built to facilitate the development. Discussion The modeling efforts demonstrated the need to address context differences and granularity choices and highlighted the inevitability of iso-semantic models. The need for content expertise and “intelligent” content tooling was also underscored. We discuss scalability and sustainability expectations for a CEM-based approach and describe the place of CEMs relative to other current efforts. Conclusions The SHARPn effort demonstrated the normalization and secondary use of EHR data. CEMs proved capable of capturing data originating from a variety of sources within the normalization pipeline and serving as suitable normalization targets.

Health disparities and solutions are heterogeneous within and among racial and ethnic groups, yet existing administrative databases lack the granularity to reflect important sociocultural distinctions. We measured the efficacy of a natural-language–processing algorithm to identify a specific immigrant group. The algorithm demonstrated accuracy and precision in identifying Somali patients from the electronic medical records at a single institution. This technology holds promise to identify and track immigrants and refugees in the United States in local health care settings.