Romero-Brufau, Santiago

Objective To compare the diagnostic accuracy of four available automated electronic medical record (EMR) retrieval methods, including a large language model (LLM)-assisted workflow, against manual chart adjudication for identifying cardiovascular events. Design Retrospective diagnostic accuracy study. Setting Three sites within a single US tertiary health system. Participants Two adult cohorts with previously adjudicated cardiovascular outcomes were included. Cohort 1 included 2258 patients treated with immune checkpoint inhibitors, and Cohort 2 included 1426 patients who underwent transcatheter aortic valve replacement. Primary and secondary outcome measures The reference standard was clinician manual chart adjudication. Outcomes included ischaemic stroke or transient ischaemic attack, myocardial infarction (MI), heart failure (HF) exacerbation or hospitalisation and a composite major adverse cardiovascular events (MACE) outcome. Automated retrieval methods included International Classification of Diseases (ICD) codes, primary diagnosis, problem list and a zero-shot LLM workflow. Area under the (receiver operating characteristic) curve (AUC), sensitivity, specificity and net reclassification improvement were assessed. Results In Cohort 1, the LLM achieved the highest AUC for stroke (0.920; 95% CI 0.881 to 0.958), MI (0.938; 95% CI 0.905 to 0.971) and composite MACE (0.880; 95% CI 0.854 to 0.907), whereas ICD-based retrieval had a higher AUC for HF (0.882; 95% CI 0.845 to 0.918 vs 0.873; 95% CI 0.831 to 0.914). In Cohort 2, the LLM achieved the highest AUC for all evaluated outcomes: stroke (0.915; 95% CI 0.862 to 0.968), MI (0.928; 95% CI 0.839 to 1.000), HF (0.844; 95% CI 0.803 to 0.884) and composite MACE (0.862; 95% CI 0.829 to 0.895). In Cohort 1, differences in AUC between the LLM and ICD methods were not statistically significant across outcomes, whereas in Cohort 2 the LLM showed significantly higher AUC for stroke and composite MACE. Conclusion In this multisite retrospective validation study, the LLM-assisted workflow showed strong but context-dependent performance for identifying cardiovascular events from the EMR. Performance varied by outcome and cohort, and ICD-based retrieval remained competitive for some use cases. These findings support a complementary role for LLM-assisted extraction in retrospective cardiovascular outcomes research.

ABSTRACT Errors in antibiotic prescriptions are frequent, often resulting from the inadequate coverage of the infection-causative microorganism. The efficacy of iAST, a machine-learning-based software offering empirical and organism-targeted antibiotic recommendations, was assessed. The study was conducted in a 12-hospital Spanish institution. After model fine-tuning with 27,531 historical antibiograms, 325 consecutive patients with acute infections were selected for retrospective validation. The primary endpoint was comparing each of the top three of iAST’s antibiotic recommendations’ success rates (confirmed by antibiogram results) with the antibiotic prescribed by the physicians. Secondary endpoints included examining the same hypothesis within specific study population subgroups and assessing antibiotic stewardship by comparing the percentage of antibiotics recommended that belonged to different World Health Organization AWaRe groups within each arm of the study. All of iAST first three recommendations were non-inferior to doctor prescription in the primary endpoint analysis population as well as the secondary endpoint. The overall success rate of doctors’ empirical treatment was 68.93%, while that of the first three iAST options was 91.06% ( P < 0.001), 90.63% ( P < 0.001), and 91.06% ( P < 001), respectively. For organism-targeted therapy, the doctor’s overall success rate was 84.16%, and that of the first three ranked iAST options was 97.83% ( P < 0.001), 94.09% ( P < 0.001), and 91.30% ( P < 0.001), respectively. In empirical therapy, compared to physician prescriptions, iAST demonstrated a greater propensity to recommend access antibiotics, fewer watch antibiotics, and higher reserve antibiotics. In organism-targeted therapy, iAST advised a higher utilization of access antibiotics. The present study demonstrates iAST accuracy in predicting antibiotic susceptibility, showcasing its potential to promote effective antibiotic stewardship. CLINICAL TRIALS This study is registered with ClinicalTrials.gov as NCT06174519 .

Sepsis is considered a medical emergency where delays in initial treatment are associated with increased morbidity and mortality, yet there is no gold standard for identifying sepsis onset and thus treatment timing. We leverage electronic health record (EHR) data with clinical expertise to develop a continuous-time Markov decision process (MDP) optimal stopping model that identifies the optimal first intervention action (anti-infective, fluid, or wait). To study the impact of initial treatment of patients at risk for developing sepsis, we define the delayed treatment population who received delayed treatment upon admission or during hospitalization and serves as an approximation of the natural history of sepsis. We apply the optimal first treatment policy to sample patient visits from the nondelayed treatment population. This analysis indicates the average risk of death could be reduced by approximately 2.2%, the average time until treatment could be reduced by 106 minutes, and the average severity of the treatment state could be reduced by 15.5% compared with the treatment they received in the hospital. We study the properties of the optimal policy to define an easily interpretable initial treatment heuristic that considers a patient’s organ dysfunction, location, and septic shock status. This generalizable framework can inform personalized treatment of patients at risk for sepsis. History: This paper has been accepted for the Decision Analysis Special Issue on Emerging Topics in Health Decision Analysis. Funding: This material is based upon work supported by the National Science Foundation [Grant 1522107 (North Carolina State University), 1522106 (Mayo Clinic), and 1833538 (Drexel University)].

Abstract Objective Artificial intelligence (AI) models may propagate harmful biases in performance and hence negatively affect the underserved. We aimed to assess the degree to which data quality of electronic health records (EHRs) affected by inequities related to low socioeconomic status (SES), results in differential performance of AI models across SES. Materials and Methods This study utilized existing machine learning models for predicting asthma exacerbation in children with asthma. We compared balanced error rate (BER) against different SES levels measured by HOUsing-based SocioEconomic Status measure (HOUSES) index. As a possible mechanism for differential performance, we also compared incompleteness of EHR information relevant to asthma care by SES. Results Asthmatic children with lower SES had larger BER than those with higher SES (eg, ratio = 1.35 for HOUSES Q1 vs Q2–Q4) and had a higher proportion of missing information relevant to asthma care (eg, 41% vs 24% for missing asthma severity and 12% vs 9.8% for undiagnosed asthma despite meeting asthma criteria). Discussion Our study suggests that lower SES is associated with worse predictive model performance. It also highlights the potential role of incomplete EHR data in this differential performance and suggests a way to mitigate this bias. Conclusion The HOUSES index allows AI researchers to assess bias in predictive model performance by SES. Although our case study was based on a small sample size and a single-site study, the study results highlight a potential strategy for identifying bias by using an innovative SES measure.

Abstract Hospital census prediction has well-described implications for efficient hospital resource utilization, and recent issues with hospital crowding due to CoVID-19 have emphasized the importance of this task. Our team has been leading an institutional effort to develop machine-learning models that can predict hospital census 12 hours into the future. We describe our efforts at developing accurate empirical models for this task. Ultimately, with limited resources and time, we were able to develop simple yet useful models for 12-hour census prediction and design a dashboard application to display this output to our hospital’s decision-makers. Specifically, we found that linear models with ElasticNet regularization performed well for this task with relative 95% error of +/− 3.4% and that this work could be completed in approximately 7 months.

Abstract Objective We aimed to develop a model for accurate prediction of general care inpatient deterioration. Materials and Methods Training and internal validation datasets were built using 2-year data from a quaternary hospital in the Midwest. Model training used gradient boosting and feature engineering (clinically relevant interactions, time-series information) to predict general care inpatient deterioration (resuscitation call, intensive care unit transfer, or rapid response team call) in 24 hours. Data from a tertiary care hospital in the Southwest were used for external validation. C-statistic, sensitivity, positive predictive value, and alert rate were calculated for different cutoffs and compared with the National Early Warning Score. Sensitivity analysis evaluated prediction of intensive care unit transfer or resuscitation call. Results Training, internal validation, and external validation datasets included 24 500, 25 784 and 53 956 hospitalizations, respectively. The Mayo Clinic Early Warning Score (MC-EWS) demonstrated excellent discrimination in both the internal and external validation datasets (C-statistic = 0.913, 0.937, respectively), and results were consistent in the sensitivity analysis (C-statistic = 0.932 in external validation). At a sensitivity of 73%, MC-EWS would generate 0.7 alerts per day per 10 patients, 45% less than the National Early Warning Score. Discussion Low alert rates are important for implementation of an alert system. Other early warning scores developed for the general care ward have achieved lower discrimination overall compared with MC-EWS, likely because MC-EWS includes both nursing assessments and extensive feature engineering. Conclusions MC-EWS achieved superior prediction of general care inpatient deterioration using sophisticated feature engineering and a machine learning approach, reducing alert rate.

Objective Create a score to identify patients at risk of death or hospice placement who may benefit from goals of care discussion earlier in the hospitalisation. Design Retrospective cohort study to develop a risk index using multivariable logistic regression. Setting Two tertiary care hospitals in Southeastern Minnesota. Participants 92 879 adult general care admissions (50% male, average age 60 years). Primary and secondary outcome measures Our outcome measure was an aggregate of inhospital death or discharge to hospice. Predictor variables for the model encompassed comorbidities, nutrition status, functional status, demographics, fall risk, mental status, Charlson Comorbidity Index and acuity of illness on admission. Resuscitation status, race, geographic area of residence and marital status were added as covariates to account for confounding. Results Inhospital mortality and discharge to hospice were rare, with incidences of 1.2% and 0.8%, respectively. The Hospital End-of-Life Prognostic Score (HELPS) demonstrated good discrimination (C-statistic=0.866 in derivation set and 0.834 in validation set). The patients with the highest 5% of scores had an 8% risk of the outcome measure, relative risk 12.9 (10.9–15.4) when compared to the bottom 95%. Conclusions HELPS is able to identify patients with a high risk of inhospital death or need for hospice at discharge. These patients may benefit from early goals of care discussions.

Background— Clinical outcomes after stent placement in patients with a history of metal allergy remain incompletely understood. We performed a single-center retrospective study to evaluate such outcomes. Methods and Results— Twenty-nine allergic patients who underwent coronary stent implantation were compared with a nonallergic group (n=250) matched for demographics and a propensity score for allergy to metal. Hypersensitivity to nickel was reported in 26 of 29 and chromium in 9 of 29. Patch testing performed in 11 of 29 patients was positive in all. Comparing allergy versus control subjects, there were no differences in number of segments treated (1.4±0.7 versus 1.5±0.7), stents placed (1.7±1.1 versus 1.6±0.9), and frequency of drug-eluting stent usage (52% versus 60%). In-hospital death (0% versus 0%), myocardial infarction (MI, 4% versus 3%, P =0.27), and 30-day death (3% versus 0%, P =0.53) and MI (3% versus 4%, P =0.71) were statistically similar. There were no differences in 4-year death (12% versus 13%), target lesion revascularization (TLR, 13 versus 17%, P =0.54), or death/MI/TLR (24% versus 34%, P =0.20). Clinically driven repeat angiography in 12 of 29 allergy patients revealed binary restenosis rates of 27% in bare metal stents and 0% in drug-eluting stents, with mean diameter in-stent restenosis of 36% and 8%, respectively. There was no change in circulating eosinophil and lymphocyte counts after stenting in the allergy group (0.19–0.20, P =0.67, and 1.90–1.79, P =0.59, respectively). Conclusions— A history of metal allergy was not associated with adverse early or late outcomes in this single-center study.