Caraballo, Pedro J.

Abstract Background Sepsis is a life-threatening condition in which delayed recognition and treatment are associated with increased mortality. While predictive models such as Epic’s Early Detection of Sepsis Model (ESM) were developed to support early intervention, their real-world impact after integration into clinical workflows remains difficult to evaluate. Objectives To evaluate the real-world impact of ESM integrated into clinical workflow on clinical outcomes, antibiotic use, and harm-benefit tradeoffs. Methods We conducted a quasi-experimental study in a single healthcare system using encounter-level data from inpatient settings. Inpatient mortality, prolonged hospitalization, antibiotic use, and sepsis prevalence were compared between the pre-implementation period (3 June 2023 to 20 August 2024) and the online period (21 August 2024 to 26 December 2024) when the model became visible to clinicians. We also applied a counterfactual framework using models trained on pre-implementation data to estimate expected outcomes without ESM and to quantify harms related to overtreatment and delayed treatment. Results Among 101,138 encounters, 86,884 occurred during the pre-implementation period and 14,254 during the online period. In unadjusted analyses, the online period had lower inpatient mortality, prolonged hospitalization, antibiotic use, and sepsis prevalence (all p≤0.002). In the counterfactual analyses, observed outcomes were lower than expected without ESM for mortality (1.21% vs 1.82%; p<0.001), prolonged hospitalization (5.56% vs 7.95%; p<0.001), and antibiotic use (43.52% vs 47.04%; p<0.001). False positive harm (37.72% vs 41.68%; p<0.001) was also lower than expected. Conclusions Integration of ESM into clinical workflow was associated with improved patient outcomes, reduced antibiotic use, and decreased harm from overtreatment, without evidence of increased harm from delayed treatment, supporting a positive net clinical benefit and the safety and effectiveness of ESM under Software as a Medical Device principles.

AbstractInterruptive alerts can negatively impact clinical workflows and contribute to alert fatigue, provider frustration, and burnout. Given that interruptive alert overriding is a heterogeneous and recurring phenomenon, occurring across different organizational contexts with varying characteristics and circumstances, we hypothesize a pragmatic approach with multimodal interventions to address malfunctioning alert populations and maintain those contributing to better patient care.This study aimed to develop a systematic approach to screen, identify, and correct malfunctioning interruptive alerts within a tertiary healthcare system.We performed screening by assessing the alert population, exploring available resources, and defining alert population inclusion and exclusion criteria. We identified interruptive alerts and then conducted an exploratory analysis. We shared insights from discussions with our expert panel to validate our findings and find gaps in current alert monitoring. We then performed focus groups and interviews as part of a root cause analysis. To address the findings of these investigations, we prioritized which alerts to improve, evaluated solutions, and recommended steps to improve our governance structure.We developed an approach to assess around 1,500 unique alerts in a tertiary center from January to June 2023. We introduced two approaches to visually analyze alert populations: alert-focused analysis and people- and systems-focused analysis. We utilized an expert panel to further enhance the power and speed of alert evaluation and then investigated one emerging alert with focus groups, identifying root causes for its malfunction. This alert demonstrated how enterprise practice changes, coupled with design and cultural issues, can trigger significant alert malfunctions.A multi-modal intervention approach is needed to evaluate interruptive alerts and act quickly on findings. Utilizing both analytical and nonanalytical methods can work in synergy to facilitate this framework. Such approaches may reduce time and be valuable tools for optimally allocating resources to tackle institutional alert challenges.

Pharmacogenomics (PGx) has the potential to improve prescribing practices. In 2012, our institution developed a multidisciplinary program to implement PGx-CDS to support the clinical practice. CDS has been implemented for 45 drug-gene interactions, in 42,000 patients with PGx tests results, and in 2023, approximately 9281 CDS interventions have triggered at the time of prescribing. The program has been successful.

Abstract Objectives In the general hospital wards, machine learning (ML)-based early warning systems (EWSs) can identify patients at risk of deterioration to facilitate rescue interventions. We assess subpopulation performance of a ML-based EWS on medical and surgical adult patients admitted to general hospital wards. Materials and Methods We assessed the scores of an EWS integrated into the electronic health record and calculated every 15 minutes to predict a composite adverse event (AE): all-cause mortality, transfer to intensive care, cardiac arrest, or rapid response team evaluation. The distributions of the First Score 3 hours after admission, the Highest Score at any time during the hospitalization, and the Last Score just before an AE or dismissal without an AE were calculated. The Last Score was used to calculate the area under the receiver operating characteristic curve (ROC-AUC) and the precision-recall curve (PRC-AUC). Results From August 23, 2021 to March 31, 2022, 35 937 medical admissions had 2173 (6.05%) AE compared to 25 214 surgical admissions with 4984 (19.77%) AE. Medical and surgical admissions had significant different (P &lt;.001) distributions of the First Score, Highest Score, and Last Score among those with an AE and without an AE. The model performed better in the medical group when compared to the surgical group, ROC-AUC 0.869 versus 0.677, and RPC-AUC 0.988 versus 0.878, respectively. Discussion Heterogeneity of medical and surgical patients can significantly impact the performance of a ML-based EWS, changing the model validity and clinical discernment. Conclusions Characterization of the target patient subpopulations has clinical implications and should be considered when developing models to be used in general hospital wards.

Background: Since QT prolongation is a risk factor for sudden death, stroke, and all-cause mortality, we developed an electronic, institution-wide QT alert system to identify patients with significant QT prolongation (i.e. QTc≥500 ms if QRS <120 ms). These patients encountered a significant increased mortality within 1 year, and nearly 40% of patients with ≥4 QTc inciting factors have died. Herein, we examined the patients with ≤1 identifiable QTc inciting factor. Methods: Between November 2010 and June 2011, 53,732 patients underwent ECG-evaluation, 1,145 QT-alerts (2.1%) were triggered, and 470 of the QT-alerted patients had ECG-isolated QT prolongation. Clinical, laboratory, and therapeutic information were collected. Excluding female gender and bradycardia, other potential QT inciting factors (drugs, electrolyte disturbances, and co-morbidities) were synthesized into a pro-QT score and patients with a score of ≤1 were compared to patients with a pro-QT score >1. Results: About one-third (N=150, 59% female) of this QT-alerted cohort had a pro-QT score of ≤1 including 42 patients with an established diagnosis of congenital long QT syndrome (LQTS). Excluding LQTS patients, the remaining 108 patients (60 female, age 59±25 years) were significantly more likely to have a QT prolonging cardiovascular co-morbidity (44%) than those with a QT-score >1 (29%; p=0.006). However, mortality was similar between those with a QT-score ≤1 (19%) and those with a QT-score >1 (27%, p=0.09). Fourteen patients (9 female; age 64±22 years; QTc 507±14 ms) had no discernible QT inciting factors (i.e. pro-QT score=0) including 6 patients with previously documented QT prolongation. Conclusions: Only a minority of patients with ECG-isolated QT-prolongation (~3%) had no discernible QT-inciting factors other than female gender in 9. Careful scrutiny of these patients with seemingly idiopathic QT prolongation may identify novel QT provoking conditions. Alternatively, considering that 14 of 53,732 patients (1 in 3888) activated the alert system without obvious QT inciting factors and that congenital LQTS has an estimated prevalence of 1 in 2000, it is possible that this universal QT alert system has caught patients with yet to be diagnosed LQTS.

Prolonged QT interval is an independent risk factor for all-cause mortality. However, evaluation of mortality associated to the implementation of a clinical decision support system to increase awareness and provide management recommendations has been challenging. Here we present our attempt to develop a model using only electronic data and different control groups.

The Deterioration Index (DI) is an automatic early warning system that utilizes a machine learning algorithm integrated into the electronic health record and was implemented to improve risk stratification of inpatients. Our pilot implementation showed superior diagnostic accuracy than standard care. A score >60 had a specificity of 88.5% and a sensitivity of 59.8% (PPV 0.1758, NPP 0.9817). However, acceptance in the clinical workflow was divided; nurses preferred standard care, while providers found it helpful.

Familial Hypercholesterolemia (FH) is underdiagnosed in the United States. Clinical decision support (CDS) could increase FH detection once implemented in clinical workflows. We deployed CDS for FH at an academic medical center and sought clinician insights using an implementation survey. In November 2020, the FH CDS was deployed in the electronic health record at all Mayo Clinic sites in two formats: a best practice advisory (BPA) and an in-basket alert. Over three months, 104 clinicians participated in the survey (response rate 11.1%). Most clinicians (81%) agreed that CDS implementation was a good option for identifying FH patients; 78% recognized the importance of implementing the tool in practice, and 72% agreed it would improve early diagnosis of FH. In comparing the two alert formats, clinicians found the in-basket alert more acceptable (p = 0.036) and more feasible (p = 0.042) than the BPA. Overall, clinicians favored implementing the FH CDS in clinical practice and provided feedback that led to iterative refinement of the tool. Such a tool can potentially increase FH detection and optimize patient management.