Heaton, Heather A.

Abstract Objective We evaluated the quality and adoption of a large language model (LLM)-based summarization tool for ongoing hospital care. Materials and methods An AI summarization tool was created to provide a “Patient Story”, specialty-specific “Recent Notes”, and “Recent Events” over the previous 24 or 72 hours. We conducted a pragmatic mixed-methods quality assessment at three tertiary-care academic hospitals utilizing (1) Provider Documentation Summarization Quality Instrument (PDSQI-9), (2) utilization analytics, and (3) qualitative end user feedback. The PDSQI-9 included whether summaries were accurate, cited, comprehensible, organized, succinct, non-stigmatizing, synthesized, thorough, and useful. 512 users were given access, from whom 52 participants submitted 205 surveys (10.2% response rate). Results 52 respondents submitted an average of 4.3 surveys (range 1–8). Users rated the tool favorably across all PDSQI-9 domains, with a combined average score of 4.68 (range 4.56–4.78, S.D. 0.67) across the eight domains scored on a 5-point modified Likert scale. Utilization metrics demonstrated strong uptake with frequent views. Positive qualitative feedback revealed cognitive offloading for complex patients and effective summarization of medical problems. Critical feedback showed a need to cross-reference narrative notes to current-state data and lack of detailed specialty-specific summarization. Discussion Generative artificial intelligence has emerged as a potentially transformative technology for generating succinct, verifiable summaries of ongoing care. In this pragmatic implementation, end-users indicated high perceived quality across PDSQI-9 domains. Conclusions An LLM-based summarization tool for ongoing hospitalization care was rated of high quality by diverse clinicians in real-world settings, demonstrated a favorable safety profile, and showed sustained utilization.

Abstract Background Hospital discharge summaries are critical for ensuring continuity of care, but their quality often varies. Large language models (LLMs) have the potential to standardize and enhance the efficiency of this documentation process. Objectives To evaluate the quality of hospital discharge summaries created by an LLM‐based hospital course drafting tool created by Epic Systems compared with human‐written summaries. Methods Retrospective study at a single tertiary‐care institution in 2024. The cohort included 100 adult hospitalizations lasting >72 h across medical and surgical dismissing services. No interventions were performed. Summaries (LLM‐generated vs. human‐written) were independently reviewed using a standardized rubric covering nine domains (e.g., comprehensiveness, clarity, relevance). Scores were normalized and compared. Readability was assessed using Flesch Reading Ease. Results LLM‐generated summaries outperformed human‐written summaries across all criteria ( p < .05), with the greatest difference observed in comprehensiveness (LLM median 0.62 vs. human −0.23). Human‐written summaries from surgical services scored lower than those from medical services, but LLM performance was consistent across both. Human summaries had higher Flesch Reading Ease scores (33.11 vs. 26.2; p < .05), reflecting simpler language. Conclusions LLM‐generated summaries demonstrated superior quality, consistency, and clinical utility compared with human‐written summaries, highlighting their potential to improve documentation efficiency and standardization.

Background: The use of validated risk stratification tools for the workup of suspected pulmonary embolism (PE) is a recommendation of the American Society of Hematology and the American College of Emergency Physicians. Methods: We designed and implemented electronic clinical pretest probability (PTP) tools for use in emergency departments (ED). Overall, 38 EDs (tertiary and regional EDs) in the United States were involved across three health systems. PTP use was analyzed between September 12, 2022 and January 11, 2023 in 270,247 ED visits. PTP use was examined in terms of the percentage of visits for which patients underwent computed tomography pulmonary angiography (CTPA). Each site chose the 3-tier Wells’ score for implementation and Site 2 designed a combined Wells’, pulmonary embolism rule-out criteria (PERC), and YEARS score. Results: At Site 3, forced use resulted in documented PTP scores in 49–53% of ordered CTPAs. At Sites 1 and 2, where PTP scores were optional, documented PTP scores occurred in 2–3% and 1–3% of CTPA orders, respectively. At Site 1, the use of PTP increased slightly over the study period, with signs that PE yield on imaging was also increasing (3.4–5.9%). At Site 2, PE yield on imaging was 9–10%, and it remained similar, with similar use of PTP tools over the study period. PE yield (6–8%) on imaging also remained similar throughout the study at Site 3. Conclusions: Guideline-endorsed PTP scores for PE at three independent health systems in the United States did not increase PE yield on imaging. Novel implementation strategies involving interdisciplinary teams are urgently needed.

Importance The emergency department (ED) offers an opportunity to initiate palliative care for older adults with serious, life-limiting illness. Objective To assess the effect of a multicomponent intervention to initiate palliative care in the ED on hospital admission, subsequent health care use, and survival in older adults with serious, life-limiting illness. Design, Setting, and Participants Cluster randomized, stepped-wedge, clinical trial including patients aged 66 years or older who visited 1 of 29 EDs across the US between May 1, 2018, and December 31, 2022, had 12 months of prior Medicare enrollment, and a Gagne comorbidity score greater than 6, representing a risk of short-term mortality greater than 30%. Nursing home patients were excluded. Intervention A multicomponent intervention (the Primary Palliative Care for Emergency Medicine intervention) included (1) evidence-based multidisciplinary education; (2) simulation-based workshops on serious illness communication; (3) clinical decision support; and (4) audit and feedback for ED clinical staff. Main Outcome and Measures The primary outcome was hospital admission. The secondary outcomes included subsequent health care use and survival at 6 months. Results There were 98 922 initial ED visits during the study period (median age, 77 years [IQR, 71-84 years]; 50% were female; 13% were Black and 78% were White; and the median Gagne comorbidity score was 8 [IQR, 7-10]). The rate of hospital admission was 64.4% during the preintervention period vs 61.3% during the postintervention period (absolute difference, −3.1% [95% CI, −3.7% to −2.5%]; adjusted odds ratio [OR], 1.03 [95% CI, 0.93 to 1.14]). There was no difference in the secondary outcomes before vs after the intervention. The rate of admission to an intensive care unit was 7.8% during the preintervention period vs 6.7% during the postintervention period (adjusted OR, 0.98 [95% CI, 0.83 to 1.15]). The rate of at least 1 revisit to the ED was 34.2% during the preintervention period vs 32.2% during the postintervention period (adjusted OR, 1.00 [95% CI, 0.91 to 1.09]). The rate of hospice use was 17.7% during the preintervention period vs 17.2% during the postintervention period (adjusted OR, 1.04 [95% CI, 0.93 to 1.16]). The rate of home health use was 42.0% during the preintervention period vs 38.1% during the postintervention period (adjusted OR, 1.01 [95% CI, 0.92 to 1.10]). The rate of at least 1 hospital readmission was 41.0% during the preintervention period vs 36.6% during the postintervention period (adjusted OR, 1.01 [95% CI, 0.92 to 1.10]). The rate of death was 28.1% during the preintervention period vs 28.7% during the postintervention period (adjusted OR, 1.07 [95% CI, 0.98 to 1.18]). Conclusions and Relevance This multicomponent intervention to initiate palliative care in the ED did not have an effect on hospital admission, subsequent health care use, or short-term mortality in older adults with serious, life-limiting illness. Trial Registration ClinicalTrials.gov Identifier: NCT03424109

Abstract Objectives We aimed to improve the operational efficiency of clinical staff, including physicians and allied health professionals, in the previsit review of patients by implementing a disease-focused dashboard within the electronic health record system. The dashboard was tailored to the unique requirements of the clinic and patient population. Methods A prospective quality improvement study was conducted at an accredited pulmonary hypertension (PH) clinic within a large academic center, staffed by two full time physicians and two allied health professionals. Physicians' review time before and after implementation of the PH dashboard was measured using activity log data derived from an EHR database. The review time for clinic staff was measured through direct observation, with review method—either conventional or newly implemented dashboard—randomly assigned. Results Over the study period, the median number of patients reviewed by physicians per day increased slightly from 5.50 (interquartile range [IQR]: 1.35) before to 5.95 (IQR: 0.85) after the implementation of the PH dashboard (p = 0.535). The median review time for the physicians decreased with the use of the dashboard, from 7.0 minutes (IQR: 1.55) to 4.95 minutes (IQR: 1.35; p < 0.001). Based on the observed timing of 70 patient encounters among allied clinical staff, no significant difference was found for experienced members (4.65 minutes [IQR: 2.02] vs. 4.43 minutes [IQR: 0.69], p = 0.752), while inexperienced staff saw a significant reduction in review time after familiarization with the dashboard (5.06 minutes [IQR: 1.51] vs. 4.12 minutes [IQR: 1.99], p = 0.034). Subjective feedback highlighted the need for further optimization of the dashboard to align with the workflow of allied health staff to achieve similar efficiency benefits. Conclusion A disease-focused dashboard significantly reduced physician previsit review time while that for clinic staff remained unchanged. Validation studies are necessary with our patient populations to explore further qualitative impacts on patient care efficiency and long-term benefits on workflow.

Objective: To implement use of obstetric (OB) hospitalist telemedicine services (TeleOB) to support clinicians facing OB emergencies in low-resource hospital settings. Methods: TeleOB was staffed by OB hospitalists working at a tertiary maternity center. The service was available via real-time high-definition audio/video technology for providers at 17 outlying hospitals across a health system spanning two states. The initial 25 service activations are described. Results: TeleOB supported 17 deliveries, two postpartum emergency department (ED) consultations, and four antenatal ED consultations. In 10 of 17 (59%) deliveries, teleneonatology was jointly activated to support neonatal resuscitation. Sixteen (94%) deliveries occurred in multiparas, and five (29%) resulted from spontaneous preterm labor. Eighty percent (20/25) of activations occurred in facilities without maternity services. Conclusions: A TeleOB service staffed by OB hospitalists successfully supports hospitals in an integrated health care system. TeleOB is feasible for support of hospitals with no delivery facilities or with limited maternity care resources.

Introduction: Use of validated risk stratification tools for work up of suspected pulmonary embolism (PE) is an endorsed recommendation of the American Society of Hematology and the American College of Emergency Physicians. Due to variations in clinical practice, electronic health record (EHR) systems, and medical provider trainings, the use, documentation, and implementation of these tools vary widely. Current processes do not allow for electronic capture and quality reporting at most institutions. Methods: We sought to design and implement clinical pre-test probability (PTP) tools at three large healthcare institutions for use in emergency departments (ED). At each site, a hematologist partnered with ED physician leadership and local information technology experts to design and implement PTP tools in clinical workflows. Overall, 38 EDs (high volume tertiary EDs and smaller regional EDs) located in the United States along the East Coast and Midwest regions were involved. A major goal of implementation was integration into clinical workflow and automatic documentation. After design and implementation, awareness and education of the new EHR tool was distributed electronically to ED providers. After implementation, use of PTP tools was analyzed between September 12, 2022 and January 11, 2023. PTP use was examined as percent of visits for which patients underwent CT pulmonary angiography (CTPA). Results: Each institution chose the 3 tier Wells' Score for implementation while one site designed an integrated Wells'/PERC or YEARS score calculator. Two sites implemented the tool as an optional flowsheet and one site implemented the tool as mandatory parameters within the orders for PE imaging studies. The mandatory tool could be bypassed with prespecified selections or if a D-dimer existed within 48 hours . Complete details of the site-specific implementation are shown in Table 1. Over the 4-month evaluation timeframe, there was a total of 270,214 ED encounters. Uptake and utilization of the PTP tools are shown in Table 2. Use of PTP was highest at the site with forced PTP documentation which ranged from 49-53% of ordered CTPAs, compared to Site 2 where use was 2-3%, and Site 1 where use ranged from 1-3%. At Site 1, use of PTP increased slightly over the study period with signs that PE yield on imaging was increasing as well (3.4% to 5.9%). At Site 2, PE yield on imaging was overall high (9-10%), and remained similar with similar use of PTP tools over the study period. Use of PTP and PE yield (6-8%) on imaging also remained similar throughout the study at Site 3. Conclusion: Forced use of PTP within the orders for CTPA led to the highest use of PTP (Site 3) but this did not result in meaningfully higher PE yield on CTPA compared to the other sites. PE yield at each site was at or above the 5% yield for PE on CTPA which has been previously reported. Given that PTP uptake was relatively stable over the measurement period at Site 2 and 3, this indicates the process had mostly stabilized and that other strategies are needed to improve uptake of PTP, reduce utilization of CTPA, and further increase yield on CTPA. Increasing PTP use and rising PE yield at Site 1 demonstrated ongoing potential for improvements past this study timeframe.

Introduction Pulmonary Embolism (PE) is a major cause of morbidity and mortality worldwide. Diagnosis is essential as untreated PE has a 30% mortality rate. Currently, computer tomography pulmonary angiogram (CTPA) is the standard of care to diagnose PE. However, this modality is expensive, exposes patients to radiation and increases risk of contrast induced nephropathy. Furthermore, only 5% of CTPAs performed are positive for PE signifying that many unnecessary tests are being performed. To address this gap, Pre-Test Probability (PTP) tools have been developed to safely exclude patients with low risk of PE and limit iatrogenic harm from over testing. Despite the tools being developed over 20 years ago, they are not widely available in electronic health record (EHR) systems and even when they are built in, uptake remains low. Methods Three large integrated health systems using Epic EHR systems across five states (MA, WI. MN, AZ, FL) built electronic PTP assessments into their EHR systems to safely exclude patients with low risk of PE and limit over testing. Two sites made the PTP optional, and one site made it a forced function (providers were forced to complete PTP prior to ordering CTPA unless a D-dimer result was present in the prior 48 hours). To assess the usability of these EHR tools, the team conducted a survey with ED clinicians across the three test sites. The survey was created to identify which PE risk stratification methods were employed by ED clinicians (EHR embedded PTP tool, apps on their phone, web applications like MDcalc, calculations from memory, or if they relied on clinical gestalt), and to identify barriers to using the EHR embedded PTP. Results Site 1 had 47 respondents, Site 2 had 94 respondents, and Site 3 had 16 respondents. Two key findings emerged from the survey (Figure 1). First, clinicians used a variety of risk stratification approaches and few of them documented their risk assessments in structured fields in the medical record. Second, clinical gestalt was the most used PE risk stratification approach, and this was true across all three test sites. In addition, the health system with a PTP prompt before clinicians could proceed with CTPA orders (Site 3) did not have a higher percentage of clinicians who reported using the EHR embedded tool than sites that did not prompt clinicians to fill in the PTP. Clinicians cited a variety of different barriers to using the EHR embedded PTP tool (Table 1). Conclusion While many clinicians indicated intermittent use of PTP criteria to aid in the diagnosis of PE, risk factors and scores from phone apps, web tools, and calculations from memory are unlikely to be documented in the patient's medical record, which means that they are not accessible to all members of the ED care team and cannot be used for electronic clinical quality measure. The biggest hurdles to increasing the use of structured PTP assessments in the ED setting to safely exclude PE and limit iatrogenic harm from over testing are (1) obtaining buy-in from ED clinicians that the PTP tools provide value over clinical gestalt, at least for a subset of patients who could most benefit, and (2) identifying best practices for incorporating PTP tools into EHR systems that make the tools easier to access and use 3) developing and implementing effective education and awareness programs around the PTP tools. This survey adds to the body of literature on PTP assessments in the ED setting and can help inform further research that is needed to continue to improve the diagnosis of PE. Funding: This project is funded in part by the Gordon and Betty Moore Foundation through Grant GBMF10778 to the American Society of Hematology.

AbstractObjectivesPatients with limited English proficiency (LEP) have been shown to experience disparities in emergency department (ED) care. The objectives of this study were to examine the associations between LEP and irregular ED departures and return ED visits.MethodsWe conducted a multicenter cross‐sectional analysis of 18 EDs within an integrated health system in the upper Midwest from January 1, 2018, to December 31, 2021. ED visits of pediatric and adult patients who were discharged on the index visit were included for analysis. We analyzed the association of LEP with irregular departures, 72‐h and 7‐day return visits, and ED disposition at the time of that return visit. Multivariable model associations were calculated using generalized estimating equations and reported as odds ratios (OR) with 95% confidence intervals (CIs).ResultsA total of 745,464 total ED visits were analyzed, including 27,906 (3.7%) visits among patients with LEP. The most common preferred languages among patients with LEP were Spanish (12,759; 45.7%), Somali (4978; 17.8%), and Arabic (3185; 11.4%). After multivariable adjustment there were no differences in proportions of irregular departures (OR 1.09, 95% CI 0.99–1.21), 72‐h returns (OR 0.99, 95% CI 0.92–1.06), or 7‐day returns (OR 0.99, 95% CI 0.93–1.05) between patients with LEP or English proficiency. Patients with LEP returning within 72 h (OR 1.19, 95% CI 1.01–1.40) and 7 days (OR 1.15, 95% CI 1.01–1.33) were more likely to be admitted to the hospital.ConclusionsAfter multivariable adjustment, we did not find an increased frequency of irregular ED departures or 72‐h or 7‐day returns among patients with LEP compared with people proficient in English. However, we did find that higher proportions of patients with LEP were admitted to the hospital at the time of the return ED visit.