Romero‐Brufau, Santiago

Objective To compare the diagnostic accuracy of four available automated electronic medical record (EMR) retrieval methods, including a large language model (LLM)-assisted workflow, against manual chart adjudication for identifying cardiovascular events. Design Retrospective diagnostic accuracy study. Setting Three sites within a single US tertiary health system. Participants Two adult cohorts with previously adjudicated cardiovascular outcomes were included. Cohort 1 included 2258 patients treated with immune checkpoint inhibitors, and Cohort 2 included 1426 patients who underwent transcatheter aortic valve replacement. Primary and secondary outcome measures The reference standard was clinician manual chart adjudication. Outcomes included ischaemic stroke or transient ischaemic attack, myocardial infarction (MI), heart failure (HF) exacerbation or hospitalisation and a composite major adverse cardiovascular events (MACE) outcome. Automated retrieval methods included International Classification of Diseases (ICD) codes, primary diagnosis, problem list and a zero-shot LLM workflow. Area under the (receiver operating characteristic) curve (AUC), sensitivity, specificity and net reclassification improvement were assessed. Results In Cohort 1, the LLM achieved the highest AUC for stroke (0.920; 95% CI 0.881 to 0.958), MI (0.938; 95% CI 0.905 to 0.971) and composite MACE (0.880; 95% CI 0.854 to 0.907), whereas ICD-based retrieval had a higher AUC for HF (0.882; 95% CI 0.845 to 0.918 vs 0.873; 95% CI 0.831 to 0.914). In Cohort 2, the LLM achieved the highest AUC for all evaluated outcomes: stroke (0.915; 95% CI 0.862 to 0.968), MI (0.928; 95% CI 0.839 to 1.000), HF (0.844; 95% CI 0.803 to 0.884) and composite MACE (0.862; 95% CI 0.829 to 0.895). In Cohort 1, differences in AUC between the LLM and ICD methods were not statistically significant across outcomes, whereas in Cohort 2 the LLM showed significantly higher AUC for stroke and composite MACE. Conclusion In this multisite retrospective validation study, the LLM-assisted workflow showed strong but context-dependent performance for identifying cardiovascular events from the EMR. Performance varied by outcome and cohort, and ICD-based retrieval remained competitive for some use cases. These findings support a complementary role for LLM-assisted extraction in retrospective cardiovascular outcomes research.

Abstract Objectives To address the gap between artificial intelligence (AI) model development and clinical implementation by developing a machine-learning model to predict cochlear implant (CI) eligibility based on routine hearing tests, while illustrating 3 practical lessons: reformulation for clinical utility, selection of problem-specific metrics, and handling input data variability. Materials and Methods Data were extracted from adult patients who underwent behavioral audiometric and subsequent CI candidacy testing (Arizona Biomedical and Consonant-Nucleus-Consonant tests) at Mayo Clinic between 2011 and 2023. Regression models were initially developed to predict AzBio and CNC scores using features from routine audiograms. Models were then reformulated into binary classification tasks using clinical thresholds (AzBio ≤60%; CNC <50%) and reported sensitivity and positive predictive value (PPV). Input data variability was assessed by analyzing test-retest differences in isophonemes scores. Results Binary classification models achieved sensitivity of 90.4% with PPV 80.2% at CNC 50/AzBio 60, outperforming referral heuristics (60/60 and 75/40). Applying the models to 50 700 historical audiograms identified 525 patients at PPV 90% (sensitivity 58%), 396 at PPV 94% (sensitivity 44%), and 1582 at PPV 71% (sensitivity 94%). Variability analyses indicated that attainable precision is bounded by noise in key predictors. Conclusion This study demonstrates how AI models can be translated into clinically actionable tools by reformulating regression outputs, selecting metrics aligned with clinical priorities, and addressing data variability. These strategies provide a scalable framework for implementing AI in health care and improving decision-making in diverse clinical contexts.

Objective: To explore the existing evidence on novel imaging modalities that may aid in prognostication of vestibular schwannoma (VS) clinical outcomes across various management approaches. Study design: Scoping review. Setting: Not applicable. Patients: Individuals diagnosed with VS. Interventions: Novel imaging techniques. Main outcome measures: Tumor growth and response to medical therapy. Results: A total of 18 articles were included in this review. Of these, 7 focused on pretreatment prognosis, 6 evaluated radiosurgery outcomes, 4 examined microsurgery outcomes, and 1 assessed response to bevacizumab. The imaging modalities investigated included radiomics (n = 10), positron emission tomography (PET; n = 2), dynamic contrast-enhanced MRI (DCE-MRI; n = 2), both PET and DCE-MRI (n = 1), magnetic resonance elastography (MRE; n = 1), contrast-enhanced CT (CE-CT; n = 1), and slip interface imaging (SII; n = 1). Radiomic studies yielded encouraging results in predicting tumor growth after wait-and-scan and radiosurgery, with reported accuracies ranging from 52% to 88%, sensitivities from 13% to 95%, specificities from 50% to 94%, and area under the curves from 0.65 to 0.99. PET studies identified several promising ligands associated with tumor growth during wait-and-scan, suggesting a potential role in prognostication. DCE-MRI also demonstrated associations with tumor growth and treatment response to bevacizumab. Studies on MRE, CE-CT, and SII identified novel imaging parameters associated with tumor stiffness and adhesion. Conclusions: Several emerging imaging modalities hold potential for improving clinical outcome prognostication in VS. Techniques such as radiomics, PET, and DCE-MRI have shown promising results; however, the supporting evidence is still limited and heterogeneous. Further research is necessary to validate and refine these approaches to enhance their clinical utility.

Objective: To develop, implement, and evaluate a novel process used for residency application review that deemphasizes metrics known to create bias with the goal of interviewing and matching a more diverse resident cohort. Methods: Between 2020 and 2023, a novel standardized rubric and application review process were developed and implemented at the authors’ academic training program. The rubric deemphasized USMLE scores, honor society membership, and number of publications while utilizing an AI-driven pre-sort of applications, facilitating holistic review. This weighted applicant attributes identified in personal statements, life experiences, achievements in community, leadership roles, otolaryngology-specific impressions highlighted in letters of recommendation, and otolaryngology-specific research and publications similarly to academic metrics (clerkship grades and standardized test scores). Demographics of applicants interviewed and matched into our program were compared pre- and post-implementation using descriptive statistics. Results: Using a standardized rubric to review residency applications that deemphasizes traditional metrics, the authors’ otolaryngology residency program has interviewed, ranked, and matched more diverse candidates. As a result, from 2019 to 2022, the overall resident cohort has diversified from 4% historically marginalized residents to 24%, from 28% female residents to 44%, and from 33% of residents outside our geographic region to 66%. Conclusion: There is significant underrepresentation among otolaryngology residents. Selection based upon USMLE Step 1 scores, honor society membership, and number of publications have been shown to restrict diversity. Purposeful holistic review increases diversity in interviewed and matched resident cohort.

AbstractObjectiveTo report the first steps of a project to automate and optimize scheduling of multidisciplinary consultations for patients with longstanding dizziness utilizing artificial intelligence.Study DesignRetrospective case review.SettingQuaternary referral center.MethodsA previsit self‐report questionnaire was developed to query patients about their complaints of longstanding dizziness. We convened an expert panel of clinicians to review diagnostic outcomes for 98 patients and used a consensus approach to retrospectively determine what would have been the ideal appointments based on the patient's final diagnoses. These results were then compared retrospectively to the actual patient schedules. From these data, a machine learning algorithm was trained and validated to automate the triage process.ResultsCompared with the ideal itineraries determined retrospectively with our expert panel, visits scheduled by the triage clinicians showed a mean concordance of 70%, and our machine learning algorithm triage showed a mean concordance of 79%.ConclusionManual triage by clinicians for dizzy patients is a time‐consuming and costly process. The formulated first‐generation automated triage algorithm achieved similar results to clinicians when triaging dizzy patients using data obtained directly from an online previsit questionnaire.

ObjectiveThe purpose of this study was to evaluate the performance of advanced large language models from OpenAI (GPT‐3.5 and GPT‐4), Google (PaLM2 and MedPaLM), and an open source model from Meta (Llama3:70b) in answering clinical test multiple choice questions in the field of otolaryngology—head and neck surgery.MethodsA dataset of 4566 otolaryngology questions was used; each model was provided a standardized prompt followed by a question. One hundred questions that were answered incorrectly by all models were further interrogated to gain insight into the causes of incorrect answers.ResultsGPT4 was the most accurate, correctly answering 3520 of 4566 questions (77.1%). MedPaLM correctly answered 3223 of 4566 (70.6%) questions, while llama3:70b, GPT3.5, and PaLM2 were correct on 3052 of 4566 (66.8%), 2672 of 4566 (58.5%), and 2583 of 4566 (56.5%) questions. Three hundred and sixty‐nine questions were answered incorrectly by all models. Prompts to provide reasoning improved accuracy in all models: GPT4 changed from incorrect to correct answer 31% of the time, while GPT3.5, Llama3, PaLM2, and MedPaLM corrected their responses 25%, 18%, 19%, and 17% of the time, respectively.ConclusionLarge language models vary in their understanding of otolaryngology‐specific clinical knowledge. OpenAI's GPT4 has a strong understanding of core concepts as well as detailed information in the field of otolaryngology. Its baseline understanding in this field makes it well‐suited to serve in roles related to head and neck surgery education provided that the appropriate precautions are taken and potential limitations are understood.Level of EvidenceNA Laryngoscope, 135:629–634, 2025

ABSTRACT Errors in antibiotic prescriptions are frequent, often resulting from the inadequate coverage of the infection-causative microorganism. The efficacy of iAST, a machine-learning-based software offering empirical and organism-targeted antibiotic recommendations, was assessed. The study was conducted in a 12-hospital Spanish institution. After model fine-tuning with 27,531 historical antibiograms, 325 consecutive patients with acute infections were selected for retrospective validation. The primary endpoint was comparing each of the top three of iAST’s antibiotic recommendations’ success rates (confirmed by antibiogram results) with the antibiotic prescribed by the physicians. Secondary endpoints included examining the same hypothesis within specific study population subgroups and assessing antibiotic stewardship by comparing the percentage of antibiotics recommended that belonged to different World Health Organization AWaRe groups within each arm of the study. All of iAST first three recommendations were non-inferior to doctor prescription in the primary endpoint analysis population as well as the secondary endpoint. The overall success rate of doctors’ empirical treatment was 68.93%, while that of the first three iAST options was 91.06% ( P < 0.001), 90.63% ( P < 0.001), and 91.06% ( P < 001), respectively. For organism-targeted therapy, the doctor’s overall success rate was 84.16%, and that of the first three ranked iAST options was 97.83% ( P < 0.001), 94.09% ( P < 0.001), and 91.30% ( P < 0.001), respectively. In empirical therapy, compared to physician prescriptions, iAST demonstrated a greater propensity to recommend access antibiotics, fewer watch antibiotics, and higher reserve antibiotics. In organism-targeted therapy, iAST advised a higher utilization of access antibiotics. The present study demonstrates iAST accuracy in predicting antibiotic susceptibility, showcasing its potential to promote effective antibiotic stewardship. CLINICAL TRIALS This study is registered with ClinicalTrials.gov as NCT06174519 .

Abstract Background A vast amount of potentially useful information such as description of patient symptoms, family, and social history is recorded as free-text notes in electronic health records (EHRs) but is difficult to reliably extract at scale, limiting their utility in research. This study aims to assess whether an “out of the box” implementation of open-source large language models (LLMs) without any fine-tuning can accurately extract social determinants of health (SDoH) data from free-text clinical notes. Methods We conducted a cross-sectional study using EHR data from the Mass General Brigham (MGB) system, analyzing free-text notes for SDoH information. We selected a random sample of 200 patients and manually labeled nine SDoH aspects. Eight advanced open-source LLMs were evaluated against a baseline pattern-matching model. Two human reviewers provided the manual labels, achieving 93% inter-annotator agreement. LLM performance was assessed using accuracy metrics for overall, mentioned, and non-mentioned SDoH, and macro F1 scores. Results LLMs outperformed the baseline pattern-matching approach, particularly for explicitly mentioned SDoH, achieving up to 40% higher Accuracymentioned. openchat_3.5 was the best-performing model, surpassing the baseline in overall accuracy across all nine SDoH aspects. The refined pipeline with prompt engineering reduced hallucinations and improved accuracy. Conclusions Open-source LLMs are effective and scalable tools for extracting SDoH from unstructured EHRs, surpassing traditional pattern-matching methods. Further refinement and domain-specific training could enhance their utility in clinical research and predictive analytics, improving healthcare outcomes and addressing health disparities.

Sepsis is considered a medical emergency where delays in initial treatment are associated with increased morbidity and mortality, yet there is no gold standard for identifying sepsis onset and thus treatment timing. We leverage electronic health record (EHR) data with clinical expertise to develop a continuous-time Markov decision process (MDP) optimal stopping model that identifies the optimal first intervention action (anti-infective, fluid, or wait). To study the impact of initial treatment of patients at risk for developing sepsis, we define the delayed treatment population who received delayed treatment upon admission or during hospitalization and serves as an approximation of the natural history of sepsis. We apply the optimal first treatment policy to sample patient visits from the nondelayed treatment population. This analysis indicates the average risk of death could be reduced by approximately 2.2%, the average time until treatment could be reduced by 106 minutes, and the average severity of the treatment state could be reduced by 15.5% compared with the treatment they received in the hospital. We study the properties of the optimal policy to define an easily interpretable initial treatment heuristic that considers a patient’s organ dysfunction, location, and septic shock status. This generalizable framework can inform personalized treatment of patients at risk for sepsis. History: This paper has been accepted for the Decision Analysis Special Issue on Emerging Topics in Health Decision Analysis. Funding: This material is based upon work supported by the National Science Foundation [Grant 1522107 (North Carolina State University), 1522106 (Mayo Clinic), and 1833538 (Drexel University)].