Carducci, Valentina

Abstract Objectives To address the gap between artificial intelligence (AI) model development and clinical implementation by developing a machine-learning model to predict cochlear implant (CI) eligibility based on routine hearing tests, while illustrating 3 practical lessons: reformulation for clinical utility, selection of problem-specific metrics, and handling input data variability. Materials and Methods Data were extracted from adult patients who underwent behavioral audiometric and subsequent CI candidacy testing (Arizona Biomedical and Consonant-Nucleus-Consonant tests) at Mayo Clinic between 2011 and 2023. Regression models were initially developed to predict AzBio and CNC scores using features from routine audiograms. Models were then reformulated into binary classification tasks using clinical thresholds (AzBio ≤60%; CNC <50%) and reported sensitivity and positive predictive value (PPV). Input data variability was assessed by analyzing test-retest differences in isophonemes scores. Results Binary classification models achieved sensitivity of 90.4% with PPV 80.2% at CNC 50/AzBio 60, outperforming referral heuristics (60/60 and 75/40). Applying the models to 50 700 historical audiograms identified 525 patients at PPV 90% (sensitivity 58%), 396 at PPV 94% (sensitivity 44%), and 1582 at PPV 71% (sensitivity 94%). Variability analyses indicated that attainable precision is bounded by noise in key predictors. Conclusion This study demonstrates how AI models can be translated into clinically actionable tools by reformulating regression outputs, selecting metrics aligned with clinical priorities, and addressing data variability. These strategies provide a scalable framework for implementing AI in health care and improving decision-making in diverse clinical contexts.

Objective: To develop, implement, and evaluate a novel process used for residency application review that deemphasizes metrics known to create bias with the goal of interviewing and matching a more diverse resident cohort. Methods: Between 2020 and 2023, a novel standardized rubric and application review process were developed and implemented at the authors’ academic training program. The rubric deemphasized USMLE scores, honor society membership, and number of publications while utilizing an AI-driven pre-sort of applications, facilitating holistic review. This weighted applicant attributes identified in personal statements, life experiences, achievements in community, leadership roles, otolaryngology-specific impressions highlighted in letters of recommendation, and otolaryngology-specific research and publications similarly to academic metrics (clerkship grades and standardized test scores). Demographics of applicants interviewed and matched into our program were compared pre- and post-implementation using descriptive statistics. Results: Using a standardized rubric to review residency applications that deemphasizes traditional metrics, the authors’ otolaryngology residency program has interviewed, ranked, and matched more diverse candidates. As a result, from 2019 to 2022, the overall resident cohort has diversified from 4% historically marginalized residents to 24%, from 28% female residents to 44%, and from 33% of residents outside our geographic region to 66%. Conclusion: There is significant underrepresentation among otolaryngology residents. Selection based upon USMLE Step 1 scores, honor society membership, and number of publications have been shown to restrict diversity. Purposeful holistic review increases diversity in interviewed and matched resident cohort.

Abstract Background A vast amount of potentially useful information such as description of patient symptoms, family, and social history is recorded as free-text notes in electronic health records (EHRs) but is difficult to reliably extract at scale, limiting their utility in research. This study aims to assess whether an “out of the box” implementation of open-source large language models (LLMs) without any fine-tuning can accurately extract social determinants of health (SDoH) data from free-text clinical notes. Methods We conducted a cross-sectional study using EHR data from the Mass General Brigham (MGB) system, analyzing free-text notes for SDoH information. We selected a random sample of 200 patients and manually labeled nine SDoH aspects. Eight advanced open-source LLMs were evaluated against a baseline pattern-matching model. Two human reviewers provided the manual labels, achieving 93% inter-annotator agreement. LLM performance was assessed using accuracy metrics for overall, mentioned, and non-mentioned SDoH, and macro F1 scores. Results LLMs outperformed the baseline pattern-matching approach, particularly for explicitly mentioned SDoH, achieving up to 40% higher Accuracymentioned. openchat_3.5 was the best-performing model, surpassing the baseline in overall accuracy across all nine SDoH aspects. The refined pipeline with prompt engineering reduced hallucinations and improved accuracy. Conclusions Open-source LLMs are effective and scalable tools for extracting SDoH from unstructured EHRs, surpassing traditional pattern-matching methods. Further refinement and domain-specific training could enhance their utility in clinical research and predictive analytics, improving healthcare outcomes and addressing health disparities.