Cook, David A.

Abstract Objectives This study used both researcher and patient perspectives to identify high-priority research questions related to clinical reasoning. Methods This study integrates findings from two previously published research prioritization exercises aimed at advancing the science of diagnostic safety, focusing on questions relevant to clinical reasoning. In Study 1 (April 2019), 97 researchers submitted 177 unique research questions that could be useful “to reduce patient harm in the diagnosis process”. In Study 2 (November 2019), 41 patients/caregivers/advocates submitted 84 unique research questions on the same topic. In an in-person meeting, a subset of these participants identified 81 of these questions as high-priority (51 from Study 1; 30 from Study 2). For the current study, five investigators identified questions that focused on clinical reasoning, first independently and then by consensus. They then categorized the clinical reasoning questions into inductively identified themes. Results Twenty-nine of the 81 high-priority research questions addressed clinical reasoning and were categorized into 5 themes: understanding clinical reasoning and cognition (12 questions), decision support/technology (6 questions), team-based decision making (5 questions), how to learn from cases/scenarios (3 questions), and patient engagement (3 questions). Questions addressed a range of basic/foundational science and clinical applications, and involved individuals, teams, and/or systems. Conclusions Identification of high-priority clinical reasoning research questions can help medical educators, clinicians, researchers, and funders catalyze research and discovery of interventions to improve clinical reasoning and ultimately improve patient outcomes.

Background: Longitudinal assessments are increasingly common as a component of continuing certification programs offered by medical specialty boards. The goal of this study was to understand anesthesiologists’ utilization pattern, performance, and perception of the longitudinal assessment program administered by the American Board of Anesthesiology (Raleigh, North Carolina). Methods: The authors analyzed assessment usage and survey data from all anesthesiologists who participated in the Maintenance of Certification in Anesthesiology (MOCA) Minute from January 2018 to December 2019. Results: During the 24-month study period, 25,404 anesthesiologists completed all 120 questions assigned in at least 1 calendar year, collectively 5,801,160 responses to 1,646 unique questions. Participants were more likely to complete questions at the end of each quarter, with the highest frequency at the end of the year. Seventy-five percent of responses were first-time attempts, and 17% were first repeats. The median interval between original question presentation and first repeat was 314 days. The mean ± SD percent correct score was 68.3 ± 8.0% for first-time questions and 75.5 ± 12.6% for first repeats. The mean ± SD time to answer a first-time question was 26.8 ± 13.1 s, with shorter times for questions with correct than incorrect responses (25.0 ± 12.7 vs . 30.7 ± 13.3 s; time difference, −5.69 s; 95% CI, −5.71 to −5.66; P < 0.001). Of 5.11 million responses with ratings of relevance (88.2% of all responses analyzed), 4.86 million (95.2%) were felt to be somewhat or very relevant to practice. The percent correct scores were higher for responses with higher relevance and confidence ratings. For responses rated “not relevant,” “somewhat relevant,” and “very relevant,” the percent correct was 58.5%, 67.6%, and 77.0%, respectively; for responses rated as “unsure,” “somewhat confident,” and “very confident,” the percent correct was 53.6%, 67.9%, and 84.4%, respectively. Conclusions: MOCA Minute was generally used as designed. Evidence of learning was suggested by improved performance on repeat questions.

ABSTRACT Objectives The increasing volume of medical education research necessitates efficient, reliable, and scalable methods for conducting quality appraisals. The Medical Education Research Study Quality Instrument (MERSQI) is a widely used tool, although its manual scoring process remains resource‐intensive. This study evaluated how well large language models (LLMs) appraise medical education research using the MERSQI tool in comparison with human judges. Methods Three LLMs (GPT‐5, Claude Sonnet 4, and Gemini 2.5 Pro) assigned MERSQI domain scores to 1423 medical education research articles. The authors compared AI‐generated scores with human‐generated scores using intraclass correlation coefficients (ICCs) across the six MERSQI domains. They evaluated the agreement between AI‐ and human‐generated MERSQI composite scores using Bland–Altman plots. Results Domain‐level ICC values ranged from fair (0.24) to near perfect (0.81), with the lowest agreement observed in the ‘sampling,’ ‘validity evidence,’ and ‘data analysis’ domains. No single LLM consistently outperformed the others across all domains. Composite score agreement with human ratings was substantial and similar across LLMs (ICC range: 0.65–0.69). GPT‐5 produced slightly lower composite scores than humans, while Claude Sonnet 4 and Gemini 2.5 Pro produced higher scores, with Gemini showing the largest deviation. The Bland–Altman plots for Gemini 2.5 Pro suggested proportional bias, indicating its agreement with human scores varied across the range of study quality. Conclusions These LLMs demonstrated substantial agreement with human raters for MERSQI composite scores, but domain‐level agreement varied. Systematic differences in scoring patterns highlight the need for human oversight and additional calibrations before integrating LLMs into systematic review appraisal workflows.

Abstract Introduction Evaluate large language models (LLMs) for scoring medical student essays, and compare various prompting techniques and models. Methods OpenAI GPT scored 51 medical student reflection essays (15 real, 36 fabricated) using a previously-reported 6-point rubric (April-May 2025). We compared 29 prompt-model conditions by systematically varying the LLM prompts (including the persona, scoring rubric, few-shot learning [exemplars], chain-of-thought reasoning, and temperature), fine-tuning, and model (including GPT-4.1, GPT-4.1-mini, GPT-o4-mini, and GPT-4-Turbo). Outcomes were accuracy (compared with human raters, measured using single-score intraclass correlation coefficient [ICC] and mean absolute difference [MAD; zero indicates perfect agreement]), within-condition reproducibility, and cost. Results Across all conditions, it took mean (SD) 3.73 (3.12) seconds to score 1 essay. The cost to score 100 essays was USD $0.04 for GPT-4.1-mini, $0.21 for GPT-4.1, $0.57 for GPT-4.1 with 3 exemplars, and $2.00 for fine-tuned GPT-4.1. When the one-time cost of fine-tuning was amortized across 10,000 essays, the cost for fine-tuned GPT-4.1 was $0.20 per 100. Accuracy was “almost perfect” (ICC >0.80) for 28/29 conditions (97%). Fine-tuned models were more accurate than non-fine-tuned models (MAD difference –0.24 [95% CI, –0.34, –0.14]). Conditions with exemplars were more accurate than those without (MAD difference –0.44 [CI, –0.57, –0.31]). Accuracy progressively decreased as 6, 3, 1, and 0 rubric levels were explicitly defined in the prompt (P<.001). Contrary to hypotheses, accuracies for chain-of-thought prompts and variations in temperature and persona were not significantly different from the baseline prompt. Reproducibility ICC was >0.80 for 28/29 conditions (97%). Discussion Automated LLM essay scoring demonstrated near-perfect accuracy and reproducibility for most prompt-model conditions. Fine-tuned models and prompts with exemplars had higher accuracy but higher cost. Fine-tuned models had lower per-essay costs for larger essay volumes. For smaller volumes, non-fine-tuned GPT-4.1 provided excellent results at moderate cost. GPT-4.1-mini provided very good results at low cost.

Abstract Purpose Supervisors may be prone to implicit (unintentional) bias when granting procedural autonomy to trainees due to the subjectivity of autonomy decisions. The authors aimed to conduct a systematic review and meta-analysis to assess the differences in perceptions of procedural autonomy granted to physician trainees based on gender and/or race. Method MEDLINE, Embase, CENTRAL, Scopus, and Web of Science were searched (search date: January 5, 2022) for studies reporting quantitative gender- or race-based differences in perceptions of procedural autonomy of physician trainees. Reviewers worked in duplicate for article selection and data abstraction. Primary measures of interest were self-reported and observer-rated procedural autonomy. Meta-analysis pooled differences in perceptions of procedural autonomy based on trainee gender. Results The search returned 2,714 articles, of which 16 were eligible for inclusion. These reported data for 6,109 trainees (median, 90 per study) and 2,763 supervisors (median, 54 per study). No studies investigated differences in perceptions of autonomy based on race. In meta-analysis of disparities between genders in autonomy ratings (positive number favoring female trainees), pooled standardized mean differences were −0.12 (95% confidence interval [CI], −0.19 to −0.04; P = .003; n = 10 studies) for trainee self-rated autonomy and −0.05 (95% CI, −0.11 to 0.01; P = .07; n = 9 studies) for supervisor ratings of autonomy. Conclusions Limited evidence suggests that female trainees perceived that they received less procedural autonomy than did males. Further research exploring the degree of gender- and race-based differences in procedural autonomy, and factors that influence these differences is warranted.

Abstract Purpose The Optimizing Performance Through Intrinsic Motivation and Attention for Learning (OPTIMAL) theory postulates that directing attention to the intended movement effect or outcome (external focus) is more effective than directing attention to the internal body motion or body part (internal focus). This study compared external versus internal focus of attention as novice students learned ultrasound-guided peripheral cannulation. The authors hypothesized that conditions promoting an external (vs internal) focus of attention would result in better performance on immediate and delayed (retention) testing. Method The authors conducted a randomized, experimental study from October 2022 to February 2023 comparing external and internal focus using ultrasound-guided peripheral venous cannulation followed by practice using an ultrasound training block. Undergraduate health science students performed 4 practice trials with instructions to focus on the needle tip (external focus) or their hands (internal focus), followed by immediate and delayed (retention) assessment 2 weeks later. The primary outcome was time to successful cannulation. Results were compared using Kaplan-Meier curves and parametric survival analysis regression (accelerated failure time) models. Results Seventy participants completed the immediate assessment. The external focus group completed successful cannulation a mean of 2.33 times faster (95% confidence interval [CI], 1.47–3.67; P &lt; .001) than those in the internal focus group (median time, 12.0 vs 29.5 seconds). Results also favored the external focus group during practice, with the external focus group successfully completing the task a mean of 1.83 times faster (95% CI, 1.04–3.21; P = .04) than the internal focus group. In the retention assessment, the external focus group successfully completed the task a mean of 2.37 times faster (95% CI, 1.19–4.74; P = .02) than the internal focus group. Conclusions External focus of attention compared with internal focus of attention improves medical motor skill learning in novice learners.