Laack, Torrey A.

Abstract Introduction Evaluate large language models (LLMs) for scoring medical student essays, and compare various prompting techniques and models. Methods OpenAI GPT scored 51 medical student reflection essays (15 real, 36 fabricated) using a previously-reported 6-point rubric (April-May 2025). We compared 29 prompt-model conditions by systematically varying the LLM prompts (including the persona, scoring rubric, few-shot learning [exemplars], chain-of-thought reasoning, and temperature), fine-tuning, and model (including GPT-4.1, GPT-4.1-mini, GPT-o4-mini, and GPT-4-Turbo). Outcomes were accuracy (compared with human raters, measured using single-score intraclass correlation coefficient [ICC] and mean absolute difference [MAD; zero indicates perfect agreement]), within-condition reproducibility, and cost. Results Across all conditions, it took mean (SD) 3.73 (3.12) seconds to score 1 essay. The cost to score 100 essays was USD $0.04 for GPT-4.1-mini, $0.21 for GPT-4.1, $0.57 for GPT-4.1 with 3 exemplars, and $2.00 for fine-tuned GPT-4.1. When the one-time cost of fine-tuning was amortized across 10,000 essays, the cost for fine-tuned GPT-4.1 was $0.20 per 100. Accuracy was “almost perfect” (ICC >0.80) for 28/29 conditions (97%). Fine-tuned models were more accurate than non-fine-tuned models (MAD difference –0.24 [95% CI, –0.34, –0.14]). Conditions with exemplars were more accurate than those without (MAD difference –0.44 [CI, –0.57, –0.31]). Accuracy progressively decreased as 6, 3, 1, and 0 rubric levels were explicitly defined in the prompt (P<.001). Contrary to hypotheses, accuracies for chain-of-thought prompts and variations in temperature and persona were not significantly different from the baseline prompt. Reproducibility ICC was >0.80 for 28/29 conditions (97%). Discussion Automated LLM essay scoring demonstrated near-perfect accuracy and reproducibility for most prompt-model conditions. Fine-tuned models and prompts with exemplars had higher accuracy but higher cost. Fine-tuned models had lower per-essay costs for larger essay volumes. For smaller volumes, non-fine-tuned GPT-4.1 provided excellent results at moderate cost. GPT-4.1-mini provided very good results at low cost.

Abstract Purpose The Optimizing Performance Through Intrinsic Motivation and Attention for Learning (OPTIMAL) theory postulates that directing attention to the intended movement effect or outcome (external focus) is more effective than directing attention to the internal body motion or body part (internal focus). This study compared external versus internal focus of attention as novice students learned ultrasound-guided peripheral cannulation. The authors hypothesized that conditions promoting an external (vs internal) focus of attention would result in better performance on immediate and delayed (retention) testing. Method The authors conducted a randomized, experimental study from October 2022 to February 2023 comparing external and internal focus using ultrasound-guided peripheral venous cannulation followed by practice using an ultrasound training block. Undergraduate health science students performed 4 practice trials with instructions to focus on the needle tip (external focus) or their hands (internal focus), followed by immediate and delayed (retention) assessment 2 weeks later. The primary outcome was time to successful cannulation. Results were compared using Kaplan-Meier curves and parametric survival analysis regression (accelerated failure time) models. Results Seventy participants completed the immediate assessment. The external focus group completed successful cannulation a mean of 2.33 times faster (95% confidence interval [CI], 1.47–3.67; P &lt; .001) than those in the internal focus group (median time, 12.0 vs 29.5 seconds). Results also favored the external focus group during practice, with the external focus group successfully completing the task a mean of 1.83 times faster (95% CI, 1.04–3.21; P = .04) than the internal focus group. In the retention assessment, the external focus group successfully completed the task a mean of 2.37 times faster (95% CI, 1.19–4.74; P = .02) than the internal focus group. Conclusions External focus of attention compared with internal focus of attention improves medical motor skill learning in novice learners.

Objective: The novel ultrasound magnetic needle navigation technique can visualize the entire needle and identify its projected trajectory. We hypothesized that this technique increases the first-attempt success rate of central venous puncture by novice learners compared with the conventional needle navigation technique. Methods: This prospective, randomized, controlled trial with a crossover design included 50 participants with limited prior experience in US-guided procedures. Participants were randomly assigned to novel or conventional technique groups and asked to perform central venous cannulation in a phantom task trainer. After the first successful attempt, participants were allocated to the other technique group. Results: Although participants in the novel technique group had a higher first-attempt success rate than did those in the conventional technique group, this difference was not statistically significant ( p = 0.17). The total number of attempts also did not significantly differ ( p = 0.16). The conventional technique group had more needle redirections ( p = 0.01) and a longer time to successful cannulation ( p = 0.01). The number of adverse effects ( p = 0.32) did not differ between groups. Participant confidence levels were higher in the novel technique group ( p < 0.001). Conclusions: Magnetic needle navigation can reduce the number of needle redirections, shorten the time to successful cannulation, and increase confidence levels by novice learners for successful US-guided central venous access.

Abstract Background The ‘OPTIMAL’ (Optimizing Performance Through Intrinsic Motivation for Learning) theory of motor learning suggests that autonomy, external focus of attention, and perceived competence can improve learning of simple motor tasks. The authors hypothesized that enhanced (vs. routine) autonomy and external (vs. internal) focus of attention would improve first-try performance of two medical motor tasks.Methods The authors conducted a randomized two-by-two factorial design study with high school students as participants. Task instructions promoted either enhanced or routine autonomy, and either external or internal focus of attention. These conditions were replicated in a crossover design for two common medical tasks (chest compressions on a manikin and a Fundamentals of Laparoscopic Surgery peg transfer task). Primary outcomes were objective measures of task performance (chest compression deviation from target depth; peg transfer time with penalties for errors). Secondary outcomes included subjective perceptions of confidence, autonomy, and workload.Results 133 high school students participated in this study. The primary outcomes concerning enhanced vs. routine autonomy demonstrated no statistically significant difference in either task (chest compression depth deviation: difference − 0.7 mm [score range 0 to 37.5 mm]; 95% confidence interval (CI95) -3.85, 2.41; p = .65; peg transfer penalized time: rate ratio 1.03; CI95 0.91, 1.31; p = .79). The authors likewise found no statistically significant difference for external vs. internal focus of attention (depth deviation: difference 1.1 mm; CI95 -2.04, 4.17; p = .50; penalized time: rate ratio 0.89; CI95 0.75, 1.13; p = .33). The authors found no statistically significant differences for either comparison in confidence, autonomy and workload (p > .09; differences ranged from − 0.83 to 0.79 [scale range 0 to 10]).Conclusions First-try performance of chest compressions and peg transfer by novice learners is not significantly affected by enhanced (vs. routine) autonomy or external (vs. internal) focus of attention.

Abstract Objective: Dyad learning has been shown to be an effective tool for teaching procedural skills, but little is known about how dyad learning may impact the stress, anxiety, and cognitive load that a student experiences when learning in this manner. In this pilot study, we investigate the relationship between dyad training on stress, anxiety, cognitive load, and performance in a simulated bradycardia scenario. Methods: Forty-one fourth-year medical school trainees were randomized as dyads (n=24) or individuals (n=17) for an education session on Day 1. Reassessment occurred on Day 4 and was completed as individuals for all trainees. Primary outcomes were cognitive load (Paas scale), stress (Cognitive Appraisal Ratio), and anxiety levels (abbreviated State-Trait Anxiety Inventory). Secondary outcomes were time-based performance metrics. Results: On Day 1 we observed significant differences for change in anxiety and stress measured before and after the training scenario between groups. Individuals compared to dyads had larger mean increases in anxiety, (19.6 versus 7.6 on 80-point scale, p=0.02) and stress ratio (1.8 versus 0.9, p=0.045). On the Day 4 post-intervention assessment, no significant differences were observed between groups. Secondary outcomes were significant for shorter time to diagnosis of bradycardia (p=0.01) and time to initiation of pacing (p=0.04) in the dyad group on Day 1. On Day 4, only time to recognizing the indication for pacing was significantly shorter for individual training (Hazard ratio [HR] = 2.26, p = 0.02).Conclusions: Dyad training results in lower stress and anxiety levels with similar performance compared to individual training.

AbstractValue‐based health care requires a balancing of medical outcomes with economic value. Administrators need to understand both the clinical and the economic effects of potentially expensive simulation programs to rationalize the costs. Given the often‐disparate priorities of clinical educators relative to health care administrators, justifying the value of simulation requires the use of economic analyses few physicians have been trained to conduct. Clinical educators need to be able to present thorough economic analyses demonstrating returns on investment and cost‐effectiveness to effectively communicate with administrators.At the 2017 Academic Emergency Medicine Consensus Conference “Catalyzing System Change through Health Care Simulation: Systems, Competency, and Outcomes,” our breakout session critically evaluated the cost‐benefit and return on investment of simulation. In this paper we provide an overview of some of the economic tools that a clinician may use to present the value of simulation training to financial officers and other administrators in the economic terms they understand. We also define three themes as a call to action for research related to cost–benefit analysis in simulation as well as four specific research questions that will help guide educators and hospital leadership to make decisions on the value of simulation for their system or program.

AbstractTeams are the building blocks of the healthcare system, with growing evidence linking the quality of healthcare to team effectiveness, and team effectiveness to team training. Simulation has been identified as an effective modality for team training and assessment. Despite this, there are gaps in methodology, measurement, and implementation that prevent maximizing the impact of simulation modalities on team performance. As part of the 2017 Academic Emergency Medicine Consensus Conference “Catalyzing System Change Through Health Care Simulation: Systems, Competency, and Outcomes,” we explored the impact of simulation on various aspects of team effectiveness. The consensus process included an extensive literature review, group discussions, and the conference “workshop” involving emergency medicine physicians, medical educators, and team science experts. The objectives of this work were to: 1) explore the antecedents and processes that support team effectiveness, 2) summarize the current role of simulation in developing and understanding team effectiveness, and 3) identify research targets to further improve team‐based training and assessment, with the ultimate goal of improving healthcare systems.