Pankratz, V. Shane

Abstract Introduction Evaluate large language models (LLMs) for scoring medical student essays, and compare various prompting techniques and models. Methods OpenAI GPT scored 51 medical student reflection essays (15 real, 36 fabricated) using a previously-reported 6-point rubric (April-May 2025). We compared 29 prompt-model conditions by systematically varying the LLM prompts (including the persona, scoring rubric, few-shot learning [exemplars], chain-of-thought reasoning, and temperature), fine-tuning, and model (including GPT-4.1, GPT-4.1-mini, GPT-o4-mini, and GPT-4-Turbo). Outcomes were accuracy (compared with human raters, measured using single-score intraclass correlation coefficient [ICC] and mean absolute difference [MAD; zero indicates perfect agreement]), within-condition reproducibility, and cost. Results Across all conditions, it took mean (SD) 3.73 (3.12) seconds to score 1 essay. The cost to score 100 essays was USD $0.04 for GPT-4.1-mini, $0.21 for GPT-4.1, $0.57 for GPT-4.1 with 3 exemplars, and $2.00 for fine-tuned GPT-4.1. When the one-time cost of fine-tuning was amortized across 10,000 essays, the cost for fine-tuned GPT-4.1 was $0.20 per 100. Accuracy was “almost perfect” (ICC >0.80) for 28/29 conditions (97%). Fine-tuned models were more accurate than non-fine-tuned models (MAD difference –0.24 [95% CI, –0.34, –0.14]). Conditions with exemplars were more accurate than those without (MAD difference –0.44 [CI, –0.57, –0.31]). Accuracy progressively decreased as 6, 3, 1, and 0 rubric levels were explicitly defined in the prompt (P<.001). Contrary to hypotheses, accuracies for chain-of-thought prompts and variations in temperature and persona were not significantly different from the baseline prompt. Reproducibility ICC was >0.80 for 28/29 conditions (97%). Discussion Automated LLM essay scoring demonstrated near-perfect accuracy and reproducibility for most prompt-model conditions. Fine-tuned models and prompts with exemplars had higher accuracy but higher cost. Fine-tuned models had lower per-essay costs for larger essay volumes. For smaller volumes, non-fine-tuned GPT-4.1 provided excellent results at moderate cost. GPT-4.1-mini provided very good results at low cost.

Abstract Purpose The Optimizing Performance Through Intrinsic Motivation and Attention for Learning (OPTIMAL) theory postulates that directing attention to the intended movement effect or outcome (external focus) is more effective than directing attention to the internal body motion or body part (internal focus). This study compared external versus internal focus of attention as novice students learned ultrasound-guided peripheral cannulation. The authors hypothesized that conditions promoting an external (vs internal) focus of attention would result in better performance on immediate and delayed (retention) testing. Method The authors conducted a randomized, experimental study from October 2022 to February 2023 comparing external and internal focus using ultrasound-guided peripheral venous cannulation followed by practice using an ultrasound training block. Undergraduate health science students performed 4 practice trials with instructions to focus on the needle tip (external focus) or their hands (internal focus), followed by immediate and delayed (retention) assessment 2 weeks later. The primary outcome was time to successful cannulation. Results were compared using Kaplan-Meier curves and parametric survival analysis regression (accelerated failure time) models. Results Seventy participants completed the immediate assessment. The external focus group completed successful cannulation a mean of 2.33 times faster (95% confidence interval [CI], 1.47–3.67; P &lt; .001) than those in the internal focus group (median time, 12.0 vs 29.5 seconds). Results also favored the external focus group during practice, with the external focus group successfully completing the task a mean of 1.83 times faster (95% CI, 1.04–3.21; P = .04) than the internal focus group. In the retention assessment, the external focus group successfully completed the task a mean of 2.37 times faster (95% CI, 1.19–4.74; P = .02) than the internal focus group. Conclusions External focus of attention compared with internal focus of attention improves medical motor skill learning in novice learners.

Abstract Purpose Information on factors influencing learner choice between in-person and video-based continuing professional development (CPD) would help course leaders with planning and implementation. This study aimed to investigate how registration patterns differ between in-person and video-based offerings for the same CPD course. Method The authors obtained data from 55 CPD courses offered in-person (at various U.S. locations) and via livestreamed video, January 2020 to April 2022. Participants included physicians, advanced practice providers, allied health professionals, nurses, and pharmacists. Registration rates were compared by participant and course features including professional role, age, country of residence, distance to and perceived desirability of the in-person destination, and timing of registration. Results Analyses included 11,072 registrations, of which 4,336 (39.2%) were for video-based learning. There was significant heterogeneity in video-based registrations across courses, ranging 14.3% to 71.4%. Multivariable analysis showed higher video-based registrations for advanced practice providers (vs physicians; adjusted odds ratio [AOR] 1.80 [99% confidence interval, 1.55–2.10]), non-U.S. residents (AOR 3.26 [1.18–9.01]), longer distance (AOR 1.19 [1.16–1.23] for each doubling of distance), and courses in July–September 2021 (vs January–April 2022; AOR 1.59 [1.24–2.02]); video-based registrations were lower for current or former employees or trainees of our institution (AOR 0.53 [0.45–0.61]), destinations of moderate or high desirability (vs low; AOR 0.42 [0.34–0.51] and 0.44 [0.33–0.58], respectively), and early registration (AOR 0.67 [0.64–0.69] for each doubling of days between registration and course start). There was no significant difference by age (AOR 0.92 [0.82–1.05] for participants &gt; 46 years old vs younger participants). The multivariable model correctly predicted actual registration 78.5% of the time. Conclusions Video-based livestreamed CPD is popular (chosen by nearly 40% of participants), although choices varied widely by course. Professional role, institutional affiliation, distance traveled, location desirability, and registration timing have small but statistically significant associations with choice of video-based vs in-person CPD.

Abstract Purpose Both overuse and underuse of clinician referrals can compromise high-value health care. The authors sought to systematically identify and synthesize published research examining associations between physician continuous professional development (CPD) and referral patterns. Method The authors searched MEDLINE, Embase, PsycInfo, and the Cochrane Database on April 23, 2020, for comparative studies evaluating CPD for practicing physicians and reporting physician referral outcomes. Two reviewers, working independently, screened all articles for inclusion. Two reviewers reviewed all included articles to extract information, including data on participants, educational interventions, study design, and outcomes (referral rate, intended direction of change, appropriateness of referral). Quantitative results were pooled using meta-analysis. Results Of 3,338 articles screened, 31 were included. These studies enrolled at least 14,458 physicians and reported 381,165 referral events. Among studies comparing CPD with no intervention, 17 studies with intent to increase referrals had a pooled risk ratio of 1.91 (95% confidence interval: 1.50, 2.44; P &lt; .001), and 7 studies with intent to decrease referrals had a pooled risk ratio of 0.68 (95% confidence interval: 0.55, 0.83; P &lt; .001). Five studies did not indicate the intended direction of change. Subgroup analyses revealed similarly favorable effects for specific instructional approaches (including lectures, small groups, Internet-based instruction, and audit/feedback) and for activities of varying duration. Four studies reported head-to-head comparisons of alternate CPD approaches, revealing no clear superiority for any approach. Seven studies adjudicated the appropriateness of referral, and 9 studies counted referrals that were actually completed (versus merely requested). Conclusions Although between-study differences are large, CPD is associated with statistically significant changes in patient referral rates in the intended direction of impact. There are few head-to-head comparisons of alternate CPD interventions using referrals as outcomes.

Abstract Background: Mammographic breast density declines during menopause. We assessed changes in volumetric breast density across the menopausal transition and factors that influence these changes. Methods: Women without a history of breast cancer, who had full field digital mammograms during both pre- and postmenopausal periods, at least 2 years apart, were sampled from four facilities within the San Francisco Mammography Registry from 2007 to 2013. Dense breast volume (DV) was assessed using Volpara on mammograms across the time period. Annualized change in DV from pre- to postmenopause was estimated using linear mixed models adjusted for covariates and per-woman random effects. Multiplicative interactions were evaluated between premenopausal risk factors and time to determine whether these covariates modified the annualized changes. Results: Among the 2,586 eligible women, 1,802 had one premenopausal and one postmenopausal mammogram, 628 had an additional perimenopausal mammogram, and 156 had two perimenopausal mammograms. Women experienced an annualized decrease in DV [−2.2 cm3 (95% confidence interval, −2.7 to −1.7)] over the menopausal transition. Declines were greater among women with a premenopausal DV above the median (54 cm3) versus below (DV, −3.5 cm3 vs. −1.0 cm3; P &lt; 0.0001). Other breast cancer risk factors, including race, body mass index, family history, alcohol, and postmenopausal hormone therapy, had no effect on change in DV over the menopausal transition. Conclusions: High premenopausal DV was a strong predictor of greater reductions in DV across the menopausal transition. Impact: We found that few factors other than premenopausal density influence changes in DV across the menopausal transition, limiting targeted prevention efforts.

We present a semi‐automatic method that uses principles from factor analysis to subdivide large pedigrees into smaller, non‐overlapping sub‐pedigrees. Application of our method to the Genetic Analysis Workshop 12 Hutterite pedigree yielded 12 sub‐pedigrees that were used to carry out a genome‐wide linkage scan for IgE serum level in SOLAR. Two‐point analyses resulted in strong evidence for linkage to two marker loci on chromosome 1, D1S3723 (lod = 3.70) and D1S534 (lod = 3.58). Multipoint analysis resulted in a maximum lod of 4.58 on chromosome 1 at 143 cM. After extending the two pedigrees that contributed the most evidence for linkage, the maximum lod score decreased to 4.18, with most of the evidence for linkage due to a single large sub‐pedigree. © 2001 Wiley‐Liss, Inc.

Purpose To characterize reporting of P values, confidence intervals (CIs), and statistical power in health professions education research (HPER) through manual and computerized analysis of published research reports. Method The authors searched PubMed, Embase, and CINAHL in May 2016, for comparative research studies. For manual analysis of abstracts and main texts, they randomly sampled 250 HPER reports published in 1985, 1995, 2005, and 2015, and 100 biomedical research reports published in 1985 and 2015. Automated computerized analysis of abstracts included all HPER reports published 1970–2015. Results In the 2015 HPER sample, P values were reported in 69/100 abstracts and 94 main texts. CIs were reported in 6 abstracts and 22 main texts. Most P values (≥77%) were ≤.05. Across all years, 60/164 two-group HPER studies had ≥80% power to detect a between-group difference of 0.5 standard deviations. From 1985 to 2015, the proportion of HPER abstracts reporting a CI did not change significantly (odds ratio [OR] 2.87; 95% CI 1.04, 7.88) whereas that of main texts reporting a CI increased (OR 1.96; 95% CI 1.39, 2.78). Comparison with biomedical studies revealed similar reporting of P values, but more frequent use of CIs in biomedicine. Automated analysis of 56,440 HPER abstracts found 14,867 (26.3%) reporting a P value, 3,024 (5.4%) reporting a CI, and increased reporting of P values and CIs from 1970 to 2015. Conclusions P values are ubiquitous in HPER, CIs are rarely reported, and most studies are underpowered. Most reported P values would be considered statistically significant.