Cook, D A

ABSTRACT Objectives The increasing volume of medical education research necessitates efficient, reliable, and scalable methods for conducting quality appraisals. The Medical Education Research Study Quality Instrument (MERSQI) is a widely used tool, although its manual scoring process remains resource‐intensive. This study evaluated how well large language models (LLMs) appraise medical education research using the MERSQI tool in comparison with human judges. Methods Three LLMs (GPT‐5, Claude Sonnet 4, and Gemini 2.5 Pro) assigned MERSQI domain scores to 1423 medical education research articles. The authors compared AI‐generated scores with human‐generated scores using intraclass correlation coefficients (ICCs) across the six MERSQI domains. They evaluated the agreement between AI‐ and human‐generated MERSQI composite scores using Bland–Altman plots. Results Domain‐level ICC values ranged from fair (0.24) to near perfect (0.81), with the lowest agreement observed in the ‘sampling,’ ‘validity evidence,’ and ‘data analysis’ domains. No single LLM consistently outperformed the others across all domains. Composite score agreement with human ratings was substantial and similar across LLMs (ICC range: 0.65–0.69). GPT‐5 produced slightly lower composite scores than humans, while Claude Sonnet 4 and Gemini 2.5 Pro produced higher scores, with Gemini showing the largest deviation. The Bland–Altman plots for Gemini 2.5 Pro suggested proportional bias, indicating its agreement with human scores varied across the range of study quality. Conclusions These LLMs demonstrated substantial agreement with human raters for MERSQI composite scores, but domain‐level agreement varied. Systematic differences in scoring patterns highlight the need for human oversight and additional calibrations before integrating LLMs into systematic review appraisal workflows.

Abstract Purpose Mayer's theory of multimedia learning proposes that personalisation and embodiment (P/E) can improve outcomes in e‐Learning. The authors hypothesised that an e‐Learning module enhanced by P/E principles would lead to higher knowledge, perceived P/E and motivation among health care professionals, compared with an unenhanced module. Methods The authors conducted a randomised trial comparing two versions of a 30‐minute multimedia e‐Learning module addressing the antibiotic management of pneumonia. The unenhanced format used slides with voiceover (human voice but no visible speaker), formal language and no specific P/E strategies. The enhanced format additionally implemented P/E strategies including conversational style, polite language, visible author, social congruence, human‐like presence and professional presence by subtly changing the script and substituting several short videos of subject matter experts. Participants included pharmacists, physicians and advanced practice providers from three academic and several community hospitals. Outcomes included knowledge, perceived P/E (assessed by the Congruence Personalisation Questionnaire, CPQ), motivation (assessed via the Instructional Materials Motivation Survey [IMMS] and Motivated Strategies for Learning Questionnaire [MSLQ]) and course satisfaction. Results There were 406 participants including 225 pharmacists, 109 physicians and 72 advanced practice providers. Post‐module knowledge was slightly higher for the enhanced versus the unenhanced format, but the difference did not reach statistical significance (adjusted mean difference, 0.04 of 10 possible, [95% CI −0.26, 0.34], p = 0.78; Cohen d 0.02). Participant perceptions of P/E (measured via CPQ) were significantly greater for the enhanced format (difference 0.46 of 5 possible [0.35, 0.56], p < 0.001; Cohen d 0.85), as were motivational features of the e‐Learning course (measured via IMMS) (difference 0.14 of 5 possible [0.02, 0.26], p = 0.02; Cohen d 0.24). Participants' overall motivational orientation (measured via MSLQ) and course satisfaction were not significantly different between the two formats ( p > 0.05). Conclusion Application of P/E principles to an e‐Learning module led to greater perceived P/E and motivational features but did not influence knowledge.

Abstract Purpose Using electrocardiogram (ECG) interpretation as an example of a widely taught diagnostic skill, the authors conducted a systematic review and meta-analysis to demonstrate how research evidence on instruction in diagnosis can be synthesized to facilitate improvement of educational activities (instructional modalities, instructional methods, and interpretation approaches), guide the content and specificity of such activities, and provide direction for research. Method The authors searched PubMed/MEDLINE, Embase, Cochrane CENTRAL, PsycInfo, CINAHL, ERIC, and Web of Science databases through February 21, 2020, for empirical investigations of ECG interpretation training enrolling medical students, residents, or practicing physicians. They appraised study quality with the Medical Education Research Study Quality Instrument and pooled standardized mean differences (SMDs) using random effects meta-analysis. Results Of 1,002 articles identified, 59 were included (enrolling 17,251 participants). Among 10 studies comparing instructional modalities, 8 compared computer-assisted and face-to-face instruction, with pooled SMD 0.23 (95% CI, 0.09, 0.36) indicating a small, statistically significant difference favoring computer-assisted instruction. Among 19 studies comparing instructional methods, 5 evaluated individual versus group training (pooled SMD −0.35 favoring group study [95% CI, −0.06, −0.63]), 4 evaluated peer-led versus faculty-led instruction (pooled SMD 0.38 favoring peer instruction [95% CI, 0.01, 0.74]), and 4 evaluated contrasting ECG features (e.g., QRS width) from 2 or more diagnostic categories versus routine examination of features within a single ECG or diagnosis (pooled SMD 0.23 not significantly favoring contrasting features [95% CI, −0.30, 0.76]). Eight studies compared ECG interpretation approaches, with pooled SMD 0.92 (95% CI, 0.48, 1.37) indicating a large, statistically significant effect favoring more systematic interpretation approaches. Conclusions Some instructional interventions appear to improve learning in ECG interpretation; however, many evidence-based instructional strategies are insufficiently investigated. The findings may have implications for future research and design of training to improve skills in ECG interpretation and other types of visual diagnosis.

Abstract Purpose Both overuse and underuse of clinician referrals can compromise high-value health care. The authors sought to systematically identify and synthesize published research examining associations between physician continuous professional development (CPD) and referral patterns. Method The authors searched MEDLINE, Embase, PsycInfo, and the Cochrane Database on April 23, 2020, for comparative studies evaluating CPD for practicing physicians and reporting physician referral outcomes. Two reviewers, working independently, screened all articles for inclusion. Two reviewers reviewed all included articles to extract information, including data on participants, educational interventions, study design, and outcomes (referral rate, intended direction of change, appropriateness of referral). Quantitative results were pooled using meta-analysis. Results Of 3,338 articles screened, 31 were included. These studies enrolled at least 14,458 physicians and reported 381,165 referral events. Among studies comparing CPD with no intervention, 17 studies with intent to increase referrals had a pooled risk ratio of 1.91 (95% confidence interval: 1.50, 2.44; P &lt; .001), and 7 studies with intent to decrease referrals had a pooled risk ratio of 0.68 (95% confidence interval: 0.55, 0.83; P &lt; .001). Five studies did not indicate the intended direction of change. Subgroup analyses revealed similarly favorable effects for specific instructional approaches (including lectures, small groups, Internet-based instruction, and audit/feedback) and for activities of varying duration. Four studies reported head-to-head comparisons of alternate CPD approaches, revealing no clear superiority for any approach. Seven studies adjudicated the appropriateness of referral, and 9 studies counted referrals that were actually completed (versus merely requested). Conclusions Although between-study differences are large, CPD is associated with statistically significant changes in patient referral rates in the intended direction of impact. There are few head-to-head comparisons of alternate CPD interventions using referrals as outcomes.

Purpose Nearly all health care professionals engage in continuous professional development (CPD), yet little is known about the cost and cost-effectiveness of physician CPD. Clarification of key concepts, comprehensive identification of published work, and determination of research gaps would facilitate application of existing evidence and planning for future investigations. The authors sought to systematically map study themes, methods, and outcomes in peer-reviewed literature on the cost and value of physician CPD. Method The authors conducted a scoping review, systematically searching MEDLINE, Embase, PsycInfo, and Cochrane Library databases for comparative economic evaluations of CPD for practicing physicians through April 2020. Two reviewers, working independently, screened all articles for inclusion. Three reviewers iteratively reviewed all included articles to inductively identify key features including participants, educational interventions, study designs, cost ingredients, and cost analyses. Two reviewers then independently reexamined all included articles to code these features. Results Of 3,338 potentially eligible studies, 111 were included. Physician specialties included internal, family, or general medicine (80 studies [72%]), surgery (14 studies [13%]), and medicine subspecialties (7 studies [6%]). Topics most often addressed general medicine (45 studies [41%]) or appropriate drug use (37 studies [33%]). Eighty-seven studies (78%) compared CPD with no intervention. Sixty-three studies (57%) reported the cost of training, and 79 (71%) evaluated the economic impact (money saved/lost following CPD). Training cost ingredients (median 3 itemized per study) and economic impact ingredients (median 1 per study) were infrequently and incompletely identified, quantified, or priced. Twenty-seven studies (24%) reported cost–impact expressions such as cost-effectiveness ratio or net value. Nineteen studies (17%) reported sensitivity analyses. Conclusions Studies evaluating the costs and economic impact of physician CPD are few. Gaps exist in identification, quantification, pricing, and analysis of cost outcomes. The authors propose a comprehensive framework for appraising ingredients and a preliminary reference case for economic evaluations.

Purpose To identify features of instruments, test procedures, study design, and validity evidence in published studies of electrocardiogram (ECG) skill assessments. Method The authors conducted a systematic review, searching MEDLINE, Embase, Cochrane CENTRAL, PsycINFO, CINAHL, ERIC, and Web of Science databases in February 2020 for studies that assessed the ECG interpretation skill of physicians or medical students. Two authors independently screened articles for inclusion and extracted information on test features, study design, risk of bias, and validity evidence. Results The authors found 85 eligible studies. Participants included medical students (42 studies), postgraduate physicians (48 studies), and practicing physicians (13 studies). ECG selection criteria were infrequently reported: 25 studies (29%) selected single-diagnosis or straightforward ECGs; 5 (6%) selected complex cases. ECGs were selected by generalists (15 studies [18%]), cardiologists (10 studies [12%]), or unspecified experts (4 studies [5%]). The median number of ECGs per test was 10. The scoring rubric was defined by 2 or more experts in 32 studies (38%), by 1 expert in 5 (6%), and using clinical data in 5 (6%). Scoring was performed by a human rater in 34 studies (40%) and by computer in 7 (8%). Study methods were appraised as low risk of selection bias in 16 studies (19%), participant flow bias in 59 (69%), instrument conduct and scoring bias in 20 (24%), and applicability problems in 56 (66%). Evidence of test score validity was reported infrequently, namely evidence of content (39 studies [46%]), internal structure (11 [13%]), relations with other variables (10 [12%]), response process (2 [2%]), and consequences (3 [4%]). Conclusions ECG interpretation skill assessments consist of idiosyncratic instruments that are too short, composed of items of obscure provenance, with incompletely specified answers, graded by individuals with underreported credentials, yielding scores with limited interpretability. The authors suggest several best practices.