Cook, David A.

Abstract Purpose Learner engagement is the energy learners exert to remain focused and motivated to learn. The Learner Engagement Instrument (LEI) was developed to measure learner engagement in a short continuing professional development (CPD) activity. The authors validated LEI scores using validity evidence of internal structure and relationships with other variables. Method Participants attended 1 of 4 CPD courses (1 in-person, 2 online livestreamed, and 1 either in-person or livestreamed) in 2018, 2020, 2021, and 2022. Confirmatory factor analysis was used to examine model fit for several alternative structural models, separately for each course. The authors also conducted a generalizability study to estimate score reliability. Associations were evaluated between LEI scores and Continuing Medical Education Teaching Effectiveness (CMETE) scores and participant demographics. Statistical methods accounted for repeated measures by participants. Results Four hundred fifteen unique participants attended 203 different CPD presentations and completed the LEI 11,567 times. The originally hypothesized 4-domain model of learner engagement (domains: emotional, behavioral, cognitive in-class, cognitive out-of-class) demonstrated best model fit in all 4 courses, with comparative fit index ≥ 0.99, standardized root mean square residual ≤ 0.031, and root mean square error of approximation ≤ 0.047. The reliability for overall scores and domain scores were all acceptable (50-rater G-coefficient ≥ 0.74) except for the cognitive in-class domain (50-rater G-coefficient of 0.55 to 0.66). Findings were similar for both in-person and online delivery modalities. Correlation of LEI scores with teaching effectiveness was confirmed (rho=0.58), and a small correlation was found with participant age (rho=0.19); other associations were small and not statistically significant. Using these findings, we generated a shortened 4-item instrument, the LEI Short Form. Conclusions This study confirms a 4-domain model of learner engagement and provides validity evidence that supports using LEI scores to measure learner engagement in both in-person and livestreamed CPD activities.

Abstract Purpose Mayer's theory of multimedia learning proposes that personalisation and embodiment (P/E) can improve outcomes in e‐Learning. The authors hypothesised that an e‐Learning module enhanced by P/E principles would lead to higher knowledge, perceived P/E and motivation among health care professionals, compared with an unenhanced module. Methods The authors conducted a randomised trial comparing two versions of a 30‐minute multimedia e‐Learning module addressing the antibiotic management of pneumonia. The unenhanced format used slides with voiceover (human voice but no visible speaker), formal language and no specific P/E strategies. The enhanced format additionally implemented P/E strategies including conversational style, polite language, visible author, social congruence, human‐like presence and professional presence by subtly changing the script and substituting several short videos of subject matter experts. Participants included pharmacists, physicians and advanced practice providers from three academic and several community hospitals. Outcomes included knowledge, perceived P/E (assessed by the Congruence Personalisation Questionnaire, CPQ), motivation (assessed via the Instructional Materials Motivation Survey [IMMS] and Motivated Strategies for Learning Questionnaire [MSLQ]) and course satisfaction. Results There were 406 participants including 225 pharmacists, 109 physicians and 72 advanced practice providers. Post‐module knowledge was slightly higher for the enhanced versus the unenhanced format, but the difference did not reach statistical significance (adjusted mean difference, 0.04 of 10 possible, [95% CI −0.26, 0.34], p = 0.78; Cohen d 0.02). Participant perceptions of P/E (measured via CPQ) were significantly greater for the enhanced format (difference 0.46 of 5 possible [0.35, 0.56], p < 0.001; Cohen d 0.85), as were motivational features of the e‐Learning course (measured via IMMS) (difference 0.14 of 5 possible [0.02, 0.26], p = 0.02; Cohen d 0.24). Participants' overall motivational orientation (measured via MSLQ) and course satisfaction were not significantly different between the two formats ( p > 0.05). Conclusion Application of P/E principles to an e‐Learning module led to greater perceived P/E and motivational features but did not influence knowledge.

Abstract Background Over the past decade, the use of technology‐enhanced simulation in emergency medicine (EM) education has grown, yet we still lack a clear understanding of its effectiveness. This systematic review aims to identify and synthesize studies evaluating the comparative effectiveness of technology‐enhanced simulation in EM. Methods We searched MEDLINE, EMBASE, PsycINFO, CINAHL, ERIC, Web of Science, and Scopus to identify EM simulation research that compares technology‐enhanced simulation with other instructional modalities. Two reviewers screened articles for inclusion and abstracted information on learners, clinical topics, instructional design features, outcomes, cost, and study quality. Standardized mean difference (SMD) effect sizes were pooled using random effects. Results We identified 60 studies, enrolling at least 5279 learners. Of these, 23 compared technology‐enhanced simulation with another instructional modality (e.g., living humans, lecture, small group), and 37 compared two forms of technology‐enhanced simulation. Compared to lecture or small groups, we found simulation to have nonsignificant differences for time skills (SMD 0.33, 95% confidence interval [CI] −0.23 to 0.89, n = 3), but a large, significant effect for non–time skills (SMD 0.82, 95% CI 0.18 to 1.46, n = 8). Comparison of alternative types of technology‐enhanced simulation found favorable associations with skills acquisition, of moderate magnitude, for computer‐assisted guidance (compared to no computer‐assisted guidance), for time skills (SMD 0.50, 95% CI −1.66 to 2.65, n = 2) and non–time skills (SMD 0.57, 95% CI 0.33 to 0.80, n = 6), and for more task repetitions (time skills SMD 1.01, 95% CI 0.16 to 1.86, n = 2) and active participation (compared to observation) for time skills (SMD 0.85, 95% CI 0.25 to 1.45, n = 2) and non–time skills (SMD 0.33 95% CI 0.08 to 0.58, n = 3). Conclusions Technology‐enhanced simulation is effective for EM learners for skills acquisition. Features such as computer‐assisted guidance, repetition, and active learning are associated with greater effectiveness.

Abstract Purpose Information on factors influencing learner choice between in-person and video-based continuing professional development (CPD) would help course leaders with planning and implementation. This study aimed to investigate how registration patterns differ between in-person and video-based offerings for the same CPD course. Method The authors obtained data from 55 CPD courses offered in-person (at various U.S. locations) and via livestreamed video, January 2020 to April 2022. Participants included physicians, advanced practice providers, allied health professionals, nurses, and pharmacists. Registration rates were compared by participant and course features including professional role, age, country of residence, distance to and perceived desirability of the in-person destination, and timing of registration. Results Analyses included 11,072 registrations, of which 4,336 (39.2%) were for video-based learning. There was significant heterogeneity in video-based registrations across courses, ranging 14.3% to 71.4%. Multivariable analysis showed higher video-based registrations for advanced practice providers (vs physicians; adjusted odds ratio [AOR] 1.80 [99% confidence interval, 1.55–2.10]), non-U.S. residents (AOR 3.26 [1.18–9.01]), longer distance (AOR 1.19 [1.16–1.23] for each doubling of distance), and courses in July–September 2021 (vs January–April 2022; AOR 1.59 [1.24–2.02]); video-based registrations were lower for current or former employees or trainees of our institution (AOR 0.53 [0.45–0.61]), destinations of moderate or high desirability (vs low; AOR 0.42 [0.34–0.51] and 0.44 [0.33–0.58], respectively), and early registration (AOR 0.67 [0.64–0.69] for each doubling of days between registration and course start). There was no significant difference by age (AOR 0.92 [0.82–1.05] for participants &gt; 46 years old vs younger participants). The multivariable model correctly predicted actual registration 78.5% of the time. Conclusions Video-based livestreamed CPD is popular (chosen by nearly 40% of participants), although choices varied widely by course. Professional role, institutional affiliation, distance traveled, location desirability, and registration timing have small but statistically significant associations with choice of video-based vs in-person CPD.

Abstract Purpose Assessing expertise using psychometric models usually yields a measure of ability that is difficult to generalize to the complexity of diagnoses in clinical practice. However, using an item response modeling framework, it is possible to create a decision-aligned response model that captures a clinician’s decision-making behavior on a continuous scale that fully represents competing diagnostic possibilities. In this proof-of-concept study, the authors demonstrate the necessary statistical conceptualization of this model using a specific electrocardiogram (ECG) example. Method The authors collected a range of ECGs with elevated ST segments due to either ST-elevation myocardial infarction (STEMI) or pericarditis. Based on pilot data, 20 ECGs were chosen to represent a continuum from “definitely STEMI” to “definitely pericarditis,” including intermediate cases in which the diagnosis was intentionally unclear. Emergency medicine and cardiology physicians rated these ECGs on a 5-point scale (“definitely STEMI” to “definitely pericarditis”). The authors analyzed these ratings using a graded response model showing the degree to which each participant could separate the ECGs along the diagnostic continuum. The authors compared these metrics with the discharge diagnoses noted on chart review. Results Thirty-seven participants rated the ECGs. As desired, the ECGs represented a range of phenotypes, including cases where participants were uncertain in their diagnosis. The response model showed that participants varied both in their propensity to diagnose one condition over another and in where they placed the thresholds between the 5 diagnostic categories. The most capable participants were able to meaningfully use all categories, with precise thresholds between categories. Conclusions The authors present a decision-aligned response model that demonstrates the confusability of a particular ECG and the skill with which a clinician can distinguish 2 diagnoses along a continuum of confusability. These results have broad implications for testing and for learning to manage uncertainty in diagnosis.

Abstract Purpose Instructional designs promoting personalization (conversational style and visible author), embodiment (human-like mannerisms and presence), and social and cognitive congruence have been shown to enhance learning. However, measurements of these instructional design features are lacking. The authors sought to develop and validate an instrument to measure personalization, embodiment, and congruence in an online course, collecting evidence of content, internal structure, and relationships with other variables. Method Through literature review, expert review, and pilot testing, the authors developed a 15-item, 5-domain instrument. Participants completed this instrument along with assessments of motivation (Instructional Materials Motivation Survey and Motivated Strategies for Learning Questionnaire), knowledge, and course satisfaction immediately after a 30-minute online multimedia learning module comprising slides with voiceover and (for some participants) short video clips, from January to April 2021. Participants included pharmacists, physicians, and advanced practice providers at academic and community hospitals. Confirmatory factor analysis (CFA) was used to examine model fit, followed by structural revisions guided by exploratory factor analysis (EFA) and tested in a new CFA using a split-sample approach. The authors evaluated associations between the revised instrument scores and motivation, knowledge, and satisfaction. Results There were 406 participants. CFA of the original 15-item instrument revealed poor model fit. EFA using 214 randomly selected participants suggested a 2-domain structure. CFA of the revised structure using the other 192 participants revealed good fit for a 9-item, 2-domain instrument: the Congruence–Personalization Questionnaire (CPQ). Linear regression confirmed positive, statistically significant associations for most hypothesized relationships (R2 ≥ 0.12 and P &lt; .001 for 14 of 18 associations with motivation and satisfaction). Associations with knowledge were weak and nonsignificant (R2 ≤ 0.004, P &gt; .23). Conclusions Evidence of content, internal structure, and relationships with other variables support the validity of CPQ scores for measuring personalization and congruence in an online module.

Abstract Introduction Engaging learners in continuing medical education (CME) is challenging. Recently, CME courses have transitioned to livestreamed CME, with learners viewing live, in‐person courses online. The authors aimed to (1) compare learner engagement and teaching effectiveness in livestreamed with in‐person CME and (2) determine how livestream engagement and teaching effectiveness is associated with (A) interactivity metrics, (B) presentation characteristics and (C) medical knowledge. Methods A 3‐year, non‐randomised study of in‐person and livestream CME was performed. The course was in‐person for 2018 but transitioned to livestream for 2020 and 2021. Learners completed the Learner Engagement Inventory and Teaching Effectiveness Instrument after each presentation. Both instruments were supported by content, internal structure and relations to other variables' validity evidence. Interactivity metrics included learner use of audience response, questions asked by learners and presentation views. Presentation characteristics included presentations using audience response, using pre/post‐test format, time of day and words per slide. Medical knowledge was assessed by audience response. A repeated measures analysis of variance ( anova ) was used for comparisons and a mixed model approach for correlations. Results A total of 159 learners (response rate 27%) completed questionnaires. Engagement did not significantly differ between in‐person or livestream CME. (4.56 versus 4.53, p = 0.64, maximum 5 = highly engaged). However, teacher effectiveness scores were higher for in‐person compared with livestream (4.77 versus 4.71 p = 0.01, maximum 5 = highly effective). For livestreamed courses, learner engagement was associated with presentation characteristics, including presentation using of audience response (yes = 4.57, no = 4.45, p < .0001), use of a pre/post‐test (yes = 4.62, no = 4.54, p < .0001) and time of presentation (morning = 4.58, afternoon = 4.53, p = .0002). Significant associations were not seen for interactivity metrics or medical knowledge. Discussion Livestreaming may be as engaging as in‐person CME. Although teaching effectiveness in livestreaming was lower, this difference was small. CME course planners should consider offering livestream CME while exploring strategies to enhance teaching effectiveness in livestreamed settings.