Cook, David A.

Purpose Compare the effect of personalized feedback (PF) vs. task demonstration (TD), both delivered via video, on laparoscopic knot-tying skills and perceived workload; and evaluate the effect of repeated practice. Method General surgery interns and research fellows completed four repetitions of a simulated laparoscopic knot-tying task at one-month intervals. Midway between repetitions, participants received via e-mail either a TD video (demonstration by an expert) or a PF video (video of their own performance with voiceover from a blinded senior surgeon). Each participant received at least one video per format, with sequence randomly assigned. Outcomes included performance scores and NASA Task Load Index (NASA-TLX) scores. To evaluate the effectiveness of repeated practice, scores from these trainees on a separate delayed retention test were compared against historical controls who did not have scheduled repetitions. Results Twenty-one trainees completed the randomized study. Mean change in performance scores was significantly greater for those receiving PF (difference = 23.1 of 150 [95% confidence interval (CI): 0, 46.2], P = .05). Perceived workload was also significantly reduced (difference = −3.0 of 20 [95% CI: −5.8, −0.3], P = .04). Compared with historical controls (N = 93), the 21 with scheduled repeated practice had higher scores on the laparoscopic knot-tying assessment two weeks after the final repetition (difference = 1.5 of 10 [95% CI: 0.2, 2.8], P = .02). Conclusions Personalized video feedback improves trainees’ procedural performance and perceived workload compared with a task demonstration video. Brief monthly practice sessions support skill acquisition and retention.

Objective Measurement of motivation and cognitive load has potential value in health professions education. Our objective was to evaluate the validity of scores from Dweck's Implicit Theories of Intelligence Scale ( ITIS ), Elliot's Achievement Goal Questionnaire‐Revised ( AGQ ‐R) and Leppink's cognitive load index ( CLI ). Methods This was a validity study evaluating internal structure using reliability and factor analysis, and relationships with other variables using the multitrait‐multimethod matrix. Two hundred and thirty‐two secondary school students participated in a medical simulation‐based training activity at an academic medical center. Pre‐activity ITIS (implicit theory [mindset] domains: incremental, entity) and AGQ ‐R (achievement goal domains: mastery‐approach, mastery‐avoidance, performance‐approach, performance‐avoidance), post‐activity CLI (cognitive load domains: intrinsic, extrinsic, germane) and task persistence (self‐directed repetitions on a laparoscopic surgery task) were measured. Results Internal consistency reliability (Cronbach's alpha) was > 0.70 for all domain scores except AGQ ‐R performance‐avoidance (alpha 0.68) and CLI extrinsic load (alpha 0.64). Confirmatory factor analysis of ITIS and CLI scores demonstrated acceptable model fit. Confirmatory factor analysis of AGQ ‐R scores demonstrated borderline fit, and exploratory factor analysis suggested a three‐domain model for achievement goals (mastery‐approach, performance and avoidance). Correlations among scores from conceptually‐related domains generally aligned with expectations, as follows: ITIS incremental and entity, r = −0.52; AGQ ‐R mastery‐avoidance and performance‐avoidance, r = 0.71; mastery‐approach and performance‐approach, r = 0.55; performance‐approach and performance‐avoidance, r = 0.43; mastery‐approach and mastery‐avoidance, r = 0.36; CLI germane and extrinsic, r = −0.35; ITIS incremental and AGQ ‐R mastery‐approach, r = 0.34; ITIS incremental and CLI germane, r = 0.44; AGQ ‐R mastery‐approach and CLI germane, r = 0.48 (all p < 0.001). We found no correlation between the number of task repetitions (i.e. persistence) and mastery‐approach scores, r = −0.01. Conclusions ITIS and CLI scores had appropriate internal structures and relationships with other variables. AGQ ‐R scores fit a three‐factor (not four‐factor) model that collapsed avoidance into one domain, although relationships of other variables with the original four domain scores generally aligned with expectations. Mastery goals are positively correlated with germane cognitive load.

Purpose To determine the past experiences with, current use of, and anticipated use of online learning and simulation-based education among practicing U.S. physicians, and how findings vary by age. Method The authors surveyed 4,648 randomly sampled board-certified U.S. physicians, September 2015 to April 2016, using Internet-based and paper questionnaires. Survey items (some optional) addressed past and current technology usage, perceived technology effectiveness, and anticipated future use of specific technology innovations. Results Of 988 respondents, 444 completed optional items. Of these, 429/442 (97.1%) had used online learning and 372/442 (84.2%) had used simulation-based education in the past five years. Desire for more online learning was modest (mean [standard deviation], 4.6 [1.5]; 1 = strongly disagree, 7 = strongly agree), as was desire for more simulation-based education (4.2 [1.7]). Both online learning and simulation-based education were perceived as effective (5.2 [1.4]; 5.0 [1.4]). Physicians believed they possess adequate skills for online learning (5.8 [1.2]) and that point-of-care learning is vital to effective patient care (5.3 [1.3]). Only 39.0% used objective performance data to guide their learning choices, although 64.6% agreed that such information would be useful. The highest-rated innovations included a central repository for listing educational opportunities and tracking continuing education credits, an app to award credit for answering patient-focused questions, 5-minute and 20-minute clinical updates, and an e-mailed “question of the week.” Responses to most survey items were similar across age groups. Conclusions Practicing physicians generally seem receptive and prepared to use a variety of educational technologies, regardless of age.

Purpose To characterize reporting of P values, confidence intervals (CIs), and statistical power in health professions education research (HPER) through manual and computerized analysis of published research reports. Method The authors searched PubMed, Embase, and CINAHL in May 2016, for comparative research studies. For manual analysis of abstracts and main texts, they randomly sampled 250 HPER reports published in 1985, 1995, 2005, and 2015, and 100 biomedical research reports published in 1985 and 2015. Automated computerized analysis of abstracts included all HPER reports published 1970–2015. Results In the 2015 HPER sample, P values were reported in 69/100 abstracts and 94 main texts. CIs were reported in 6 abstracts and 22 main texts. Most P values (≥77%) were ≤.05. Across all years, 60/164 two-group HPER studies had ≥80% power to detect a between-group difference of 0.5 standard deviations. From 1985 to 2015, the proportion of HPER abstracts reporting a CI did not change significantly (odds ratio [OR] 2.87; 95% CI 1.04, 7.88) whereas that of main texts reporting a CI increased (OR 1.96; 95% CI 1.39, 2.78). Comparison with biomedical studies revealed similar reporting of P values, but more frequent use of CIs in biomedicine. Automated analysis of 56,440 HPER abstracts found 14,867 (26.3%) reporting a P value, 3,024 (5.4%) reporting a CI, and increased reporting of P values and CIs from 1970 to 2015. Conclusions P values are ubiquitous in HPER, CIs are rarely reported, and most studies are underpowered. Most reported P values would be considered statistically significant.

Purpose Professional development (PD)—both for-credit continuing medical education (CME) and informal self-directed or point-of-care learning—is vital to all physicians. The authors sought to understand physicians’ PD perceptions and practices and how these vary by specialty and practice type. Method The authors administered an Internet and paper survey, from September 2015 to April 2016, to randomly sampled U.S. physicians. Survey items addressed perceived PD needs and barriers and how physicians identify knowledge/skills gaps. Results Of 4,648 invitees, 988 (21.6%) responded. Respondents believed that they already know what they need to learn (mean 5.8 [1 = strongly disagree; 7 = strongly agree]), can answer clinical questions using available resources (5.9), and want credit for learning during patient care (5.1). They did not strongly desire help identifying learning gaps (4.0) or indicate difficulty accumulating CME credits (3.1). Most PD was done during personal time (5.5). Competencies regarding medical knowledge/skills, wellness, informatics, and practice/systems improvement were rated the highest priority, while research, teaching, and professionalism were rated the lowest. The most important sources used to identify knowledge/skills gaps were immediate patient care needs (4.1 [1 = not important; 5 = extremely important]), personal awareness (3.8), and practice updates (3.7). The most important barriers were time (3.5) and cost (2.9). Differences by specialty and practice type were generally small and not statistically significant. Conclusions Physicians feel confident in identifying their own learning needs, perceive medical knowledge/skills as their highest-priority need, and desire more credit for learning during patient care.

AbstractObjective: To better understand clinician information needs and learning opportunities by exploring the use of best-practice algorithms across different training levels and specialties.Methods: We developed interactive online algorithms (care process models [CPMs]) that integrate current guidelines, recent evidence, and local expertise to represent cross-disciplinary best practices for managing clinical problems. We reviewed CPM usage logs from January 2014 to June 2015 and compared usage across specialty and provider type.Results: During the study period, 4009 clinicians (2014 physicians in practice, 1117 resident physicians, and 878 nurse practitioners/physician assistants [NP/PAs]) viewed 140 CPMs a total of 81 764 times. Usage varied from 1 to 809 views per person, and from 9 to 4615 views per CPM. Residents and NP/PAs viewed CPMs more often than practicing physicians. Among 2742 users with known specialties, generalists (N = 1397) used CPMs more often (mean 31.8, median 7 views) than specialists (N = 1345; mean 6.8, median 2; P &lt; .0001). The topics used by specialists largely aligned with topics within their specialties. The top 20% of available CPMs (28/140) collectively accounted for 61% of uses. In all, 2106 clinicians (52%) returned to the same CPM more than once (average 7.8 views per topic; median 4, maximum 195). Generalists revisited topics more often than specialists (mean 8.8 vs 5.1 views per topic; P &lt; .0001).Conclusions: CPM usage varied widely across topics, specialties, and individual clinicians. Frequently viewed and recurrently viewed topics might warrant special attention. Specialists usually view topics within their specialty and may have unique information needs.

Context Complete reporting of research is essential to enable consumers to accurately appraise, interpret and apply findings. Quality appraisal checklists are giving way to tools that judge the risk for bias. Objectives We sought to determine the prevalence of these complementary aspects of research reports (completeness of reporting and perceived risk for bias) of randomised studies in health professions education. Methods We searched bibliographic databases for randomised studies of health professions education. We appraised two cohorts representing different time periods (2008–2010 and 2014, respectively) and worked in duplicate to apply the CONSORT guidelines and Cochrane Risk of Bias tool. We explored differences between time periods using independent‐samples t ‐tests or the chi‐squared test, as appropriate. Results We systematically identified 180 randomised studies (2008–2010, n = 150; 2014, n = 30). Frequencies of reporting of CONSORT elements within full‐text reports were highly variable and most elements were reported in fewer than 50% of studies. We found a statistically significant difference in the CONSORT reporting index (maximum score: 500) between the 2008–2010 (mean ± standard deviation [SD]: 242.7 ± 55.6) and 2014 (mean ± SD: 311.6 ± 53.2) cohorts (p < 0.001). High or unclear risk for bias was most common for allocation concealment (157, 87%) and blinding of participants (147, 82%), personnel (152, 84%) and outcome assessors (112, 62%). Most risk for bias elements were judged to be unclear (range: 51–84%). Risk for bias elements significantly improved over time for blinding of participants (p = 0.007), incomplete data (p < 0.001) and the presence of other sources of bias (p < 0.001). Conclusions Reports of randomised studies in health professions education frequently omit elements recommended by the CONSORT statement. Most reports were assessed as having a high or unclear risk for bias. Greater attention to how studies are reported at study outset and in manuscript preparation could improve levels of complete transparent reporting.

Purpose In-training evaluation reports (ITERs) constitute an integral component of medical student and postgraduate physician trainee (resident) assessment. ITER narrative comments have received less attention than the numeric scores. The authors sought both to determine what validity evidence informs the use of narrative comments from ITERs for assessing medical students and residents and to identify evidence gaps. Method Reviewers searched for relevant English-language studies in MEDLINE, EMBASE, Scopus, and ERIC (last search June 5, 2015), and in reference lists and author files. They included all original studies that evaluated ITERs for qualitative assessment of medical students and residents. Working in duplicate, they selected articles for inclusion, evaluated quality, and abstracted information on validity evidence using Kane’s framework (inferences of scoring, generalization, extrapolation, and implications). Results Of 777 potential articles, 22 met inclusion criteria. The scoring inference is supported by studies showing that rich narratives are possible, that changing the prompt can stimulate more robust narratives, and that comments vary by context. Generalization is supported by studies showing that narratives reach thematic saturation and that analysts make consistent judgments. Extrapolation is supported by favorable relationships between ITER narratives and numeric scores from ITERs and non-ITER performance measures, and by studies confirming that narratives reflect constructs deemed important in clinical work. Evidence supporting implications is scant. Conclusions The use of ITER narratives for trainee assessment is generally supported, except that evidence is lacking for implications and decisions. Future research should seek to confirm implicit assumptions and evaluate the impact of decisions.

Objective To succinctly summarise five contemporary theories about motivation to learn, articulate key intersections and distinctions among these theories, and identify important considerations for future research. Results Motivation has been defined as the process whereby goal‐directed activities are initiated and sustained. In expectancy‐value theory, motivation is a function of the expectation of success and perceived value. Attribution theory focuses on the causal attributions learners create to explain the results of an activity, and classifies these in terms of their locus, stability and controllability. Social‐ cognitive theory emphasises self‐efficacy as the primary driver of motivated action, and also identifies cues that influence future self‐efficacy and support self‐regulated learning. Goal orientation theory suggests that learners tend to engage in tasks with concerns about mastering the content (mastery goal, arising from a ‘growth’ mindset regarding intelligence and learning) or about doing better than others or avoiding failure (performance goals, arising from a ‘fixed’ mindset). Finally, self‐determination theory proposes that optimal performance results from actions motivated by intrinsic interests or by extrinsic values that have become integrated and internalised. Satisfying basic psychosocial needs of autonomy, competence and relatedness promotes such motivation. Looking across all five theories, we note recurrent themes of competence, value, attributions, and interactions between individuals and the learning context. Conclusions To avoid conceptual confusion, and perhaps more importantly to maximise the theory‐building potential of their work, researchers must be careful (and precise) in how they define, operationalise and measure different motivational constructs. We suggest that motivation research continue to build theory and extend it to health professions domains, identify key outcomes and outcome measures, and test practical educational applications of the principles thus derived.

Objective: Infobuttons appear as small icons adjacent to electronic health record (EHR) data (e.g., medications, diagnoses, or test results) that, when clicked, access online knowledge resources tailored to the patient, care setting, or task. Infobuttons are required for “Meaningful Use” certification of US EHRs. We sought to evaluate infobuttons’ impact on clinical practice and identify features associated with improved outcomes. Methods: We conducted a systematic review, searching MEDLINE, EMBASE, and other databases from inception to July 6, 2015. We included and cataloged all original research in any language describing implementation of infobuttons or other context-sensitive links. Studies evaluating clinical implementations with outcomes of usage or impact were reviewed in greater detail. Reviewers worked in duplicate to select articles, evaluate quality, and abstract information. Results: Of 599 potential articles, 77 described infobutton implementation. The 17 studies meriting detailed review, including 3 randomized trials, yielded the following findings. Infobutton usage frequency ranged from 0.3 to 7.4 uses per month per potential user. Usage appeared to be influenced by EHR task. Five studies found that infobuttons are used less often than non–context-sensitive links (proportionate usage 0.20–0.34). In 3 studies, users answered their clinical question in &gt; 69% of infobutton sessions. Seven studies evaluated alternative approaches to infobutton design and implementation. No studies isolated the impact of infobuttons on objectively measured patient outcomes. Conclusions: Weak evidence suggests that infobuttons can help providers answer clinical questions. Research on optimal infobutton design and implementation, and on the impact on patient outcomes and provider behaviors, is needed.