Cook, David A.

Purpose To identify features of instruments, test procedures, study design, and validity evidence in published studies of electrocardiogram (ECG) skill assessments. Method The authors conducted a systematic review, searching MEDLINE, Embase, Cochrane CENTRAL, PsycINFO, CINAHL, ERIC, and Web of Science databases in February 2020 for studies that assessed the ECG interpretation skill of physicians or medical students. Two authors independently screened articles for inclusion and extracted information on test features, study design, risk of bias, and validity evidence. Results The authors found 85 eligible studies. Participants included medical students (42 studies), postgraduate physicians (48 studies), and practicing physicians (13 studies). ECG selection criteria were infrequently reported: 25 studies (29%) selected single-diagnosis or straightforward ECGs; 5 (6%) selected complex cases. ECGs were selected by generalists (15 studies [18%]), cardiologists (10 studies [12%]), or unspecified experts (4 studies [5%]). The median number of ECGs per test was 10. The scoring rubric was defined by 2 or more experts in 32 studies (38%), by 1 expert in 5 (6%), and using clinical data in 5 (6%). Scoring was performed by a human rater in 34 studies (40%) and by computer in 7 (8%). Study methods were appraised as low risk of selection bias in 16 studies (19%), participant flow bias in 59 (69%), instrument conduct and scoring bias in 20 (24%), and applicability problems in 56 (66%). Evidence of test score validity was reported infrequently, namely evidence of content (39 studies [46%]), internal structure (11 [13%]), relations with other variables (10 [12%]), response process (2 [2%]), and consequences (3 [4%]). Conclusions ECG interpretation skill assessments consist of idiosyncratic instruments that are too short, composed of items of obscure provenance, with incompletely specified answers, graded by individuals with underreported credentials, yielding scores with limited interpretability. The authors suggest several best practices.

Purpose To explore what influences clinicians in selecting continuing medical education (CME) activities in the United States. Method In August 2018, the authors conducted an Internet-based national survey, sampling 100 respondents from each of 5 groups: family medicine physicians, internal medicine and hospitalist physicians, medicine specialist physicians, nurse practitioners, and physician assistants. In total, 1,895 clinicians were invited and 500 (26%) responded. Questions addressed the selection and anticipated use of CME delivery modalities and perceived characteristics of specific CME providers. Response formats used best–worst scaling or 5-point ordinal response options. Results The factors identified as most important in selecting CME activities were topic (best–worst scaling net positivity 0.54), quality of content (0.51), availability of CME credit (0.43), and clinical practice focus (0.41), while referral frequency (−0.57) ranked lowest. The activities that the respondents anticipated using most in the future were live (mean 3.8 [1 = not likely, 5 = very likely]), online (mean 3.5), point-of-care (mean 3.5), and print-based (mean 3.5) activities. For online CME, the features of greatest appeal were that learning could be done when clinicians had time (mean 4.4), at their own pace (mean 4.2), and at lower cost (mean 4.2). For live CME, the features of greatest appeal were that the subject was best taught using this modality (mean 4.0), or the activity was located in a destination spot (mean 4.0) or a regional location (mean 3.9). When rating specific CME providers, most academic institutions received relatively high ratings for research focus and clinical practice focus, whereas commercial providers had slightly higher ratings for ease of access. Responses were generally similar across clinician types and age groups. Conclusions Physicians, nurse practitioners, and physician assistants are interested in using a variety of CME delivery modalities. Appealing features of online and live CME were different.

Context High‐quality research into education costs can inform better decision making. Improvements to cost research can be guided by information about the research questions, methods and reporting of studies evaluating costs in health professions education (HPE). Our objective was to appraise the overall state of the field and evaluate temporal trends in the methods and reporting quality of cost evaluations in HPE research. Methods We searched the MEDLINE , CINAHL (Cumulative Index to Nursing and Allied Health Literature), EMBASE , Business Source Complete and ERIC (Education Resources Information Centre) databases on 31 July 2017. To evaluate trends over time, we sampled research reports at 5‐year intervals (2001, 2006, 2011 and 2016). All original research studies in HPE that reported a cost outcome were included. The Medical Education Research Study Quality Instrument ( MERSQI ) and the BMJ economic checklist were used to appraise methodological and reporting quality, respectively. Trends in quality over time were analysed. Results A total of 78 studies were included, of which 16 were published in 2001, 15 in 2006, 20 in 2011 and 27 in 2016. The region most commonly represented was the USA ( n = 43). The profession most commonly referred to was that of the physician ( n = 46). The mean ± standard deviation ( SD ) MERSQI score was 10.9 ± 2.6 out of 18, with no significant change over time (p = 0.55). The mean ± SD BMJ score was 13.5 ± 7.1 out of 35, with no significant change over time (p = 0.39). A total of 49 (63%) studies stated a cost‐related research question, 23 (29%) stated the type of cost evaluation used, and 31 (40%) described the method of estimating resource quantities and unit costs. A total of 16 studies compared two or more interventions and reported both cost and learning outcomes. Conclusions The absolute number of cost evaluations in HPE is increasing. However, there are shortcomings in the quality of methodology and reporting, and these are not improving over time.

Purpose To examine how qualitative narrative comments and quantitative ratings from end-of-rotation assessments change for a cohort of residents from entry to graduation, and explore associations between comments and ratings. Method The authors obtained end-of-rotation quantitative ratings and narrative comments for 1 cohort of internal medicine residents at the University of Illinois at Chicago College of Medicine from July 2013–June 2016. They inductively identified themes in comments, coded orientation (praising/critical) and relevance (specificity and actionability) of feedback, examined associations between codes and ratings, and evaluated changes in themes and ratings across years. Results Data comprised 1,869 assessments (828 comments) on 33 residents. Five themes aligned with ACGME competencies (interpersonal and communication skills, professionalism, medical knowledge, patient care, and systems-based practice), and 3 did not (personal attributes, summative judgment, and comparison to training level). Work ethic was the most frequent subtheme. Comments emphasized medical knowledge more in year 1 and focused more on autonomy, leadership, and teaching in later years. Most comments (714/828 [86%]) contained high praise, and 412/828 (50%) were very relevant. Average ratings correlated positively with orientation (β = 0.46, P < .001) and negatively with relevance (β = −0.09, P = .01). Ratings increased significantly with each training year (year 1, mean [standard deviation]: 5.31 [0.59]; year 2: 5.58 [0.47]; year 3: 5.86 [0.43]; P < .001). Conclusions Narrative comments address resident attributes beyond the ACGME competencies and change as residents progress. Lower quantitative ratings are associated with more specific and actionable feedback.

Substantial research has illuminated the clinical reasoning processes involved in diagnosis (diagnostic reasoning). Far less is known about the processes entailed in patient management (management reasoning), including decisions about treatment, further testing, follow-up visits, and allocation of limited resources. The authors’ purpose is to articulate key differences between diagnostic and management reasoning, implications for health professions education, and areas of needed research. Diagnostic reasoning focuses primarily on classification (i.e., assigning meaningful labels to a pattern of symptoms, signs, and test results). Management reasoning involves negotiation of a plan and ongoing monitoring/adjustment of that plan. A diagnosis can usually be established as correct or incorrect, whereas there are typically multiple reasonable management approaches. Patient preferences, clinician attitudes, clinical contexts, and logistical constraints should not influence diagnosis, whereas management nearly always involves prioritization among such factors. Diagnostic classifications do not necessarily require direct patient interaction, whereas management prioritizations require communication and negotiation. Diagnoses can be defined at a single time point (given enough information), whereas management decisions are expected to evolve over time. Finally, management is typically more complex than diagnosis. Management reasoning may require educational approaches distinct from those used for diagnostic reasoning, including teaching distinct skills (e.g., negotiating with patients, tolerating uncertainty, and monitoring treatment) and developing assessments that account for underlying reasoning processes and multiple acceptable solutions. Areas of needed research include if and how cognitive processes differ for management and diagnostic reasoning, how and when management reasoning abilities develop, and how to support management reasoning in clinical practice.