Hatala, Rose

Abstract Purpose Assessing expertise using psychometric models usually yields a measure of ability that is difficult to generalize to the complexity of diagnoses in clinical practice. However, using an item response modeling framework, it is possible to create a decision-aligned response model that captures a clinician’s decision-making behavior on a continuous scale that fully represents competing diagnostic possibilities. In this proof-of-concept study, the authors demonstrate the necessary statistical conceptualization of this model using a specific electrocardiogram (ECG) example. Method The authors collected a range of ECGs with elevated ST segments due to either ST-elevation myocardial infarction (STEMI) or pericarditis. Based on pilot data, 20 ECGs were chosen to represent a continuum from “definitely STEMI” to “definitely pericarditis,” including intermediate cases in which the diagnosis was intentionally unclear. Emergency medicine and cardiology physicians rated these ECGs on a 5-point scale (“definitely STEMI” to “definitely pericarditis”). The authors analyzed these ratings using a graded response model showing the degree to which each participant could separate the ECGs along the diagnostic continuum. The authors compared these metrics with the discharge diagnoses noted on chart review. Results Thirty-seven participants rated the ECGs. As desired, the ECGs represented a range of phenotypes, including cases where participants were uncertain in their diagnosis. The response model showed that participants varied both in their propensity to diagnose one condition over another and in where they placed the thresholds between the 5 diagnostic categories. The most capable participants were able to meaningfully use all categories, with precise thresholds between categories. Conclusions The authors present a decision-aligned response model that demonstrates the confusability of a particular ECG and the skill with which a clinician can distinguish 2 diagnoses along a continuum of confusability. These results have broad implications for testing and for learning to manage uncertainty in diagnosis.

Purpose In-training evaluation reports (ITERs) constitute an integral component of medical student and postgraduate physician trainee (resident) assessment. ITER narrative comments have received less attention than the numeric scores. The authors sought both to determine what validity evidence informs the use of narrative comments from ITERs for assessing medical students and residents and to identify evidence gaps. Method Reviewers searched for relevant English-language studies in MEDLINE, EMBASE, Scopus, and ERIC (last search June 5, 2015), and in reference lists and author files. They included all original studies that evaluated ITERs for qualitative assessment of medical students and residents. Working in duplicate, they selected articles for inclusion, evaluated quality, and abstracted information on validity evidence using Kane’s framework (inferences of scoring, generalization, extrapolation, and implications). Results Of 777 potential articles, 22 met inclusion criteria. The scoring inference is supported by studies showing that rich narratives are possible, that changing the prompt can stimulate more robust narratives, and that comments vary by context. Generalization is supported by studies showing that narratives reach thematic saturation and that analysts make consistent judgments. Extrapolation is supported by favorable relationships between ITER narratives and numeric scores from ITERs and non-ITER performance measures, and by studies confirming that narratives reflect constructs deemed important in clinical work. Evidence supporting implications is scant. Conclusions The use of ITER narratives for trainee assessment is generally supported, except that evidence is lacking for implications and decisions. Future research should seek to confirm implicit assumptions and evaluate the impact of decisions.

Quantitative scores fail to capture all important features of learner performance. This awareness has led to increased use of qualitative data when assessing health professionals. Yet the use of qualitative assessments is hampered by incomplete understanding of their role in forming judgments, and lack of consensus in how to appraise the rigor of judgments therein derived. The authors articulate the role of qualitative assessment as part of a comprehensive program of assessment, and translate the concept of validity to apply to judgments arising from qualitative assessments. They first identify standards for rigor in qualitative research, and then use two contemporary assessment validity frameworks to reorganize these standards for application to qualitative assessment. Standards for rigor in qualitative research include responsiveness, reflexivity, purposive sampling, thick description, triangulation, transparency, and transferability. These standards can be reframed using Messick’s five sources of validity evidence (content, response process, internal structure, relationships with other variables, and consequences) and Kane’s four inferences in validation (scoring, generalization, extrapolation, and implications). Evidence can be collected and evaluated for each evidence source or inference. The authors illustrate this approach using published research on learning portfolios. The authors advocate a “methods-neutral” approach to assessment, in which a clearly stated purpose determines the nature of and approach to data collection and analysis. Increased use of qualitative assessments will necessitate more rigorous judgments of the defensibility (validity) of inferences and decisions. Evidence should be strategically sought to inform a coherent validity argument.

Context Assessment is central to medical education and the validation of assessments is vital to their use. Earlier validity frameworks suffer from a multiplicity of types of validity or failure to prioritise among sources of validity evidence. Kane's framework addresses both concerns by emphasising key inferences as the assessment progresses from a single observation to a final decision. Evidence evaluating these inferences is planned and presented as a validity argument. Objectives We aim to offer a practical introduction to the key concepts of Kane's framework that educators will find accessible and applicable to a wide range of assessment tools and activities. Results All assessments are ultimately intended to facilitate a defensible decision about the person being assessed. Validation is the process of collecting and interpreting evidence to support that decision. Rigorous validation involves articulating the claims and assumptions associated with the proposed decision (the interpretation/use argument), empirically testing these assumptions, and organising evidence into a coherent validity argument. Kane identifies four inferences in the validity argument: Scoring (translating an observation into one or more scores); Generalisation (using the score[s] as a reflection of performance in a test setting); Extrapolation (using the score[s] as a reflection of real‐world performance), and Implications (applying the score[s] to inform a decision or action). Evidence should be collected to support each of these inferences and should focus on the most questionable assumptions in the chain of inference. Key assumptions (and needed evidence) vary depending on the assessment's intended use or associated decision. Kane's framework applies to quantitative and qualitative assessments, and to individual tests and programmes of assessment. Conclusions Validation focuses on evaluating the key claims, assumptions and inferences that link assessment scores with their intended interpretations and uses. The Implications and associated decisions are the most important inferences in the validity argument.

Context Self‐regulated learning ( SRL ) requires an active learner who has developed a set of processes for managing the achievement of learning goals. Simulation‐based training is one context in which trainees can safely practise learning how to learn. Objectives The purpose of the present study was to evaluate, in the simulation‐based training context, the effectiveness of interventions designed to support trainees in SRL activities. We used the social‐cognitive model of SRL to guide a systematic review and meta‐analysis exploring the links between instructor supervision, supports or scaffolds for SRL , and educational outcomes. Methods We searched databases including MEDLINE and Scopus, and previous reviews, for material published until December 2011. Studies comparing simulation‐based SRL interventions with another intervention for teaching health professionals were included. Reviewers worked independently and in duplicate to extract information on learners, study quality and educational outcomes. We used random‐effects meta‐analysis to compare the effects of supervision (instructor present or absent) and SRL educational supports (e.g. goal‐setting study guides present or absent). Results From 11 064 articles, we included 32 studies enrolling 2482 trainees. Only eight of the 32 studies included educational supports for SRL . Compared with instructor‐supervised interventions, unsupervised interventions were associated with poorer immediate post‐test outcomes (pooled effect size: −0.34, p = 0.09; n = 19 studies) and negligible effects on delayed (i.e. > 1 week) retention tests (pooled effect size: 0.11, p = 0.63; n = 8 studies). Interventions including SRL supports were associated with small benefits compared with interventions without supports on both immediate post‐tests (pooled effect size: 0.23, p = 0.22; n = 5 studies) and delayed retention tests (pooled effect size: 0.44, p = 0.067; n = 3 studies). Conclusions Few studies in the simulation literature have designed SRL training to explicitly support trainees’ capacity to self‐regulate their learning. We recommend that educators and researchers shift from thinking about SRL as learning alone to thinking of SRL as comprising a shared responsibility between the trainee and the instructional designer (i.e. learning using designed supports that help prepare individuals for future learning).

Context The relative advantages and disadvantages of checklists and global rating scales ( GRS s) have long been debated. To compare the merits of these scale types, we conducted a systematic review of the validity evidence for checklists and GRS s in the context of simulation‐based assessment of health professionals. Methods We conducted a systematic review of multiple databases including MEDLINE , EMBASE and S copus to F ebruary 2013. We selected studies that used both a GRS and checklist in the simulation‐based assessment of health professionals. Reviewers working in duplicate evaluated five domains of validity evidence, including correlation between scales and reliability. We collected information about raters, instrument characteristics, assessment context, and task. We pooled reliability and correlation coefficients using random‐effects meta‐analysis. Results We found 45 studies that used a checklist and GRS in simulation‐based assessment. All studies included physicians or physicians in training; one study also included nurse anaesthetists. Topics of assessment included open and laparoscopic surgery ( n = 22), endoscopy ( n = 8), resuscitation ( n = 7) and anaesthesiology ( n = 4). The pooled GRS –checklist correlation was 0.76 (95% confidence interval [ CI ] 0.69–0.81, n = 16 studies). Inter‐rater reliability was similar between scales ( GRS 0.78, 95% CI 0.71–0.83, n = 23; checklist 0.81, 95% CI 0.75–0.85, n = 21), whereas GRS inter‐item reliabilities (0.92, 95% CI 0.84–0.95, n = 6) and inter‐station reliabilities (0.80, 95% CI 0.73–0.85, n = 10) were higher than those for checklists (0.66, 95% CI 0–0.84, n = 4 and 0.69, 95% CI 0.56–0.77, n = 10, respectively). Content evidence for GRS s usually referenced previously reported instruments ( n = 33), whereas content evidence for checklists usually described expert consensus ( n = 26). Checklists and GRS s usually had similar evidence for relations to other variables. Conclusions Checklist inter‐rater reliability and trainee discrimination were more favourable than suggested in earlier work, but each task requires a separate checklist. Compared with the checklist, the GRS has higher average inter‐item and inter‐station reliability, can be used across multiple tasks, and may better capture nuanced elements of expertise.