Cook, David A.

Quantitative scores fail to capture all important features of learner performance. This awareness has led to increased use of qualitative data when assessing health professionals. Yet the use of qualitative assessments is hampered by incomplete understanding of their role in forming judgments, and lack of consensus in how to appraise the rigor of judgments therein derived. The authors articulate the role of qualitative assessment as part of a comprehensive program of assessment, and translate the concept of validity to apply to judgments arising from qualitative assessments. They first identify standards for rigor in qualitative research, and then use two contemporary assessment validity frameworks to reorganize these standards for application to qualitative assessment. Standards for rigor in qualitative research include responsiveness, reflexivity, purposive sampling, thick description, triangulation, transparency, and transferability. These standards can be reframed using Messick’s five sources of validity evidence (content, response process, internal structure, relationships with other variables, and consequences) and Kane’s four inferences in validation (scoring, generalization, extrapolation, and implications). Evidence can be collected and evaluated for each evidence source or inference. The authors illustrate this approach using published research on learning portfolios. The authors advocate a “methods-neutral” approach to assessment, in which a clearly stated purpose determines the nature of and approach to data collection and analysis. Increased use of qualitative assessments will necessitate more rigorous judgments of the defensibility (validity) of inferences and decisions. Evidence should be strategically sought to inform a coherent validity argument.

Because tests that do not alter management (i.e., influence decisions and actions) should not be performed, data on the consequences of assessment constitute a critical source of validity evidence. Consequences validity evidence is challenging for many educators to understand, perhaps because it has no counterpart in the older framework of content, criterion, and construct validity. The authors’ purpose is to explain consequences validity evidence and propose a framework for organizing its collection and interpretation. Both clinical and educational assessments can be viewed as interventions. The act of administering or taking a test, the interpretation of scores, and the ensuing decisions and actions influence those being assessed (e.g., patients or students) and other people and systems (e.g., physicians, teachers, hospitals, schools). Consequences validity evidence examines such impacts of assessments. Despite its importance, consequences evidence is reported infrequently in health professions education (range 5%–20% of studies in recent systematic reviews) and is typically limited in scope and rigor. Consequences validity evidence can derive from evaluations of the impact on examinees, educators, schools, or the end target of practice (e.g., patients or health care systems); and the downstream impact of classifications (e.g., different score cut points and labels). Impact can result from the uses of scores or from the assessment activity itself, and can be intended or unintended and beneficial or harmful. Both quantitative and qualitative research methods are useful. The type, quantity, and rigor of consequences evidence required will vary depending on the assessment and the claims for its use.

Context Assessment is central to medical education and the validation of assessments is vital to their use. Earlier validity frameworks suffer from a multiplicity of types of validity or failure to prioritise among sources of validity evidence. Kane's framework addresses both concerns by emphasising key inferences as the assessment progresses from a single observation to a final decision. Evidence evaluating these inferences is planned and presented as a validity argument. Objectives We aim to offer a practical introduction to the key concepts of Kane's framework that educators will find accessible and applicable to a wide range of assessment tools and activities. Results All assessments are ultimately intended to facilitate a defensible decision about the person being assessed. Validation is the process of collecting and interpreting evidence to support that decision. Rigorous validation involves articulating the claims and assumptions associated with the proposed decision (the interpretation/use argument), empirically testing these assumptions, and organising evidence into a coherent validity argument. Kane identifies four inferences in the validity argument: Scoring (translating an observation into one or more scores); Generalisation (using the score[s] as a reflection of performance in a test setting); Extrapolation (using the score[s] as a reflection of real‐world performance), and Implications (applying the score[s] to inform a decision or action). Evidence should be collected to support each of these inferences and should focus on the most questionable assumptions in the chain of inference. Key assumptions (and needed evidence) vary depending on the assessment's intended use or associated decision. Kane's framework applies to quantitative and qualitative assessments, and to individual tests and programmes of assessment. Conclusions Validation focuses on evaluating the key claims, assumptions and inferences that link assessment scores with their intended interpretations and uses. The Implications and associated decisions are the most important inferences in the validity argument.

Context Self‐regulated learning ( SRL ) requires an active learner who has developed a set of processes for managing the achievement of learning goals. Simulation‐based training is one context in which trainees can safely practise learning how to learn. Objectives The purpose of the present study was to evaluate, in the simulation‐based training context, the effectiveness of interventions designed to support trainees in SRL activities. We used the social‐cognitive model of SRL to guide a systematic review and meta‐analysis exploring the links between instructor supervision, supports or scaffolds for SRL , and educational outcomes. Methods We searched databases including MEDLINE and Scopus, and previous reviews, for material published until December 2011. Studies comparing simulation‐based SRL interventions with another intervention for teaching health professionals were included. Reviewers worked independently and in duplicate to extract information on learners, study quality and educational outcomes. We used random‐effects meta‐analysis to compare the effects of supervision (instructor present or absent) and SRL educational supports (e.g. goal‐setting study guides present or absent). Results From 11 064 articles, we included 32 studies enrolling 2482 trainees. Only eight of the 32 studies included educational supports for SRL . Compared with instructor‐supervised interventions, unsupervised interventions were associated with poorer immediate post‐test outcomes (pooled effect size: −0.34, p = 0.09; n = 19 studies) and negligible effects on delayed (i.e. > 1 week) retention tests (pooled effect size: 0.11, p = 0.63; n = 8 studies). Interventions including SRL supports were associated with small benefits compared with interventions without supports on both immediate post‐tests (pooled effect size: 0.23, p = 0.22; n = 5 studies) and delayed retention tests (pooled effect size: 0.44, p = 0.067; n = 3 studies). Conclusions Few studies in the simulation literature have designed SRL training to explicitly support trainees’ capacity to self‐regulate their learning. We recommend that educators and researchers shift from thinking about SRL as learning alone to thinking of SRL as comprising a shared responsibility between the trainee and the instructional designer (i.e. learning using designed supports that help prepare individuals for future learning).

Context The relative advantages and disadvantages of checklists and global rating scales ( GRS s) have long been debated. To compare the merits of these scale types, we conducted a systematic review of the validity evidence for checklists and GRS s in the context of simulation‐based assessment of health professionals. Methods We conducted a systematic review of multiple databases including MEDLINE , EMBASE and S copus to F ebruary 2013. We selected studies that used both a GRS and checklist in the simulation‐based assessment of health professionals. Reviewers working in duplicate evaluated five domains of validity evidence, including correlation between scales and reliability. We collected information about raters, instrument characteristics, assessment context, and task. We pooled reliability and correlation coefficients using random‐effects meta‐analysis. Results We found 45 studies that used a checklist and GRS in simulation‐based assessment. All studies included physicians or physicians in training; one study also included nurse anaesthetists. Topics of assessment included open and laparoscopic surgery ( n = 22), endoscopy ( n = 8), resuscitation ( n = 7) and anaesthesiology ( n = 4). The pooled GRS –checklist correlation was 0.76 (95% confidence interval [ CI ] 0.69–0.81, n = 16 studies). Inter‐rater reliability was similar between scales ( GRS 0.78, 95% CI 0.71–0.83, n = 23; checklist 0.81, 95% CI 0.75–0.85, n = 21), whereas GRS inter‐item reliabilities (0.92, 95% CI 0.84–0.95, n = 6) and inter‐station reliabilities (0.80, 95% CI 0.73–0.85, n = 10) were higher than those for checklists (0.66, 95% CI 0–0.84, n = 4 and 0.69, 95% CI 0.56–0.77, n = 10, respectively). Content evidence for GRS s usually referenced previously reported instruments ( n = 33), whereas content evidence for checklists usually described expert consensus ( n = 26). Checklists and GRS s usually had similar evidence for relations to other variables. Conclusions Checklist inter‐rater reliability and trainee discrimination were more favourable than suggested in earlier work, but each task requires a separate checklist. Compared with the checklist, the GRS has higher average inter‐item and inter‐station reliability, can be used across multiple tasks, and may better capture nuanced elements of expertise.

Context Studies that investigate research questions that have already been resolved represent a waste of resources. However, the failure to collect sufficient evidence to resolve a given question results in ambiguity. Objectives The present study was conducted to reanalyse the results of a meta‐analysis of simulation‐based education ( SBE ) to determine: (i) whether researchers continue to replicate research studies after the answer to a research question has become known, and (ii) whether researchers perform enough replications to definitively answer important questions. Methods A systematic search of multiple databases to May 2011 was conducted to identify original research evaluating SBE for health professionals in comparison with no intervention or any active intervention, using skill outcomes. Data were extracted by reviewers working in duplicate. Data synthesis involved a cumulative meta‐analysis to illuminate patterns of evidence by sequentially adding studies according to a variable of interest (e.g. publication year) and re‐calculating the pooled effect size with each addition. Cumulative meta‐analysis by publication year was applied to 592 comparative studies using several thresholds of ‘sufficiency’, including: statistical significance; stable effect size classification and magnitude (Hedges’ g ± 0.1), and precise estimates (confidence intervals of less than ± 0.2). Results Among studies that compared the outcomes of SBE with those of no intervention, evidence supporting a favourable effect of SBE on skills existed as early as 1973 (one publication) and further evidence confirmed a quantitatively large effect of SBE by 1997 (28 studies). Since then, a further 404 studies were published. Among studies comparing SBE with non‐simulation instruction, the effect initially favoured non‐simulation training, but the addition of a third study in 1997 brought the pooled effect to slightly favour simulation, and by 2004 (14 studies) this effect was statistically significant (p < 0.05) and the magnitude had stabilised (small effect). A further 37 studies were published after 2004. By contrast, evidence from studies evaluating repetition continued to show borderline statistical significance and wide confidence intervals in 2011. Conclusions Some replication is necessary to obtain stable estimates of effect and to explore different contexts, but the number of studies of SBE often exceeds the minimum number of replications required.