Lien, Cynthia A.

In 2018, the American Board of Anesthesiology (ABA) became the first US medical specialty certifying board to incorporate an Objective Structured Clinical Examination (OSCE) into its initial certification examination system. Previously, the ABA’s staged examination system consisted of 2 written examinations (the BASIC and ADVANCED examinations) and the Standardized Oral Examination (SOE). The OSCE and the existing SOE are now 2 separate components of the APPLIED Examination. This report presents the results of the first-year OSCE administration. A total of 1410 candidates took both the OSCE and the SOE in 2018. Candidate performance approximated a normal distribution for both the OSCE and the SOE, and was not associated with the timing of the examination, including day of the week, morning versus afternoon session, and order of the OSCE and the SOE. Practice-based Learning and Improvement was the most difficult station, while Application of Ultrasonography was the least difficult. The correlation coefficient between SOE and OSCE scores was 0.35 ([95% confidence interval {CI}, 0.30–0.39]; P < .001). Scores for the written ADVANCED Examination were modestly correlated with scores for the SOE ( r = 0.29 [95% CI, 0.25–0.34]; P < .001) and the OSCE ( r = 0.15 [95% CI, 0.10–0.20]; P < .001). Most of the candidates who failed the SOE passed the OSCE, and most of the candidates who failed the OSCE passed the SOE. Of the 1410 candidates, 77 (5.5%) failed the OSCE, 155 (11.0%) failed the SOE, and 25 (1.8%) failed both. Thus, 207 (14.7%) failed at least 1 component of the APPLIED Examination. Adding an OSCE to a board certification examination system is feasible. Preliminary evidence indicates that the OSCE measures aspects of candidate abilities distinct from those measured by other examinations used for initial board certification.

With its first administration of an Objective Structured Clinical Examination (OSCE) in 2018, the American Board of Anesthesiology (ABA) became the first US medical specialty certifying board to incorporate this type of assessment into its high-stakes certification examination system. The fundamental rationale for the ABA’s introduction of the OSCE is to include an assessment that allows candidates for board certification to demonstrate what they actually “do” in domains relevant to clinical practice. Inherent in this rationale is that the OSCE will capture competencies not well assessed in the current written and oral examinations—competencies that will allow the ABA to judge whether a candidate meets the standards expected for board certification more properly. This special article describes the ABA’s journey from initial conceptualization through first administration of the OSCE, including the format of the OSCE, the process for scenario development, the standardized patient program that supports OSCE administration, examiner training, scoring, and future assessment of reliability, validity, and impact of the OSCE. This information will be beneficial to both those involved in the initial certification process, such as residency graduate candidates and program directors, and others contemplating the use of high-stakes summative OSCE assessments.

Abstract Background The American Board of Anesthesiology recently introduced the BASIC Examination, a component of its new staged examinations for primary certification, typically offered to residents at the end of their first year of clinical anesthesiology training. This analysis tested the hypothesis that the introduction of the BASIC Examination was associated with an acceleration of knowledge acquisition during the residency training period, as measured by increments in annual In-Training Examination scores. Methods In-Training Examination performance was compared longitudinally among four resident cohorts (n = 6,488) before and after the introduction of the staged system using mixed-effects models that accounted for possible covariates. Results Compared with previous cohorts in the traditional examination system, the first resident cohort in the staged system had a greater improvement in In-Training Examination scores between the first and second years of clinical anesthesiology training (by an estimated 2.0 points in scaled score on a scale of 1 to 50 [95% CI, 1.7 to 2.3]). By their second year, they had achieved a score similar to that of third-year clinical anesthesiology residents in previous cohorts. The second cohort to enter the staged system had a greater improvement of the scores between the clinical base year and the first clinical anesthesiology year, compared with the previous cohorts. Conclusions These results support the hypothesis that the introduction of the BASIC Examination is associated with accelerated knowledge acquisition in residency training and provides evidence for the value of the new staged system in promoting desired educational outcomes of anesthesiology training.

Abstract Background As part of the Maintenance of Certification in Anesthesiology Program® (MOCA®), the American Board of Anesthesiology (Raleigh, North Carolina) developed the MOCA Minute program, a web-based intensive longitudinal assessment involving weekly questions with immediate feedback and links to learning resources. This observational study tested the hypothesis that individuals who participate in the MOCA Minute program perform better on the MOCA Cognitive Examination (CE) compared with those who do not participate. Methods Two separate cohorts of individuals eligible for July 2014 and January 2015 CEs were invited to participate in this pilot. The CE scores for each cohort were compared between those who did and did not participate, controlling for the factors known to affect performance. For the first cohort, examination performances for topics covered and not covered by the MOCA Minute were analyzed separately. Results Six hundred sixteen diplomates in July 2014 and 684 diplomates in January 2015 took the CE for the first time. In multiple regression analysis, those actively participating scored 9.9 points (95% CI, 0.8 to 18.9) and 9.3 points (95% CI, 2.3 to 16.3) higher when compared with those not enrolled, respectively. Compared to the group that did not enroll in MOCA Minute, those who enrolled but did not actively participate demonstrated no improvement in scores. MOCA Minute participation was associated with improvement in both questions covering topics included the MOCA Minute and questions not covering these topics. Conclusions This analysis provides evidence that voluntary active participation in a program featuring frequent knowledge assessments accompanied by targeted learning resources is associated with improved performance on a high-stakes CE.