Citations

With its first administration of an Objective Structured Clinical Examination (OSCE) in 2018, the American Board of Anesthesiology (ABA) became the first US medical specialty certifying board to incorporate this type of assessment into its high-stakes certification examination system. The fundamental rationale for the ABA’s introduction of the OSCE is to include an assessment that allows candidates for board certification to demonstrate what they actually “do” in domains relevant to clinical practice. Inherent in this rationale is that the OSCE will capture competencies not well assessed in the current written and oral examinations—competencies that will allow the ABA to judge whether a candidate meets the standards expected for board certification more properly. This special article describes the ABA’s journey from initial conceptualization through first administration of the OSCE, including the format of the OSCE, the process for scenario development, the standardized patient program that supports OSCE administration, examiner training, scoring, and future assessment of reliability, validity, and impact of the OSCE. This information will be beneficial to both those involved in the initial certification process, such as residency graduate candidates and program directors, and others contemplating the use of high-stakes summative OSCE assessments.

BACKGROUND: In January 2016, as part of the Maintenance of Certification in Anesthesiology (MOCA) program, the American Board of Anesthesiology launched MOCA Minute, a web-based longitudinal assessment, to supplant the former cognitive examination. We investigated the association between participation and performance in MOCA Minute and disciplinary actions against medical licenses of anesthesiologists. METHODS: All anesthesiologists with time-limited certificates (ie, certified in 2000 or after) who were required to register for MOCA Minute in 2016 were followed up through December 31, 2016. The incidence of postcertification prejudicial license actions was compared between those who did and did not register and compared between registrants who did and did not meet the MOCA Minute performance standard. RESULTS: The cumulative incidence of license actions was 1.2% (245/20,006) in anesthesiologists required to register for MOCA Minute. Nonregistration was associated with a higher incidence of license actions (hazard ratio, 2.93 [95% confidence interval {CI}, 2.15–4.00]). For the 18,534 (92.6%) who registered, later registration (after June 30, 2016) was associated with a higher incidence of license actions. In 2016, 16,308 (88.0%) anesthesiologists met the MOCA Minute performance standard. Of those not meeting the standard (n = 2226), most (n = 2093, 94.0%) failed because they did not complete the required 120 questions. Not meeting the standard was associated with a higher incidence of license actions (hazard ratio, 1.92 [95% CI, 1.36–2.72]). CONCLUSIONS: Both timely participation and meeting performance standard in MOCA Minute are associated with a lower likelihood of being disciplined by a state medical board.

The American Board of Anesthesiology (ABA) has been administering an oral examination as part of its initial certification process since 1939. Among the 24 member boards of the American Board of Medical Specialties, 13 other boards also require passing an oral examination for physicians to become certified in their specialties. However, the methods used to develop, administer, and score these examinations have not been published. The purpose of this report is to describe the history and evolution of the anesthesiology Standardized Oral Examination, its current examination development and administration, the psychometric model and scoring, physician examiner training and auditing, and validity evidence. The many-facet Rasch model is the analytic method used to convert examiner ratings into scaled scores for candidates and takes into account how difficult grader examiners are and the difficulty of the examination tasks. Validity evidence of the oral examination includes that it measures aspects of clinical performance not accounted for by written certifying examinations, and that passing the oral examination is associated with a decreased risk of subsequent license actions against the anesthesiologist. Explaining the details of the Standardized Oral Examination provides transparency about this component of initial certification in anesthesiology.

Abstract Editor’s Perspective What We Already Know about This Topic What This Article Tells Us That Is New Background This repeated cross-sectional survey study was conducted to determine the prevalence of, and factors associated with, burnout, distress, and depression among anesthesiology residents and first-year graduates. We hypothesized that heavy workload and student debt burden were associated with a higher risk of physician burnout, distress, and depression, and that perception of having adequate workplace resources, work–life balance, and social support were associated with a lower risk. Methods Physicians beginning U.S. anesthesiology residency between 2013 and 2016 were invited to take online surveys annually from their clinical anesthesia year 1 to 1 yr after residency graduation. The Maslach Burnout Inventory, the Physician Well-Being Index, and the Harvard Department of Psychiatry/National Depression Screening Day Scale were used to measure burnout, distress, and depression, respectively. Logistic regression analyses were conducted to examine whether self-reported demographics, personal, and professional factors were associated with the risk of burnout, distress, and depression. Results The response rate was 36% (5,295 of 14,529). The prevalence of burnout, distress, and depression was 51% (2,531 of 4,966), 32% (1,575 of 4,941), and 12% (565 of 4,840), respectively. Factors associated with a lower risk of all three outcomes included respondents’ perceived workplace resource availability, (odds ratio = 0.51 [95% CI, 0.45 to 0.57] for burnout; 0.51 [95% CI, 0.45 to 0.56] for distress; 0.52 [95% CI, 0.45 to 0.60] for depression) and perceived ability to maintain work–life balance (0.61 [95% CI, 0.56 to 0.67] for burnout; 0.50 [95% CI, 0.46 to 0.55] for distress; 0.58 [95% CI, 0.51 to 0.65] for depression). A greater number of hours worked per week and a higher amount of student debt were associated with a higher risk of distress and depression, but not burnout. Conclusions Burnout, distress, and depression are notable among anesthesiology residents. Perceived institutional support, work–life balance, strength of social support, workload, and student debt impact physician well-being.

This study compared anesthesiology residency graduates’ written certification examination performance before and after the American Board of Anesthesiology (ABA) introduced the staged examination system. After equating test scores using common test items, the first 2 cohorts (2013, 2014) in the staged system scored 7.1 points and 8.3 points higher than the 2011 baseline cohort in the former examination system. The 2013 and 2014 cohorts’ pass rates (94.2% and 95.9%) were also higher than the 2011 and 2012 cohorts (91.9% and 92.6%) if a common standard had been applied. The staged examination system may be associated with improved knowledge of anesthesiology graduates.

Abstract Background The American Board of Anesthesiology recently introduced the BASIC Examination, a component of its new staged examinations for primary certification, typically offered to residents at the end of their first year of clinical anesthesiology training. This analysis tested the hypothesis that the introduction of the BASIC Examination was associated with an acceleration of knowledge acquisition during the residency training period, as measured by increments in annual In-Training Examination scores. Methods In-Training Examination performance was compared longitudinally among four resident cohorts (n = 6,488) before and after the introduction of the staged system using mixed-effects models that accounted for possible covariates. Results Compared with previous cohorts in the traditional examination system, the first resident cohort in the staged system had a greater improvement in In-Training Examination scores between the first and second years of clinical anesthesiology training (by an estimated 2.0 points in scaled score on a scale of 1 to 50 [95% CI, 1.7 to 2.3]). By their second year, they had achieved a score similar to that of third-year clinical anesthesiology residents in previous cohorts. The second cohort to enter the staged system had a greater improvement of the scores between the clinical base year and the first clinical anesthesiology year, compared with the previous cohorts. Conclusions These results support the hypothesis that the introduction of the BASIC Examination is associated with accelerated knowledge acquisition in residency training and provides evidence for the value of the new staged system in promoting desired educational outcomes of anesthesiology training.

Abstract Background The American Board of Anesthesiology administers written and oral examinations for its primary certification. This retrospective cohort study tested the hypothesis that the risk of a disciplinary action against a physician’s medical license is lower in those who pass both examinations than those who pass only the written examination. Methods Physicians who entered anesthesiology training from 1971 to 2011 were followed up to 2014. License actions were ascertained via the Disciplinary Action Notification Service of the Federation of State Medical Boards. Results The incidence rate of license actions was relatively stable over the study period, with approximately 2 to 3 new cases per 1,000 person-years. In multivariable models, the risk of license actions was higher in men (hazard ratio = 1.88 [95% CI, 1.66 to 2.13]) and lower in international medical graduates (hazard ratio = 0.73 [95% CI, 0.66 to 0.81]). Compared with those passing both examinations on the first attempt, those passing neither examination (hazard ratio = 3.60 [95% CI, 3.14 to 4.13]) and those passing only the written examination (hazard ratio = 3.51 [95% CI, 2.87 to 4.29]) had an increased risk of receiving an action from a state medical board. The risk was no different between the latter two groups ( P = 0.81), showing that passing the oral but not the written primary certification examination is associated with a decreased risk of subsequent license actions. For those with residency performance information available, having at least one unsatisfactory training record independently increased the risk of license actions. Conclusions These findings support the concept that an oral examination assesses domains important to physician performance that are not fully captured in a written examination.

Abstract Background As part of the Maintenance of Certification in Anesthesiology Program® (MOCA®), the American Board of Anesthesiology (Raleigh, North Carolina) developed the MOCA Minute program, a web-based intensive longitudinal assessment involving weekly questions with immediate feedback and links to learning resources. This observational study tested the hypothesis that individuals who participate in the MOCA Minute program perform better on the MOCA Cognitive Examination (CE) compared with those who do not participate. Methods Two separate cohorts of individuals eligible for July 2014 and January 2015 CEs were invited to participate in this pilot. The CE scores for each cohort were compared between those who did and did not participate, controlling for the factors known to affect performance. For the first cohort, examination performances for topics covered and not covered by the MOCA Minute were analyzed separately. Results Six hundred sixteen diplomates in July 2014 and 684 diplomates in January 2015 took the CE for the first time. In multiple regression analysis, those actively participating scored 9.9 points (95% CI, 0.8 to 18.9) and 9.3 points (95% CI, 2.3 to 16.3) higher when compared with those not enrolled, respectively. Compared to the group that did not enroll in MOCA Minute, those who enrolled but did not actively participate demonstrated no improvement in scores. MOCA Minute participation was associated with improvement in both questions covering topics included the MOCA Minute and questions not covering these topics. Conclusions This analysis provides evidence that voluntary active participation in a program featuring frequent knowledge assessments accompanied by targeted learning resources is associated with improved performance on a high-stakes CE.

AbstractAbstract In a nested, matched case-cohort design of 384 anesthesia residents who developed substance use disorder (SUD) and 768 controls who did not, receiving medical education in the United States, but not anesthesia knowledge early in residency, was associated with risk of developing SUD. By the end of follow-up, 54 anesthesiology residents (14.1%) with SUD and 10 controls (1.3%) were dead. Those with SUD were 15-, 10-, and 7-fold more likely to not complete residency, to not become board certified, or have adverse medical licensure actions, respectively. Background: The goal of this work is to evaluate selected risk factors and outcomes for substance use disorder (SUD) in physicians enrolled in anesthesiology residencies approved by the Accreditation Council for Graduate Medical Education. Methods: For each of 384 individuals with evidence of SUD while in primary residency training in anesthesiology from 1975 to 2009, two controls (n = 768) who did not develop SUD were identified and matched for sex, age, primary residency program, and program start date. Risk factors evaluated included location of medical school training (United States vs. other) and anesthesia knowledge as assessed by In-Training Examination performance. Outcomes (assessed to December 31, 2013, with a median follow-up time of 12.2 and 15.1 yr for cases and controls, respectively) included mortality and profession-related outcomes. Results: Receiving medical education within the United States, but not performance on the first in-training examination, was associated with an increased risk of developing SUD as a resident. Cases demonstrated a marked increase in the risk of death after training (hazard ratio, 7.9; 95% CI, 3.1 to 20.5), adverse training outcomes including failure to complete residency (odds ratio, 14.9; 95% CI, 9.0 to 24.6) or become board certified (odds ratio, 10.4; 95% CI, 7.0 to 15.5), and adverse medical licensure actions subsequent to residency (hazard ratio, 6.8; 95% CI, 3.8 to 12.2). As of the end of follow-up, 54 cases (14.1%) were deceased compared with 10 controls (1.3%); 28 cases and no controls died during residency. Conclusion: The attributable risk of SUD to several adverse outcomes during and after residency training, including death and adverse medical license actions, is substantial.