Sun, Huaping

Background: Longitudinal assessments are increasingly common as a component of continuing certification programs offered by medical specialty boards. The goal of this study was to understand anesthesiologists’ utilization pattern, performance, and perception of the longitudinal assessment program administered by the American Board of Anesthesiology (Raleigh, North Carolina). Methods: The authors analyzed assessment usage and survey data from all anesthesiologists who participated in the Maintenance of Certification in Anesthesiology (MOCA) Minute from January 2018 to December 2019. Results: During the 24-month study period, 25,404 anesthesiologists completed all 120 questions assigned in at least 1 calendar year, collectively 5,801,160 responses to 1,646 unique questions. Participants were more likely to complete questions at the end of each quarter, with the highest frequency at the end of the year. Seventy-five percent of responses were first-time attempts, and 17% were first repeats. The median interval between original question presentation and first repeat was 314 days. The mean ± SD percent correct score was 68.3 ± 8.0% for first-time questions and 75.5 ± 12.6% for first repeats. The mean ± SD time to answer a first-time question was 26.8 ± 13.1 s, with shorter times for questions with correct than incorrect responses (25.0 ± 12.7 vs . 30.7 ± 13.3 s; time difference, −5.69 s; 95% CI, −5.71 to −5.66; P < 0.001). Of 5.11 million responses with ratings of relevance (88.2% of all responses analyzed), 4.86 million (95.2%) were felt to be somewhat or very relevant to practice. The percent correct scores were higher for responses with higher relevance and confidence ratings. For responses rated “not relevant,” “somewhat relevant,” and “very relevant,” the percent correct was 58.5%, 67.6%, and 77.0%, respectively; for responses rated as “unsure,” “somewhat confident,” and “very confident,” the percent correct was 53.6%, 67.9%, and 84.4%, respectively. Conclusions: MOCA Minute was generally used as designed. Evidence of learning was suggested by improved performance on repeat questions.

Abstract Purpose The COVID-19 pandemic prompted training institutions and national credentialing organizations to administer examinations virtually. This study compared task difficulty, examiner grading, candidate performance, and other psychometric properties between in-person and virtual standardized oral examinations (SOEs) administered by the American Board of Anesthesiology. Method This retrospective study included SOEs administered in person from March 2018 to March 2020 and virtually from December 2020 to November 2021. The in-person and virtual SOEs share the same structure, including 4 tasks of preoperative evaluation, intraoperative management, postoperative care, and additional topics. The Many-Facet Rasch Model was used to estimate candidate performance, examiner grading severity, and task difficulty for the in-person and virtual SOEs separately; the virtual SOE was equated to the in-person SOE by common examiners and all tasks. The independent-samples and partially overlapping-samples t tests were used to compare candidate performance and examiner grading severity between these 2 formats, respectively. Results In-person (n = 3,462) and virtual (n = 2,959) first-time candidates were comparable in age, sex, race and ethnicity, and whether they were U.S. medical school graduates. The mean (standard deviation [SD]) candidate performance was 2.96 (1.76) logits for the virtual SOE, which was statistically significantly better than that for the in-person SOE (mean [SD], 2.86 [1.75]; Welch independent-samples t test, P = .02); however, the effect size was negligible (Cohen d = 0.06). The difference in the grading severity of examiners who rated the in-person (n = 398; mean [SD], 0.00 [0.73]) versus virtual (n = 341; mean [SD], 0.07 [0.77]) SOE was not statistically significant (Welch partially overlapping-samples t test, P = .07). Conclusions Candidate performance and examiner grading severity were comparable between the in-person and virtual SOEs, supporting the reliability and validity of the virtual oral exam in this large-volume, high-stakes setting.

Abstract Editor’s Perspective What We Already Know about This Topic What This Article Tells Us That Is New Background This repeated cross-sectional survey study was conducted to determine the prevalence of, and factors associated with, burnout, distress, and depression among anesthesiology residents and first-year graduates. We hypothesized that heavy workload and student debt burden were associated with a higher risk of physician burnout, distress, and depression, and that perception of having adequate workplace resources, work–life balance, and social support were associated with a lower risk. Methods Physicians beginning U.S. anesthesiology residency between 2013 and 2016 were invited to take online surveys annually from their clinical anesthesia year 1 to 1 yr after residency graduation. The Maslach Burnout Inventory, the Physician Well-Being Index, and the Harvard Department of Psychiatry/National Depression Screening Day Scale were used to measure burnout, distress, and depression, respectively. Logistic regression analyses were conducted to examine whether self-reported demographics, personal, and professional factors were associated with the risk of burnout, distress, and depression. Results The response rate was 36% (5,295 of 14,529). The prevalence of burnout, distress, and depression was 51% (2,531 of 4,966), 32% (1,575 of 4,941), and 12% (565 of 4,840), respectively. Factors associated with a lower risk of all three outcomes included respondents’ perceived workplace resource availability, (odds ratio = 0.51 [95% CI, 0.45 to 0.57] for burnout; 0.51 [95% CI, 0.45 to 0.56] for distress; 0.52 [95% CI, 0.45 to 0.60] for depression) and perceived ability to maintain work–life balance (0.61 [95% CI, 0.56 to 0.67] for burnout; 0.50 [95% CI, 0.46 to 0.55] for distress; 0.58 [95% CI, 0.51 to 0.65] for depression). A greater number of hours worked per week and a higher amount of student debt were associated with a higher risk of distress and depression, but not burnout. Conclusions Burnout, distress, and depression are notable among anesthesiology residents. Perceived institutional support, work–life balance, strength of social support, workload, and student debt impact physician well-being.

Abstract Background As part of the Maintenance of Certification in Anesthesiology Program® (MOCA®), the American Board of Anesthesiology (Raleigh, North Carolina) developed the MOCA Minute program, a web-based intensive longitudinal assessment involving weekly questions with immediate feedback and links to learning resources. This observational study tested the hypothesis that individuals who participate in the MOCA Minute program perform better on the MOCA Cognitive Examination (CE) compared with those who do not participate. Methods Two separate cohorts of individuals eligible for July 2014 and January 2015 CEs were invited to participate in this pilot. The CE scores for each cohort were compared between those who did and did not participate, controlling for the factors known to affect performance. For the first cohort, examination performances for topics covered and not covered by the MOCA Minute were analyzed separately. Results Six hundred sixteen diplomates in July 2014 and 684 diplomates in January 2015 took the CE for the first time. In multiple regression analysis, those actively participating scored 9.9 points (95% CI, 0.8 to 18.9) and 9.3 points (95% CI, 2.3 to 16.3) higher when compared with those not enrolled, respectively. Compared to the group that did not enroll in MOCA Minute, those who enrolled but did not actively participate demonstrated no improvement in scores. MOCA Minute participation was associated with improvement in both questions covering topics included the MOCA Minute and questions not covering these topics. Conclusions This analysis provides evidence that voluntary active participation in a program featuring frequent knowledge assessments accompanied by targeted learning resources is associated with improved performance on a high-stakes CE.