Joldersma, Kevin B.

AbstractBackgroundThe American Board of Emergency Medicine (ABEM) in‐person Oral Certification Examination (OCE) was halted abruptly in 2020 due to the COVID‐19 pandemic. The OCE was reconfigured to be administered in a virtual environment starting in December 2020.ObjectivesThe purpose of this investigation was to determine whether there was sufficient validity and reliability evidence to support the continued use of the ABEM virtual Oral Examination (VOE) for certification decisions.MethodsThis retrospective, descriptive study used multiple data sources to provide validity evidence and reliability data. Validity evidence focused on test content, response processes, internal structure (e.g., internal consistency and item response theory), and the consequences of testing. A multifaceted Rasch reliability coefficient was used to measure reliability. Study data were from two 2019 in‐person OCEs and the first four VOE administrations.ResultsThere were 2279 physicians who took the 2019 in‐person OCE examination and 2153 physicians who took the VOE during the study period. Among the OCE group, 92.0% agreed or strongly agreed that the cases on the examination were cases that an emergency physician should be expected to see; 91.1% of the VOE group agreed or strongly agreed. A similar pattern of responses given to a question about whether the cases on the examination were cases that they had seen. Additional evidence of validity was obtained by the use of the EM Model, the process for case development, the use of think‐aloud protocols, and similar test performance patterns (e.g., pass rates). For reliability, the Rasch reliability coefficients for the OCE and the VOE during the study period were all >0.90.ConclusionsThere was substantial validity evidence and reliability to support ongoing use of the ABEM VOE to make confident and defensible certification decisions.

AbstractObjectivesBurnout afflicts emergency physicians (EPs) to a significant degree. The impact of burnout spans from decreased clinical efficiency to increased medical errors to heightened risk of physician suicide. This large‐scale study captures responses from emergency medicine (EM) residents regarding two burnout items and examines the correlation between in‐training examination (ITE) scores and burnout risk as well as that between residency year and burnout risk.MethodsThis was a prospective, mixed‐methods, cross‐sectional cohort study. All residents in U.S. categorical EM residents who took the 2019 ITE were included. At the end of the ITE, residents were invited to complete a voluntary survey that included two items from the Maslach Burnout Inventory (MBI) that have been found to be strongly indicative of burnout: one about self‐perception of being burned out and one about feelings of callousness. Responses were on a 7‐level Likert scale (1–7), ranging from very low frequency (1) to very high frequency (7). Measurements included the number of residents in each year‐level of training (EM1–EM4), the MBI item ratings, and the ABEM ITE score. Performance, as measured by the scaled, equated score, was compared to the MBI item responses. A corrected Spearman’s correlation coefficient (ρ) was used to compare continuous data (score) against a discrete ordinal variable (MBI Likert response).ResultsThere were 2,501 EM1 residents, 2,389 EM2 residents, 2,206 EM3 residents, and 616 EM4 residents in the study group. There were 7,206 (93.4%) physicians who completed the first MBI question about burnout; 7,172 (93%) completed the second MBI question about callousness. There was no statistically significant association between the burnout item response and ITE performance (ρ = –0.03; p = 0.015). There was a positive, statistically significant association between the callousness item response and higher ITE performance (ρ = 0.07; p < 0.001). There was a statistically significant association between the response to the burnout item and training level (ρ = 0.07; p <0.001). There was also a statistically significant association between the response to the callousness item and training level (ρ = 0.15; p < 0.001). The overall prevalence of burnout risk in various training levels were EM1, 28.2%; EM2, 39%; EM3, 41.1%; and EM4, 43.3%.ConclusionsOur study found no significant correlation between ITE score and burnout risk. There was a weakly positive correlation between ITE scores and callousness. Based on our study results, ITE scores may not be useful in prognosticating burnout risk for EM residents and, interestingly, higher ITE scores correlated to stronger feelings of callousness. Our study indicates that EM residents at higher levels of training reported stronger self‐perceptions of burnout and callousness. Further investigation into why residents at higher levels of training may experience greater burnout risk is warranted.

AbstractObjectiveThis study was undertaken to expand on results from a 2014 study on the association between physician age and performance on the American Board of Emergency Medicine (ABEM) ConCert examination.MethodsThis was a retrospective, longitudinal growth study comparing performance on the ConCert examination and physicians’ ages at the time of examination. All examination attempts from 1990 to 2016 made by residency‐trained physicians were eligible for inclusion. Multilevel growth models were constructed to examine the relationship between age at time of examination and performance, controlling for physician characteristics.ResultsThe study group included 15,533 examination attempts by 12,786 physicians. The mean (±SD) age of the physicians across all examination administrations was 45.02 (±5.18) years (range = 35 to 72 years). The mean (±SD) ConCert examination score across all administrations was 85.39 (±5.71; range = 51 to 100). Among first‐time ConCert examination takers, older age was associated with lower examination scores (r = –0.25, p < 0.0001). Across all examination attempts, age was negatively correlated to examination scores (r = –0.24; p < 0.0001).ConclusionsAfter physician characteristics were controlled for, there was an association between advancing age and declining performance on the ABEM ConCert examination. This information may be important to the individual physician to develop targeted competency assessment and professional development.