Fahy, Brenda G.

Objectives: In the United States, training for physicians who manage critically ill adult patients (intensivists) evolved through parallel subspecialty critical care medicine (CCM) pathways with significant commonality. The Society of Critical Care Medicine Adult Critical Care Physician Core Knowledge and Skills Task Force aimed to delineate the common core knowledge and skills required of all intensivists. Design: A master list of content areas and procedural skills was compiled from all CCM subspecialty program requirements and blueprints of the certification examinations. Using a modified Delphi approach, participants were asked to categorize the knowledge items as “advanced knowledge is essential,” “general, but not advanced, knowledge is essential,” or “knowledge is not essential.” Procedures were categorized as “intensivist performs routinely,” “intensivist only performs in an emergency,” or “intensivist knows” about the procedure. Setting: Representatives from CCM stakeholder organizations, including accreditation and certification organizations, critical care societies, and program directors’ societies, were invited to participate. SUBJECTS: Members of the Adult Critical Care Physician Core Knowledge and Skills Task Force of the Society of Critical Care Medicine. Interventions: For the first two rounds of the modified Delphi process, Research Electronic Data Capture was used. For the third and fourth rounds, the process was completed through online meetings with Zoom (Zoom Video Corporations, San Jose, CA) utilizing Zoom’s polling feature. Measurements and Main Results: A total of 541 items were determined to be essential, with 145 requiring advanced knowledge and 323 requiring general knowledge. For 73 items, consensus regarding advanced vs. general could not be achieved, but they remained essential. Only eight items were felt to be nonessential. Of the 16 procedures, most were categorized as “intensivist performs.” Conclusions: The large number of items included in the list of essential knowledge and skills demonstrates the complexity of modern CCM. Utilization of a common framework across the subspecialties of CCM could lead to greater harmonization among the fellowship program requirements and certification examinations.

BACKGROUND: The American Board of Anesthesiology administers the APPLIED Examination as a part of initial certification, which as of 2018 includes 2 components—the Standardized Oral Examination (SOE) and the Objective Structured Clinical Examination (OSCE). The goal of this study is to investigate the measurement construct(s) of the APPLIED Examination to assess whether the SOE and the OSCE measure distinct constructs (ie, factors). METHODS: Exploratory item factor analysis of candidates’ performance ratings was used to determine the number of constructs, and confirmatory item factor analysis to estimate factor loadings within each construct and correlation(s) between the constructs. RESULTS: In exploratory item factor analysis, the log-likelihood ratio test and Akaike information criterion index favored the 3-factor model, with factors reflecting the SOE, OSCE Communication and Professionalism, and OSCE Technical Skills. The Bayesian information criterion index favored the 2-factor model, with factors reflecting the SOE and the OSCE. In confirmatory item factor analysis, both models suggest moderate correlation between the SOE factor and the OSCE factor; the correlation was 0.49 (95% confidence interval [CI], 0.42-0.55) for the 3-factor model and 0.61 (95% CI, 0.54-0.64) for the 2-factor model. The factor loadings were lower for Technical Skills stations of the OSCE (ranging from 0.11 to 0.25) compared with those of the SOE and Communication and Professionalism stations of the OSCE (ranging from 0.36 to 0.50). CONCLUSIONS: The analyses provide evidence that the SOE and the OSCE measure distinct constructs, supporting the rationale for administering both components of the APPLIED Examination for initial certification in anesthesiology.

In 2018, the American Board of Anesthesiology (ABA) became the first US medical specialty certifying board to incorporate an Objective Structured Clinical Examination (OSCE) into its initial certification examination system. Previously, the ABA’s staged examination system consisted of 2 written examinations (the BASIC and ADVANCED examinations) and the Standardized Oral Examination (SOE). The OSCE and the existing SOE are now 2 separate components of the APPLIED Examination. This report presents the results of the first-year OSCE administration. A total of 1410 candidates took both the OSCE and the SOE in 2018. Candidate performance approximated a normal distribution for both the OSCE and the SOE, and was not associated with the timing of the examination, including day of the week, morning versus afternoon session, and order of the OSCE and the SOE. Practice-based Learning and Improvement was the most difficult station, while Application of Ultrasonography was the least difficult. The correlation coefficient between SOE and OSCE scores was 0.35 ([95% confidence interval {CI}, 0.30–0.39]; P < .001). Scores for the written ADVANCED Examination were modestly correlated with scores for the SOE ( r = 0.29 [95% CI, 0.25–0.34]; P < .001) and the OSCE ( r = 0.15 [95% CI, 0.10–0.20]; P < .001). Most of the candidates who failed the SOE passed the OSCE, and most of the candidates who failed the OSCE passed the SOE. Of the 1410 candidates, 77 (5.5%) failed the OSCE, 155 (11.0%) failed the SOE, and 25 (1.8%) failed both. Thus, 207 (14.7%) failed at least 1 component of the APPLIED Examination. Adding an OSCE to a board certification examination system is feasible. Preliminary evidence indicates that the OSCE measures aspects of candidate abilities distinct from those measured by other examinations used for initial board certification.

With its first administration of an Objective Structured Clinical Examination (OSCE) in 2018, the American Board of Anesthesiology (ABA) became the first US medical specialty certifying board to incorporate this type of assessment into its high-stakes certification examination system. The fundamental rationale for the ABA’s introduction of the OSCE is to include an assessment that allows candidates for board certification to demonstrate what they actually “do” in domains relevant to clinical practice. Inherent in this rationale is that the OSCE will capture competencies not well assessed in the current written and oral examinations—competencies that will allow the ABA to judge whether a candidate meets the standards expected for board certification more properly. This special article describes the ABA’s journey from initial conceptualization through first administration of the OSCE, including the format of the OSCE, the process for scenario development, the standardized patient program that supports OSCE administration, examiner training, scoring, and future assessment of reliability, validity, and impact of the OSCE. This information will be beneficial to both those involved in the initial certification process, such as residency graduate candidates and program directors, and others contemplating the use of high-stakes summative OSCE assessments.

The American Board of Anesthesiology (ABA) has been administering an oral examination as part of its initial certification process since 1939. Among the 24 member boards of the American Board of Medical Specialties, 13 other boards also require passing an oral examination for physicians to become certified in their specialties. However, the methods used to develop, administer, and score these examinations have not been published. The purpose of this report is to describe the history and evolution of the anesthesiology Standardized Oral Examination, its current examination development and administration, the psychometric model and scoring, physician examiner training and auditing, and validity evidence. The many-facet Rasch model is the analytic method used to convert examiner ratings into scaled scores for candidates and takes into account how difficult grader examiners are and the difficulty of the examination tasks. Validity evidence of the oral examination includes that it measures aspects of clinical performance not accounted for by written certifying examinations, and that passing the oral examination is associated with a decreased risk of subsequent license actions against the anesthesiologist. Explaining the details of the Standardized Oral Examination provides transparency about this component of initial certification in anesthesiology.