Mitchell, John

Background: The Accreditation Council for Graduate Medical Education (ACGME) case log system establishes minimum clinical experiences for anesthesiology residency. While core program requirements were last updated in 2016, case log minimums themselves have not undergone a major revision in approximately two decades. Consequently, current requirements may not fully reflect changes in anesthetic practice and the evolving scope of perioperative medicine. In anticipation of major ACGME revisions in 2027, this study aimed to develop consensus-driven recommendations for modernizing residency case logs. Methods: A three-round modified Delphi study was conducted with a nationally representative panel of 60 stakeholders, including anesthesiology residency program directors and academic and private practice anesthesiologists. Panelists evaluated a comprehensive list of potential case categories across three domains: technical skills, surgical case characteristics, and patient demographics. In Round 1, panelists proposed minimum numbers or suggested that no requirement was necessary. Rounds 2 and 3 utilized iterative feedback to determine consensus on the necessity of requirements and specific minimum thresholds. Consensus was defined as at least two-thirds agreement. Results: Sixty invited experts completed all three rounds (100% response rate). The panel reached consensus to recommend new minimum requirements across several areas not currently tracked with minimums, including arterial lines (40), central lines (20), fiberoptic intubation (10), one-lung ventilation (10), neuromonitoring (10), POCUS (10 Cardiac, 10 Lung), and NORA (20 cases). Regarding existing requirements, consensus was reached to recommend increasing total peripheral nerve blocks from 40 to 60, with added requirements for anatomic specificity, and cardiac case minimums from 20 to 25. The panel also established consensus definitions for life-threatening pathologies to standardize reporting. The panel did not achieve consensus on several areas, including, for example, whether a minimum requirement for patients younger than 3 months should be maintained. Conclusion: These findings provide stakeholder-validated recommendations to inform upcoming ACGME requirement revisions.

BACKGROUND: In 2018, a set of entrustable professional activities (EPAs) and procedural skills assessments were developed for anesthesiology training, but they did not assess all the Accreditation Council for Graduate Medical Education (ACGME) milestones. The aims of this study were to (1) remap the 2018 EPA and procedural skills assessments to the revised ACGME Anesthesiology Milestones 2.0, (2) develop new assessments that combined with the original assessments to create a system of assessment that addresses all level 1 to 4 milestones, and (3) provide evidence for the validity of the assessments. METHODS: Using a modified Delphi process, a panel of anesthesiology education experts remapped the original assessments developed in 2018 to the Anesthesiology Milestones 2.0 and developed new assessments to create a system that assessed all level 1 through 4 milestones. Following a 24-month pilot at 7 institutions, the number of EPA and procedural skill assessments and mean scores were computed at the end of the academic year. Milestone achievement and subcompetency data for assessments from a single institution were compared to scores assigned by the institution’s clinical competency committee (CCC). RESULTS: New assessment development, 2 months of testing and feedback, and revisions resulted in 5 new EPAs, 11 nontechnical skills assessments (NTSAs), and 6 objective structured clinical examinations (OSCEs). Combined with the original 20 EPAs and procedural skills assessments, the new system of assessment addresses 99% of level 1 to 4 Anesthesiology Milestones 2.0. During the 24-month pilot, aggregate mean EPA and procedural skill scores significantly increased with year in training. System subcompetency scores correlated significantly with 15 of 23 (65.2%) corresponding CCC scores at a single institution, but 8 correlations (36.4%) were <30.0, illustrating poor correlation. CONCLUSIONS: A panel of experts developed a set of EPAs, procedural skill assessment, NTSAs, and OSCEs to form a programmatic system of assessment for anesthesiology residency training in the United States. The method used to develop and pilot test the assessments, the progression of assessment scores with time in training, and the correlation of assessment scores with CCC scoring of milestone achievement provide evidence for the validity of the assessments.

The evolution of medical education, from a time-based to a competency-based platform, began nearly 30 years ago and continues to slowly take shape. The development of valid and reproducible assessment tools is the first step. Medical educators across specialties acknowledge the challenges and remain motivated to develop a relevant, generalizable, and measurable system. The Accreditation Council for Graduate Medical Education (ACGME) remains committed to its responsibility to the public by assuring that the process and outcome of graduate medical education in the nation’s residency programs produce competent, safe, and compassionate doctors. The Milestones Project is the ACGME’s current strategy in the evolution to a competency-based system, which allows each specialty to develop its own set of subcompetencies and 5-level progression, or milestones, along a continuum of novice to expert. The education community has now had nearly 5 years of experience with these rubrics. While not perfect, Milestones 1.0 provided important foundational information and insights. The first iteration of the Anesthesiology Milestones highlighted some mismatch between subcompetencies and current and future clinical practices. They have also highlighted challenges with assessment and evaluation of learners, and the need for faculty development tools. Committed to an iterative process, the ACGME assembled representatives from stakeholder groups within the Anesthesiology community to develop the second generation of Milestones. This special article describes the foundational data from Milestones 1.0 that was useful in the development process of Milestones 2.0, the rationale behind the important changes, and the additional tools made available with this iteration.