Brian, Riley

Background. Assessment tools in surgical education help identify improvement areas for trainees. We developed and present validity evidence for a novel Transplant Operative Assessment Tool to optimize technical feedback for surgical trainees in transplantation. Methods. The Transplant Operative Assessment Tool assesses learners on anatomy, technical skills, judgment, and communication. Performance is rated across 5 competency levels. Content validity evidence included a construct map using existing literature and expert cognitive interviews. Ratings were used to develop an item response theory calibration, which was then used to evaluate the tool and provide validity evidence. Results. Eighty-two assessments for 28 trainees (16 residents, 12 fellows) were completed by 25 faculty surgeons across 4 programs. Fifty-six kidney transplants, 13 liver transplants, and 13 donor nephrectomies were assessed. The mean assessment completion time was 4.1 ± 4.3 min. Internal structure validity was evaluated by calculating item infit (range, 0.86–1.15), indicating a strong model fit and confirming a high quality of items. Internal consistency was also high (Cronbach’s alpha = 0.94). There was a positive correlation between ratings and training level ( P < 0.01), indicating strong discriminatory power. Warm ischemia time for transplant operations did not correlate with performance ( P = 0.16). Conclusions. These data provide validity evidence for a novel operative assessment tool. This instrument can measure performance, monitor progress, document competency, and has the potential to provide valuable feedback.