Papageorge, Christina M.

Background. Assessment tools in surgical education help identify improvement areas for trainees. We developed and present validity evidence for a novel Transplant Operative Assessment Tool to optimize technical feedback for surgical trainees in transplantation. Methods. The Transplant Operative Assessment Tool assesses learners on anatomy, technical skills, judgment, and communication. Performance is rated across 5 competency levels. Content validity evidence included a construct map using existing literature and expert cognitive interviews. Ratings were used to develop an item response theory calibration, which was then used to evaluate the tool and provide validity evidence. Results. Eighty-two assessments for 28 trainees (16 residents, 12 fellows) were completed by 25 faculty surgeons across 4 programs. Fifty-six kidney transplants, 13 liver transplants, and 13 donor nephrectomies were assessed. The mean assessment completion time was 4.1 ± 4.3 min. Internal structure validity was evaluated by calculating item infit (range, 0.86–1.15), indicating a strong model fit and confirming a high quality of items. Internal consistency was also high (Cronbach’s alpha = 0.94). There was a positive correlation between ratings and training level ( P < 0.01), indicating strong discriminatory power. Warm ischemia time for transplant operations did not correlate with performance ( P = 0.16). Conclusions. These data provide validity evidence for a novel operative assessment tool. This instrument can measure performance, monitor progress, document competency, and has the potential to provide valuable feedback.

Objective: To characterize agreement between administrative and registry data in the determination of patient-level comorbidities. Background: Previous research finds poor agreement between these 2 types of data in the determination of outcomes. We hypothesized that concordance between administrative and registry data would also be poor. Methods: A cohort of inpatient operations (length of stay 1 day or greater) was obtained from a consortium of 8 hospitals. Within each hospital, National Surgical Quality Improvement Program (NSQIP) data were merged with intra-institutional inpatient administrative data. Twelve different comorbidities (diabetes, hypertension, congestive heart failure, hemodialysis-dependence, cancer diagnosis, chronic obstructive pulmonary disease, ascites, sepsis, smoking, steroid, congestive heart failure, acute renal failure, and dyspnea) were analyzed in terms of agreement between administrative and NSQIP data. Results: Forty-one thousand four hundred thirty-two inpatient surgical hospitalizations were analyzed in this study. Concordance (Cohen Kappa value) between the 2 data sources varied from 0.79 (diabetes) to 0.02 (dyspnea). Hospital variation in concordance (intersite variation) was quantified using a test of homogeneity. This test found significant intersite variation at a level of P < 0.001 for each of the comorbidities except for dialysis (P = 0.07) and acute renal failure (P = 0.19). These findings imply significant differences between hospitals in their generation of comorbidity data. Conclusion: This study finds significant differences in how administrative versus registry data assess patient-level comorbidity. These differences are of concern to patients, payers, and providers, each of which had a stake in the integrity of these data. Standardized definitions of comorbidity and periodic audits are necessary to ensure data accuracy and minimize bias.