Park, Yoon Soo

Abstract Purpose Consensus recommendations and guidelines for reporting quantitative research in health professions education (HPE) research are lacking. This study synthesizes available quantitative reporting recommendations and guidelines to present a harmonized framework and derive quantitative standards for reporting HPE research. Method The authors identified existing standards, recommendations, and guidelines from peer-reviewed scientific journals in medicine, education, and social sciences by searching PubMed, Web of Science, and Google from August 2006 through April 2025. The authors also requested input from experts in HPE (scholars and journal editors) via electronic communication over 6 months (November 2024 to April 2025). The articles identified were then used to synthesize and generate an initial list of standards for reporting quantitative research in HPE. Two authors reviewed each article to rate its alignment with HPE research. All authors independently evaluated the full list of standards and discussed them as a group for their relevance and alignment for reporting in HPE; all authors also made recommendations for each standard (to report as is, report but modify, or drop). The group subsequently reviewed and collectively reached consensus on the final set of quantitative reporting standards. Results The authors reviewed a total of 34 articles, with 19 identified as having alignment with HPE quantitative research standards. Mean interrater agreement among author pairs was 98% (kappa = .96). Authors initially generated 40 reporting standards, of which 18 (45%) were modified and 1 (3%) was identified to be dropped. The final set of quantitative guidelines consisted of 39 standards, with each standard nested in one of 28 topic areas. Conclusions The quantitative reporting standards identified in this study provide guidance to ensure rigor in reporting expectations for HPE research. These standards may also facilitate critical appraisal of articles and enhance the quality and impact of quantitative HPE research.

Purpose To examine validity evidence for a standardized patient scenario assessing medical students’ ability to promote value using patient-centered communication (in response to a patient requesting an unnecessary test) and to explore the potential effect of various implementation and curricular factors on student scores. Method Third-year medical students (N = 516) from 5 U.S. MD-granting medical schools completed the communication scenario between 2014 and 2017 as part of a larger objective structured clinical examination (OSCE). Centralized raters assessed performance using an 11-item checklist. The authors collected multiple sources of validity evidence. Results The mean checklist score was 0.85 (standard deviation 0.09). Interrater reliability for checklist scores was excellent (0.87, 95% confidence interval = 0.78–0.93). Generalizability and Phi-coefficients were, respectively, 0.65 and 0.57. Scores decreased as the number of OSCE stations increased ( r = −0.15, P = .001) and increased when they were used for summative purposes ( r = 0.26, P < .001). Scores were not associated with curricular time devoted to high-value care ( r = 0.02, P = .67) and decreased when more clerkships were completed before the assessment ( r = −0.12, P = .006). Conclusions This multisite study provides validity evidence supporting the use of scenario scores to assess the ability of medical students to promote value in clinical encounters using patient-centered communication. Findings illuminate the potential effect of OSCE structure and purpose on student performance and suggest clerkship learning experiences may not reinforce what students are taught in the formal curriculum regarding high-value care. Devoting more time to the topic appears insufficient to counteract this erosion.

Purpose To examine how qualitative narrative comments and quantitative ratings from end-of-rotation assessments change for a cohort of residents from entry to graduation, and explore associations between comments and ratings. Method The authors obtained end-of-rotation quantitative ratings and narrative comments for 1 cohort of internal medicine residents at the University of Illinois at Chicago College of Medicine from July 2013–June 2016. They inductively identified themes in comments, coded orientation (praising/critical) and relevance (specificity and actionability) of feedback, examined associations between codes and ratings, and evaluated changes in themes and ratings across years. Results Data comprised 1,869 assessments (828 comments) on 33 residents. Five themes aligned with ACGME competencies (interpersonal and communication skills, professionalism, medical knowledge, patient care, and systems-based practice), and 3 did not (personal attributes, summative judgment, and comparison to training level). Work ethic was the most frequent subtheme. Comments emphasized medical knowledge more in year 1 and focused more on autonomy, leadership, and teaching in later years. Most comments (714/828 [86%]) contained high praise, and 412/828 (50%) were very relevant. Average ratings correlated positively with orientation (β = 0.46, P < .001) and negatively with relevance (β = −0.09, P = .01). Ratings increased significantly with each training year (year 1, mean [standard deviation]: 5.31 [0.59]; year 2: 5.58 [0.47]; year 3: 5.86 [0.43]; P < .001). Conclusions Narrative comments address resident attributes beyond the ACGME competencies and change as residents progress. Lower quantitative ratings are associated with more specific and actionable feedback.

Objectives To develop neurology scenarios for use with the Quality Improvement Knowledge Application Tool Revised (QIKAT-R), gather and evaluate validity evidence, and project the impact of scenario number, rater number and rater type on score reliability. Methods Six neurological case scenarios were developed. Residents were randomly assigned three scenarios before and after a quality improvement (QI) course in 2015 and 2016. For each scenario, residents crafted an aim statement, selected a measure and proposed a change to address a quality gap. Responses were scored by six faculty raters (two with and four without QI expertise) using the QIKAT-R. Validity evidence from content, response process, internal structure, relations to other variables and consequences was collected. A generalisability (G) study examined sources of score variability, and decision analyses estimated projected reliability for different numbers of raters and scenarios and raters with and without QI expertise. Results Raters scored 163 responses from 28 residents. The mean QIKAT-R score was 5.69 (SD 1.06). G-coefficient and Phi-coefficient were 0.65 and 0.60, respectively. Interrater reliability was fair for raters without QI expertise (intraclass correlation = 0.53, 95% CI 0.30 to 0.72) and acceptable for raters with QI expertise (intraclass correlation = 0.66, 95% CI 0.02 to 0.88). Postcourse scores were significantly higher than precourse scores (6.05, SD 1.48 vs 5.22, SD 1.5; p < 0.001). Sufficient reliability for formative assessment (G-coefficient > 0.60) could be achieved by three raters scoring six scenarios or two raters scoring eight scenarios, regardless of rater QI expertise. Conclusions Validity evidence was sufficient to support the use of the QIKAT-R with multiple scenarios and raters to assess resident QI knowledge application for formative or low-stakes summative purposes. The results provide practical information for educators to guide implementation decisions.