Citations

Abstract Purpose Consensus recommendations and guidelines for reporting quantitative research in health professions education (HPE) research are lacking. This study synthesizes available quantitative reporting recommendations and guidelines to present a harmonized framework and derive quantitative standards for reporting HPE research. Method The authors identified existing standards, recommendations, and guidelines from peer-reviewed scientific journals in medicine, education, and social sciences by searching PubMed, Web of Science, and Google from August 2006 through April 2025. The authors also requested input from experts in HPE (scholars and journal editors) via electronic communication over 6 months (November 2024 to April 2025). The articles identified were then used to synthesize and generate an initial list of standards for reporting quantitative research in HPE. Two authors reviewed each article to rate its alignment with HPE research. All authors independently evaluated the full list of standards and discussed them as a group for their relevance and alignment for reporting in HPE; all authors also made recommendations for each standard (to report as is, report but modify, or drop). The group subsequently reviewed and collectively reached consensus on the final set of quantitative reporting standards. Results The authors reviewed a total of 34 articles, with 19 identified as having alignment with HPE quantitative research standards. Mean interrater agreement among author pairs was 98% (kappa = .96). Authors initially generated 40 reporting standards, of which 18 (45%) were modified and 1 (3%) was identified to be dropped. The final set of quantitative guidelines consisted of 39 standards, with each standard nested in one of 28 topic areas. Conclusions The quantitative reporting standards identified in this study provide guidance to ensure rigor in reporting expectations for HPE research. These standards may also facilitate critical appraisal of articles and enhance the quality and impact of quantitative HPE research.

AbstractObjectivesThis needs assessment aimed to improve understanding of flexible endoscopic intubation training and practice in emergency medicine (EM), providing insights to educators and practice leaders seeking to improve education and practices.MethodsWe conducted a multicenter, mixed‐methods needs assessment of emergency physicians (EPs) incorporating focus groups and a survey. Focus groups comprised community EPs, academic EPs, and resident EPs. We analyzed focus group transcripts using grounded theory, qualitatively describing EM endoscopic intubation. The qualitative analysis shaped our survey instrument, which we deployed in cross‐sectional fashion. We report survey data with descriptive statistics.ResultsFocus groups with 13 EPs identified three themes: indications for use of endoscopic intubation, factors impacting a physician's decision to endoscopically intubate, and attaining and maintaining endoscopic intubation competency. Of 257 surveyed EPs (33% response rate), 79% had received endoscopic intubation training during residency, though 82% had performed this procedure 10 or fewer times in their career. Despite 97% acknowledging the necessity of competency, only 23% felt highly confident in their ability to perform endoscopic intubation. Participants (93%) reported scarce opportunities to perform the procedure and identified factors believed to facilitate competency acquisition and maintenance, including opportunities to perform endoscopic intubation in practice (98%), local champions (93%), and performing nasopharyngoscopy (87%).ConclusionsWhile most EPs acknowledged the importance of competency in endoscopic intubation, they reported scarce procedural opportunities and commonly expressed low confidence. Further research is needed on this topic, and we propose avenues to enhance education and practices related to endoscopic intubation. These include development of robust procedural curricula, support of local champions, and incorporating nasopharyngoscopy into EM practice.

Meaningful Equity, Diversity, and Inclusion (EDI) efforts may be stymied by concerns about whether proposed initiatives are performative or tokenistic. The purpose of this project was to analyze discussions by the Research in Medical Education (RIME) Program Planning committee about how best to recognize and support underrepresented in medicine (URiM) researchers in medical education to generate lessons learned that might inform local, national, and international actions to implement meaningful EDI initiatives. Ten RIME Program Planning Committee members and administrative staff participated in a focus group held virtually in August 2021. Focus group questions elicited opinions about “if and how” to establish a URiM research award. The focus group was recorded, transcribed, and thematically analyzed. Recognition of privilege, including who has it and who doesn’t, underpinned the focus group discussion, which revolved around 2 themes: (1) tensions between optics and semantics, and (2) potential unintended consequences of trying to level the medical education playing field. The overarching storyline threaded throughout the focus group discussion was intentionality. Focus group participants sought to avoid performativity by creating an award that would be meaningful to recipients and to career gatekeepers such as department chairs and promotion and tenure committees. Ultimately, participants decided to create an award that focused on exemplary Equity, Diversity, and Inclusion (EDI) scholarship, which was eventually named the “RIME URiM Research Award.” Difficult but productive conversations about EDI initiatives are necessary to advance underrepresented in medicine (URiM) scholarship. This transparent commentary may trigger further critical conversations.

AbstractIntroduction/AimsGraduate medical education programs must ensure residents and fellows acquire skills needed for independent practice. Workplace‐based observational assessments are informative but can be time‐ and resource‐intensive. In this study we sought to gather “relations‐to‐other‐variables” validity evidence for scores generated by the Electromyography Direct Observation Tool (EMG‐DOT) to inform its use as a measure of electrodiagnostic skill acquisition.MethodsScores on multiple assessments were compiled by trainees during Clinical Neurophysiology and Electromyography rotations at a large US academic medical center. Relationships between workplace‐based EMG‐DOT scores (n = 298) and scores on a prerequisite simulated patient exercise, patient experience surveys (n = 199), end‐of‐rotation evaluations (n = 301), and an American Association of Neuromuscular & Electrodiagnostic Medicine (AANEM) self‐assessment examination were assessed using Pearson correlations.ResultsAmong 23 trainees, EMG‐DOT scores assigned by physician raters correlated positively with end‐of‐rotation evaluations (r = 0.63, P = .001), but EMG‐DOT scores assigned by technician raters did not (r = 0.10, P = .663). When physician and technician ratings were combined, higher EMG‐DOT scores correlated with better patient experience survey scores (r = 0.42, P = .047), but not with simulated patient or AANEM self‐assessment examination scores.DiscussionEnd‐of‐rotation evaluations can provide valid assessments of trainee performance when completed by individuals with ample opportunities to directly observe trainees. Inclusion of observational assessments by technicians and patients provides a more comprehensive view of trainee performance. Workplace‐ and classroom‐based assessments provide complementary information about trainee performance, reflecting underlying differences in types of skills measured.

Purpose To examine validity evidence for a standardized patient scenario assessing medical students’ ability to promote value using patient-centered communication (in response to a patient requesting an unnecessary test) and to explore the potential effect of various implementation and curricular factors on student scores. Method Third-year medical students (N = 516) from 5 U.S. MD-granting medical schools completed the communication scenario between 2014 and 2017 as part of a larger objective structured clinical examination (OSCE). Centralized raters assessed performance using an 11-item checklist. The authors collected multiple sources of validity evidence. Results The mean checklist score was 0.85 (standard deviation 0.09). Interrater reliability for checklist scores was excellent (0.87, 95% confidence interval = 0.78–0.93). Generalizability and Phi-coefficients were, respectively, 0.65 and 0.57. Scores decreased as the number of OSCE stations increased ( r = −0.15, P = .001) and increased when they were used for summative purposes ( r = 0.26, P < .001). Scores were not associated with curricular time devoted to high-value care ( r = 0.02, P = .67) and decreased when more clerkships were completed before the assessment ( r = −0.12, P = .006). Conclusions This multisite study provides validity evidence supporting the use of scenario scores to assess the ability of medical students to promote value in clinical encounters using patient-centered communication. Findings illuminate the potential effect of OSCE structure and purpose on student performance and suggest clerkship learning experiences may not reinforce what students are taught in the formal curriculum regarding high-value care. Devoting more time to the topic appears insufficient to counteract this erosion.

Purpose To examine how qualitative narrative comments and quantitative ratings from end-of-rotation assessments change for a cohort of residents from entry to graduation, and explore associations between comments and ratings. Method The authors obtained end-of-rotation quantitative ratings and narrative comments for 1 cohort of internal medicine residents at the University of Illinois at Chicago College of Medicine from July 2013–June 2016. They inductively identified themes in comments, coded orientation (praising/critical) and relevance (specificity and actionability) of feedback, examined associations between codes and ratings, and evaluated changes in themes and ratings across years. Results Data comprised 1,869 assessments (828 comments) on 33 residents. Five themes aligned with ACGME competencies (interpersonal and communication skills, professionalism, medical knowledge, patient care, and systems-based practice), and 3 did not (personal attributes, summative judgment, and comparison to training level). Work ethic was the most frequent subtheme. Comments emphasized medical knowledge more in year 1 and focused more on autonomy, leadership, and teaching in later years. Most comments (714/828 [86%]) contained high praise, and 412/828 (50%) were very relevant. Average ratings correlated positively with orientation (β = 0.46, P < .001) and negatively with relevance (β = −0.09, P = .01). Ratings increased significantly with each training year (year 1, mean [standard deviation]: 5.31 [0.59]; year 2: 5.58 [0.47]; year 3: 5.86 [0.43]; P < .001). Conclusions Narrative comments address resident attributes beyond the ACGME competencies and change as residents progress. Lower quantitative ratings are associated with more specific and actionable feedback.

Objectives To develop neurology scenarios for use with the Quality Improvement Knowledge Application Tool Revised (QIKAT-R), gather and evaluate validity evidence, and project the impact of scenario number, rater number and rater type on score reliability. Methods Six neurological case scenarios were developed. Residents were randomly assigned three scenarios before and after a quality improvement (QI) course in 2015 and 2016. For each scenario, residents crafted an aim statement, selected a measure and proposed a change to address a quality gap. Responses were scored by six faculty raters (two with and four without QI expertise) using the QIKAT-R. Validity evidence from content, response process, internal structure, relations to other variables and consequences was collected. A generalisability (G) study examined sources of score variability, and decision analyses estimated projected reliability for different numbers of raters and scenarios and raters with and without QI expertise. Results Raters scored 163 responses from 28 residents. The mean QIKAT-R score was 5.69 (SD 1.06). G-coefficient and Phi-coefficient were 0.65 and 0.60, respectively. Interrater reliability was fair for raters without QI expertise (intraclass correlation = 0.53, 95% CI 0.30 to 0.72) and acceptable for raters with QI expertise (intraclass correlation = 0.66, 95% CI 0.02 to 0.88). Postcourse scores were significantly higher than precourse scores (6.05, SD 1.48 vs 5.22, SD 1.5; p < 0.001). Sufficient reliability for formative assessment (G-coefficient > 0.60) could be achieved by three raters scoring six scenarios or two raters scoring eight scenarios, regardless of rater QI expertise. Conclusions Validity evidence was sufficient to support the use of the QIKAT-R with multiple scenarios and raters to assess resident QI knowledge application for formative or low-stakes summative purposes. The results provide practical information for educators to guide implementation decisions.