Rush, Emily

AI skill development is essential in health professions education (HPE), yet this type of faculty development is complex to design and implement. While competency frameworks and AI education programs are emerging, many faculty developers are themselves still building AI literacy and confidence, and practical guidance for how to design a faculty development program on AI that is accessible, skill-oriented, and responsive to rapid technological change remains limited. Although webinars and workshops can only do so much, a well-planned faculty development program can provide a great introduction to AI, alleviate fears, and inspire participants to keep growing and learning. Drawing on our experience designing and hosting the AAMC GEA AI Skill-Building Webinar Series and the current literature in AI, this article offers twelve practical tips for faculty developers who are creating AI-focused learning opportunities for everyone involved in HPE. These tips will help readers create robust AI skill-building sessions that move participants beyond awareness of AI and shift their perspectives toward applied, reflective, and responsible use of AI in HPE.

ABSTRACT Objectives The increasing volume of medical education research necessitates efficient, reliable, and scalable methods for conducting quality appraisals. The Medical Education Research Study Quality Instrument (MERSQI) is a widely used tool, although its manual scoring process remains resource‐intensive. This study evaluated how well large language models (LLMs) appraise medical education research using the MERSQI tool in comparison with human judges. Methods Three LLMs (GPT‐5, Claude Sonnet 4, and Gemini 2.5 Pro) assigned MERSQI domain scores to 1423 medical education research articles. The authors compared AI‐generated scores with human‐generated scores using intraclass correlation coefficients (ICCs) across the six MERSQI domains. They evaluated the agreement between AI‐ and human‐generated MERSQI composite scores using Bland–Altman plots. Results Domain‐level ICC values ranged from fair (0.24) to near perfect (0.81), with the lowest agreement observed in the ‘sampling,’ ‘validity evidence,’ and ‘data analysis’ domains. No single LLM consistently outperformed the others across all domains. Composite score agreement with human ratings was substantial and similar across LLMs (ICC range: 0.65–0.69). GPT‐5 produced slightly lower composite scores than humans, while Claude Sonnet 4 and Gemini 2.5 Pro produced higher scores, with Gemini showing the largest deviation. The Bland–Altman plots for Gemini 2.5 Pro suggested proportional bias, indicating its agreement with human scores varied across the range of study quality. Conclusions These LLMs demonstrated substantial agreement with human raters for MERSQI composite scores, but domain‐level agreement varied. Systematic differences in scoring patterns highlight the need for human oversight and additional calibrations before integrating LLMs into systematic review appraisal workflows.