Citations

Background: Large language models (LLMs) are increasingly used in patient-facing clinical applications, creating a need for scalable methods to evaluate the accuracy, safety, and appropriateness of their responses. Although physician review remains the reference standard, it is resource-intensive and difficult to scale. Objective: To develop and clinically validate an LLM-as-a-judge framework for evaluating responses to patients’ periprocedural questions. Methods: We retrospectively evaluated 1285 patient–AIVA interactions from 112 patients across two Mayo Clinic sites and multiple surgical specialties. Physician reviewers classified interactions using a predefined true-positive (TP), false-negative (FN), true-negative (TN), and false-positive (FP) framework. The finalized LLM judge independently evaluated the same interactions. Agreement was assessed using four-class and category-specific agreement, Cohen’s kappa, and a secondary binary analysis of response correctness. Results: Overall, four-class agreement was 93.1% (95% CI, 91.7–94.5%), with Cohen’s κ = 0.852 (95% CI, 0.823–0.882). Category-specific agreement was 94.1% for TP, 90.7% for FN, 95.5% for TN, and 84.9% for FP classifications. In the binary correctness analysis, accuracy was 93.4% (95% CI, 92.0–94.7%), sensitivity 94.6%, specificity 90.2%, precision 96.2%, and F1-score 0.954. McNemar’s test demonstrated no significant asymmetry between paired classifications (p = 0.11). Conclusions: A clinically grounded LLM judge closely approximated physician evaluation of patient-facing AI responses. These findings support its potential as a scalable assistive monitoring tool while preserving physician oversight for uncertain or safety-sensitive interactions.

Background Effective leadership is critical to the success of complex multidisciplinary healthcare organisations. As the organisation emerged from the global pandemic, Mayo Clinic recognised a need to transform its traditional, linear care model into a dynamic, platform-based approach to support its strategy. This required a new leadership model that could support and equip leaders with the skills needed to drive patient-centred healthcare transformation from within. Objective To develop and validate a new leadership competency framework capable of supporting future-ready leadership. Methods A multiphase, evidence-based approach was employed: (1) a visionary interview, goal setting and literature review; (2) focus groups with 23 leaders from across the organisation; (3) iterative competency framework development; and (4) a validation study involving a stratified sample of 129 physician leaders who rated competencies on importance and differentiation, and a qualitative verification by key stakeholders. Results Five core competencies with 23 behaviours were identified. The validation analysis established all behaviours as important for differentiating leadership performance. Stakeholder review confirmed the competencies were actionable and applicable across leadership roles at all levels. Conclusion The Leads Self, Leads Others, Inspires Others, Forward Thinking and Engages Others (abbreviated LLIFE) competency model positions a healthcare organisation to advance leadership excellence in a rapidly evolving healthcare environment that requires comprehensive transformation. It aligns with strategic goals, embeds values-based behaviours and supports continuous development, establishing a model that other healthcare institutions may learn from and adapt.

Ultrasound has been used more frequently in the anatomical sciences to reinforce core anatomy concepts in the classroom and introduce commonly applied imaging technologies encountered in a clinical setting. Interprofessional Education (IPE) is being incorporated into health sciences training as a learning model for real world interactions. The primary objective of the current study was to determine the impact of an IPE ultrasound curriculum on students' anatomy and ultrasound knowledge, as well as attitudes towards interprofessional collaboration. We hypothesized that students who participated in an IPE experience would have improved recall of anatomical and ultrasound concepts as well as improved attitudes and perceptions towards IPE when compared to students who did not participate in an IPE session. First year medical students (MD, n = 40) and first year physical therapy students (PT, n = 25) were randomly assigned to PT‐only, MD‐only, or mixed PT‐MD ultrasound instruction groups studying the shoulder. Students were assessed using an anatomy and ultrasound knowledge test and University of the West of England Interprofessional Questionnaire (UWEIPQ) prior to and following the session. The results of this study suggest that while ultrasound instruction significantly improves ultrasound post‐test scores when compared to pre‐test scores regardless of group assignment (PT‐only: p < 0.001; MD‐only: p = 0.007; PT‐MD: p < 0.001), learning in an interprofessional PT‐MD group did not significantly impact knowledge recall of ultrasound concepts when compared to non‐IPE groups. Interestingly, students in PT‐MD groups had significantly higher anatomy post‐test scores when compared to the MD‐only groups (PT‐MD post‐test = 93.2%; MD‐only post‐test = 84.1%; p = 0.041), but not when compared to the PT‐only groups. Additionally, the results of the UWEIPQ are consistent with the impression that students who participate in an IPE experience are more likely to be cognizant of inherent biases in the interactions between different health professionals as well as demonstrate increased confidence in their interprofessional relationships. In conclusion, our study reinforces that IPE programs should quantify knowledge recall in order to assess the effectiveness of a specific program. Furthermore, such endeavors should assess important competencies including communication, teamwork, and interdisciplinary respect that are necessary to succeed in a multidisciplinary health care team. Support or Funding Information Department of Anatomy, Mayo Clinic College of Medicine and Science, Mayo Clinic, Rochester, MN; Social Science and Humanities Research Council, Government of Canada

Use of ultrasound as a diagnostic tool is fast becoming an expectation in clinical practice. With formal outcomes now being included as competencies in many residency programs, earlier exposure to use of ultrasound is being achieved through informal activities embedded within medical school curricula. Using ultrasound to learn anatomy is now standard practice for many courses in the United States. However, very few if any offer formal opportunities to assess student skill and anatomical knowledge at the same time. As part of the Mini Ultrasound Skills Institute (MUSI) in Anatomy, a component of the first year anatomy course, students were expected to complete a team‐based, hands‐on ultrasound skills assessment. Throughout the course, students received both didactic briefing sessions and hands‐on training on ultrasound basics, image interpretation, and scanning techniques. The ultrasound content was presented in parallel to anatomy and radiology curriculum. In the past, ultrasound knowledge was assessed via static images; however, due to the contextual nature of ultrasound interpretation, a live, hands‐on method was devised to assess student learning. At regular intervals during the course, students completed an ultrasound skills assessment. In their dissection teams of four, students completed four tasks: setup the machine, obtained, oriented, and labeled a desired image using a live model. Each student was given a unique role within this exercise. Students were graded on a rubric scale for each of the four components and given an overall team score. At the conclusion of the course, students completed a graded ultrasound objective structural practical examination (OSPE). Rotating through four stations, students completed the four tasks on four different structures. Students rotated through roles within their team as they moved between stations. OSPE grade was incorporated into their final laboratory practical grade. Preliminary data from formative OSPE assessments ‐‐‐‐showed an average score of 72% (±21%) with a minimum of 50%, maximum of 100%. Within the skill specific breakdown, students were very comfortable at labeling obtained images (100%) and had the most challenging time with either selecting the correct probe or setting up the scanner 40% (±51%). Overall, team‐based OSPE is an effective method to promote and evaluate ultrasound skills learned in an anatomy course.

This commentary offers the reader an alternative to mentoring through the use of PODCASTS. By providing the listener with an understanding of the challenges and opportunities for self-reflection and sharing of experiences by the interviewees, we are impacting the listener attitudes and future goals through lessons learned.

Recently many medical schools have introduced ultrasound education, with several institutions adopting formal teaching of ultrasound into their anatomy curricula. Additionally, medical schools nationwide have begun to incorporate various forms of interprofessional education (IPE) into their programs to prepare medical students for the teamwork necessary to serve as physicians. The Anatomy course at Mayo Medical School incorporated several IPE sessions with physical therapy students. Since the course is integrated with radiology, ultrasound imaging was recently added to complement static images and visualize living anatomy. Due to the success of prior IPE endeavors in anatomy, a new elective experience for hands‐on ultrasound practice sessions was offered to both medical and physical therapy students. The aim of this study was to assess student perception of elective ultrasound practice sessions as an opportunity for interprofessional peer‐learning. Practice sessions were scheduled once a week for 4 hours and a trained teaching assistant was available during the sessions. Students were encouraged to discuss different techniques and anatomical knowledge based on their respective experiences. A sign‐in sheet was kept to monitor how many students from both groups attended. Students were asked to fill out a survey at the end of 4 weeks to assess perceptions of the interprofessional experience. Preliminary results indicate that students appreciated the diversity of an interprofessional approach to learning ultrasound skills and enjoyed teaching each other anatomy through ultrasound.

Ultrasound education continues to be integrated into gross anatomy courses in the United States. The team based design of the anatomy course at Mayo Clinic School of Medicine incorporates team‐based ultrasound oriented activities that are formalized through defined course objectives and assessed in the final examination. Providing personal handheld ultrasound devices to first‐year medical students during the gross anatomy course has been the mainstay of ultrasound education for the past three years. Handheld ultrasound devices have been provided to students for limited periods in some medical schools within the United States, but unique to this curriculum, students are assigned devices that can be utilized outside formal class time. This provides unlimited opportunity to apply their understanding of three‐dimensional, living anatomy, while simultaneously promoting student comfort and engagement with advancing ultrasound technology. Student perception related to the use of ultrasound outside of formal curricular time was evaluated on a five point Likert scale. Both quantitative and qualitative analyses were used to interpret the data obtained from this study. A total of three surveys were sent to medical students over a period of twelve months with an average response rate of 58.9%. Students indicated that (1) ultrasound devices at personal disposal during the anatomy course improved understanding of anatomical structures imaged ( mean 3.9, SD=+/− 1.01 ), (2) the handheld ultrasound device was easy to use ( mean 3.9, SD = +/− 0.8), and (3) it would be beneficial if ultrasound was implemented as part of the longitudinal learning experience in medical school ( mean 4.1, SD = +/ − 0.7 ). Students were not convinced that (1) having a personal ultrasound device influenced future choice of medical specialty ( mean 2.8, SD = +/− 1.0 ) and (2) the opportunity augmented coursework in the past three months of medical school at time of taking the survey ( mean 2.7, SD =+/ − 1.25 ). Results of the study indicated that access to personal handheld devices motivated students to use ultrasound devices and that they did so with increasing frequency outside of class time. In addition, students were able to apply and reinforce anatomical concepts by using landmarks to first locate and orientate the anatomic field and then interpret images of underlying structures. The fact that students did not see the value of ultrasound in determining future career choices or relate their learning of ultrasound to subsequent courses should not deter prospective opportunity for use of ultrasound in clinical practice. Basic ultrasound proficiency and an ability to effectively interpret scans are skills that in the long term will significantly decrease the learning curve in the use of technology as a diagnostic tool in future clinical settings. This abstract is from the Experimental Biology 2018 Meeting. There is no full text article associated with this abstract published in The FASEB Journal .

Despite increase in residency programs including ultrasound training, few medical schools have incorporated it into their curricula. The Gross Anatomy course at Mayo Medical School has introduced ultrasound in the curriculum. Cadaver dissection teaches students static anatomical relationships, but ultrasound offers dynamic display of how those relationships can change with movement. Ultrasound curriculum consists of four 1 hour didactic sessions and five 30 minute hands‐on modules, covering Carpal tunnel, Heart, Abdominal viscera, and Doppler imaging of blood flow. Each module is guided by a checklist of techniques and structures. Students are graded using ARS system, and ultrasound objectives are incorporated into the final exam. This study aimed to assess effectiveness of ultrasound curriculum in a 7 week anatomy course. Students were asked to complete pre‐ and post‐ test surveys that assessed whether ultrasound sessions allowed them to better appreciate living anatomy, learn the basics of operating a portable ultrasound machine, and become more comfortable with medical technology. Pre and post surveys showed that ultrasound helped students appreciate living anatomy, and that they were comfortable with technology. There was an increase in students’ perception of their ability to interpret ultrasound images with a p‐value of 0.000026. Ultrasound was a successful addition to the anatomy curriculum.