Citations

Abstract Introduction Evaluate large language models (LLMs) for scoring medical student essays, and compare various prompting techniques and models. Methods OpenAI GPT scored 51 medical student reflection essays (15 real, 36 fabricated) using a previously-reported 6-point rubric (April-May 2025). We compared 29 prompt-model conditions by systematically varying the LLM prompts (including the persona, scoring rubric, few-shot learning [exemplars], chain-of-thought reasoning, and temperature), fine-tuning, and model (including GPT-4.1, GPT-4.1-mini, GPT-o4-mini, and GPT-4-Turbo). Outcomes were accuracy (compared with human raters, measured using single-score intraclass correlation coefficient [ICC] and mean absolute difference [MAD; zero indicates perfect agreement]), within-condition reproducibility, and cost. Results Across all conditions, it took mean (SD) 3.73 (3.12) seconds to score 1 essay. The cost to score 100 essays was USD $0.04 for GPT-4.1-mini, $0.21 for GPT-4.1, $0.57 for GPT-4.1 with 3 exemplars, and $2.00 for fine-tuned GPT-4.1. When the one-time cost of fine-tuning was amortized across 10,000 essays, the cost for fine-tuned GPT-4.1 was $0.20 per 100. Accuracy was “almost perfect” (ICC >0.80) for 28/29 conditions (97%). Fine-tuned models were more accurate than non-fine-tuned models (MAD difference –0.24 [95% CI, –0.34, –0.14]). Conditions with exemplars were more accurate than those without (MAD difference –0.44 [CI, –0.57, –0.31]). Accuracy progressively decreased as 6, 3, 1, and 0 rubric levels were explicitly defined in the prompt (P<.001). Contrary to hypotheses, accuracies for chain-of-thought prompts and variations in temperature and persona were not significantly different from the baseline prompt. Reproducibility ICC was >0.80 for 28/29 conditions (97%). Discussion Automated LLM essay scoring demonstrated near-perfect accuracy and reproducibility for most prompt-model conditions. Fine-tuned models and prompts with exemplars had higher accuracy but higher cost. Fine-tuned models had lower per-essay costs for larger essay volumes. For smaller volumes, non-fine-tuned GPT-4.1 provided excellent results at moderate cost. GPT-4.1-mini provided very good results at low cost.

Abstract Introduction: Tribal Nations across the Southwest have striking disparities in cancer incidence and mortality. Compounded by disproportionately low rates of cancer screening and limited access to healthcare, Native Americans are more often diagnosed at later stages of disease and have the poorest outcomes in all types of cancer when compared to any other racial and ethnic group in the United States. At the University of New Mexico Comprehensive Cancer Center, the Participant Engagement Cancer Genomic Sequencing (PE-CGS) Center is developing culturally appropriate materials and strategies to enhance the engagement and recruitment of Native Americans into genomic research. This effort aims to determine participant knowledges, attitudes, cultural beliefs, and communication preferences regarding clinical genomics testing. Ultimately, targeting improvement of cancer outcomes in Southwestern Tribal communities. Cultural adaptations are crucial with regards to biospecimen collection, storage, and usage, due to historical mistrust of research among Native communities. (i.e., Havasupai Tribe v. the Arizona Board of Regents). A Tribal Advisory Committee, consisting of Tribal leaders, Native physicians, and patient advocates oversee all aspects of the research study. The goal is to identify appropriate optimized health literacy strategies for engaging Native peoples in cancer genomic research. Methods: The team conducted semi-structured discussions with participants to understand their perspectives on motivations to join, cultural safety components, and effectiveness of engagement strategies. With participant consent, the discussions were recorded and transcribed. Analysis included the transcripts and notes taken by the interviewer and followed Grounded Theory principles. All transcripts and notes followed line-by-line analysis by three of the authors (JS, JR and TG) who met regularly to identify and discuss emergent themes. Results: As of May 31, 2024, the PE-CGS study has enrolled 112 participants and completed 39 discussions. Participants emphasized altruistic motives tied to family and community, echoing traditional values, in their decision to participate in the study. This notion emerged during interactions with Native team members, where comfort stemmed from shared experiences (healthcare system, barriers), use of Native language, and adherence to Native social norms. Additional considerations included present autonomy throughout various stages of the study (sample use, data sharing, discussions). Return of genetic testing results improved understanding of treatment options, hereditary risk, and health literacy. Conclusion: Feedback highlights the importance of aligning cultural beliefs with Western medicine through a strengths-based approach. The findings will inform improvements in respectful engagement, communication, information exchange, and recruitment processes with Native American cancer patients and cancer survivors to genomic testing studies. Citation Format: Jessica Sanchez, Koby Blackwater, Joseph Rodman, Tamar Ginossar, Andrew Sussman, Shiraz Mishra, Cindy Blair, Mallery Quetawki, Debra MacKenzie, V. Shane Pankratz, Ursa Brown-Glaberman, Jeffery Trent, Cheryl Willman. Journey to understanding indigenous peoples engagement in cancer genomic research [abstract]. In: Proceedings of the 17th AACR Conference on the Science of Cancer Health Disparities in Racial/Ethnic Minorities and the Medically Underserved; 2024 Sep 21-24; Los Angeles, CA. Philadelphia (PA): AACR; Cancer Epidemiol Biomarkers Prev 2024;33(9 Suppl):Abstract nr A084.

Abstract Purpose The Optimizing Performance Through Intrinsic Motivation and Attention for Learning (OPTIMAL) theory postulates that directing attention to the intended movement effect or outcome (external focus) is more effective than directing attention to the internal body motion or body part (internal focus). This study compared external versus internal focus of attention as novice students learned ultrasound-guided peripheral cannulation. The authors hypothesized that conditions promoting an external (vs internal) focus of attention would result in better performance on immediate and delayed (retention) testing. Method The authors conducted a randomized, experimental study from October 2022 to February 2023 comparing external and internal focus using ultrasound-guided peripheral venous cannulation followed by practice using an ultrasound training block. Undergraduate health science students performed 4 practice trials with instructions to focus on the needle tip (external focus) or their hands (internal focus), followed by immediate and delayed (retention) assessment 2 weeks later. The primary outcome was time to successful cannulation. Results were compared using Kaplan-Meier curves and parametric survival analysis regression (accelerated failure time) models. Results Seventy participants completed the immediate assessment. The external focus group completed successful cannulation a mean of 2.33 times faster (95% confidence interval [CI], 1.47–3.67; P &lt; .001) than those in the internal focus group (median time, 12.0 vs 29.5 seconds). Results also favored the external focus group during practice, with the external focus group successfully completing the task a mean of 1.83 times faster (95% CI, 1.04–3.21; P = .04) than the internal focus group. In the retention assessment, the external focus group successfully completed the task a mean of 2.37 times faster (95% CI, 1.19–4.74; P = .02) than the internal focus group. Conclusions External focus of attention compared with internal focus of attention improves medical motor skill learning in novice learners.