Tao, Cui

Background: Data accessibility remains a significant barrier in healthcare AI due to privacy constraints and logistical challenges. Synthetic data, which mimics real patient information while remaining both realistic and non-identifiable, offers a promising solution. Large Language Models (LLMs) create new opportunities to generate high-fidelity clinical conversations between patients and physicians. However, the value of this synthetic data depends on careful evaluation of its realism, accuracy, and practical relevance. Objective: To assess the performance of four leading LLMs: ChatGPT 4.5, ChatGPT 4o, Claude 3.7 Sonnet, and Gemini Pro 2.5 in generating synthetic transcripts of patient–physician interactions in plastic surgery scenarios. Methods: Each model generated transcripts for ten plastic surgery scenarios. Transcripts were independently evaluated by three clinically trained raters using a seven-criterion rubric: Medical Accuracy, Realism, Persona Consistency, Fidelity, Empathy, Relevancy, and Usability. Raters were blinded to the model identity to reduce bias. Each was rated on a 5-point Likert scale, yielding 840 total evaluations. Descriptive statistics were computed, and a two-way repeated measures ANOVA was used to test for differences across models and metrics. In addition, transcripts were analyzed using automated linguistic and content-based metrics. Results: All models achieved strong performance, with mean ratings exceeding 4.5 across all criteria. Gemini 2.5 Pro received mean scores (5.00 ± 0.00) in Medical Accuracy, Realism, Persona Consistency, Relevancy, and Usability. Claude 3.7 Sonnet matched the scores in Persona Consistency and Relevancy and led in Empathy (4.96 ± 0.18). ChatGPT 4.5 also achieved perfect scores in Relevancy, with high scores in Empathy (4.93 ± 0.25) and Usability (4.96 ± 0.18). ChatGPT 4o demonstrated consistently strong but slightly lower performance across most dimensions. ANOVA revealed no statistically significant differences across models (F(3, 6) = 0.85, p = 0.52). Automated analysis showed substantial variation in transcript length, style, and content richness: Gemini 2.5 Pro generated the longest and most emotionally expressive dialogues, while ChatGPT 4o produced the shortest and most concise outputs. Conclusions: Leading LLMs can generate medically accurate, emotionally appropriate synthetic dialogues suitable for educational and research use. Despite high performance, demographic homogeneity in generated patients highlights the need for improved diversity and bias mitigation in model outputs. These findings support the cautious, context-aware integration of LLM-generated dialogues into medical training, simulation, and research.

Objective: Our project aims to use Artificial Intelligent (AI)-based methods to address overlooked issues in adolescent nutrition/diet and microbiome, and to identify biomarkers and risk factors that affect long-term health outcomes. Hypothesis: Many diseases, including diabetes, dementia, and Alzheimer’s disease, are linked to microbiome dysbiosis and diet-related risk factors, which can develop during adolescence and influence health outcomes throughout adulthood. Methods and Data: We conducted a systematic literature review on observational, clinical, and laboratory data to examine the relationships among diet quality, nutrition component, microbiome profiles, and energy expenditure, and their effects on behavior and health outcomes/diseases across adolescence, adulthood, and elderly populations (65+ yrs). We designed an analysis workflow to systematically extract and categorize data. Using AI-based methods we constructed knowledge models to uncover the relationships among key concepts on nutrition, diet, microbiome, genetic factors, age, sex, socioeconomic factors, and disease outcomes. Using the citations from PubMed literature, the Dietary Guidelines for Americans, and recommendations from authoritative health organizations, we developed a novel ChatGPT-based chatbot platform, Teen Fuel Up, designed to provide teens with real-time assistance to published knowledge about diet and nutrition. Conclusions: AI-based methods are highly effective in knowledge discovery and health promotion, identifying connections between adolescent nutrition, the microbiome, and chronic diseases later in life, as well as their differences using sex as a biological variable to increase scientific understanding of conditions that are specific to women. We will continue developing the ChatGPT-based chatbot to provide real-time cited knowledge that adapts to the evolving dietary needs of individuals. National Institute of Allergy And Infectious Diseases of the National Institutes of Health, grant number R01AI130460 and U24AI171008, and CPRIT award RP220244. This abstract was presented at the American Physiology Summit 2025 and is only available in HTML format. There is no downloadable file or PDF version. The Physiology editorial board was not involved in the peer review process.

The increasing integration of large language models (LLMs) into healthcare presents significant opportunities, but also critical risks related to patient safety, accuracy, and ethical alignment. Despite these concerns, no standardized framework exists for systematically evaluating and stress testing LLM behavior in clinical decision-making. The PIEE cycle—Planning and Preparation, Information Gathering and Prompt Generation, Execution, and Evaluation—is a structured red-teaming framework developed specifically to address artificial intelligence (AI) safety risks in healthcare decision-making. PIEE enables clinicians and informatics teams to simulate adversarial prompts, including jailbreaking, social engineering, and distractor attacks, to stress-test language models in real-world clinical scenarios. Model performance is evaluated using specific metrics such as true positive and false positive rates for detecting harmful content, hallucination rates measured through adapted TruthfulQA scoring, safety and reliability assessments, bias detection via adapted BBQ benchmarks, and ethical evaluation using structured Likert-based scoring rubrics. The framework is illustrated using examples from plastic surgery, but is adaptable across specialties, and is intended for use by all medical providers, regardless of their backgrounds or familiarity with artificial intelligence. While the framework is currently conceptual and validation is ongoing, PIEE provides a practical foundation for assessing the clinical reliability and ethical robustness of LLMs in medicine.

(1) Burn injuries demand multidisciplinary, evidence-based care, yet the extensive literature complicates timely decision making. Retrieval-augmented generation (RAG) synthesizes research while addressing inaccuracies in pretrained models. However, citation bias in sourcing for RAG often prioritizes highly cited studies, overlooking less-cited but valuable research. This study examines RAG’s performance in burn management, comparing citation levels to enhance evidence synthesis, reduce selection bias, and guide decisions. (2) Two burn management datasets were assembled: 30 highly cited (mean: 303) and 30 less-cited (mean: 21). The Gemini-1.0-Pro-002 RAG model addressed 30 questions, ranging from foundational principles to advanced surgical approaches. Responses were evaluated for accuracy (5-point scale), readability (Flesch–Kincaid metrics), and response time with Wilcoxon rank sum tests (p < 0.05). (3) RAG achieved comparable accuracy (4.6 vs. 4.2, p = 0.49), readability (Flesch Reading Ease: 42.8 vs. 46.5, p = 0.26; Grade Level: 9.9 vs. 9.5, p = 0.29), and response time (2.8 vs. 2.5 s, p = 0.39) for the highly and less-cited datasets. (4) Less-cited research performed similarly to highly cited sources. This equivalence broadens clinicians’ access to novel, diverse insights without sacrificing quality. As plastic surgery evolves, RAG’s inclusive approach fosters innovation, improves patient care, and reduces cognitive burden by integrating underutilized studies. Embracing RAG could propel the field toward dynamic, forward-thinking care.

Background/Objectives: Facial analysis is critical for preoperative planning in facial plastic surgery, but traditional methods can be time consuming and subjective. This study investigated the potential of Artificial Intelligence (AI) for objective and efficient facial analysis in plastic surgery, with a specific focus on Multimodal Large Language Models (MLLMs). We evaluated their ability to analyze facial skin quality, volume, symmetry, and adherence to aesthetic standards such as neoclassical facial canons and the golden ratio. Methods: We evaluated four MLLMs—ChatGPT-4o, ChatGPT-4, Gemini 1.5 Pro, and Claude 3.5 Sonnet—using two evaluation forms and 15 diverse facial images generated by a Generative Adversarial Network (GAN). The general analysis form evaluated qualitative skin features (texture, type, thickness, wrinkling, photoaging, and overall symmetry). The facial ratios form assessed quantitative structural proportions, including division into equal fifths, adherence to the rule of thirds, and compatibility with the golden ratio. MLLM assessments were compared with evaluations from a plastic surgeon and manual measurements of facial ratios. Results: The MLLMs showed promise in analyzing qualitative features, but they struggled with precise quantitative measurements of facial ratios. Mean accuracy for general analysis were ChatGPT-4o (0.61 ± 0.49), Gemini 1.5 Pro (0.60 ± 0.49), ChatGPT-4 (0.57 ± 0.50), and Claude 3.5 Sonnet (0.52 ± 0.50). In facial ratio assessments, scores were lower, with Gemini 1.5 Pro achieving the highest mean accuracy (0.39 ± 0.49). Inter-rater reliability, based on Cohen’s Kappa values, ranged from poor to high for qualitative assessments (κ > 0.7 for some questions) but was generally poor (near or below zero) for quantitative assessments. Conclusions: Current general purpose MLLMs are not yet ready to replace manual clinical assessments but may assist in general facial feature analysis. These findings are based on testing models not specifically trained for facial analysis and serve to raise awareness among clinicians regarding the current capabilities and inherent limitations of readily available MLLMs in this specialized domain. This limitation may stem from challenges with spatial reasoning and fine-grained detail extraction, which are inherent limitations of current MLLMs. Future research should focus on enhancing the numerical accuracy and reliability of MLLMs for broader application in plastic surgery, potentially through improved training methods and integration with other AI technologies such as specialized computer vision algorithms for precise landmark detection and measurement.

Background: Anatomically accurate illustrations are imperative in medical education, serving as crucial tools to facilitate comprehension of complex anatomical structures. While traditional illustration methods involving human artists remain the gold standard, the rapid advancement of Generative Artificial Intelligence (GAI) models presents a new opportunity to automate and accelerate this process. This study evaluated the potential of GAI models to produce craniofacial anatomy illustrations for educational purposes. Methods: Four GAI models, including Midjourney v6.0, DALL-E 3, Gemini Ultra 1.0, and Stable Diffusion 2.0 were used to generate 736 images across multiple views of surface anatomy, bones, muscles, blood vessels, and nerves of the cranium in both oil painting and realistic photograph styles. Four reviewers evaluated the images for anatomical detail, aesthetic quality, usability, and cost-effectiveness. Inter-rater reliability analysis assessed evaluation consistency. Results: Midjourney v6.0 scored highest for aesthetic quality and cost-effectiveness, and DALL-E 3 performed best for anatomical detail and usability. The inter-rater reliability analysis demonstrated a high level of agreement among reviewers (ICC = 0.858, 95% CI). However, all models showed significant flaws in depicting crucial anatomical details such as foramina, suture lines, muscular origins/insertions, and neurovascular structures. These limitations were further characterized by abstract depictions, mixing of layers, shadowing, abnormal muscle arrangements, and labeling errors. Conclusions: These findings highlight GAI’s potential for rapidly creating craniofacial anatomy illustrations but also its current limitations due to inadequate training data and incomplete understanding of complex anatomy. Refining these models through precise training data and expert feedback is vital. Ethical considerations, such as potential biases, copyright challenges, and the risks of propagating inaccurate information, must also be carefully navigated. Further refinement of GAI models and ethical safeguards are essential for safe use.

Abstract Objective The rapid evolution of the COVID-19 virus has led to the development of different vaccine shots, each designed to combat specific variants and enhance overall efficacy. While vaccines have been crucial in controlling the spread of the virus, they can also cause adverse events (AEs). Understanding these relationships is vital for vaccine safety monitoring and surveillance. Methods In our study, we collected data from the Vaccine Adverse Event Reporting System (VAERS) and social media platforms (Twitter and Reddit) to extract relationships between COVID-19 vaccine shots and adverse events. The dataset comprised 771 relation pairs, enabling a comprehensive analysis of adverse event patterns. We employed state-of-the-art GPT models, including GPT-3.5 and GPT-4, alongside traditional models such as Recurrent Neural Networks (RNNs) and BioBERT, to extract these relationships. Additionally, we used two sets of post-processing rules to further refine the extracted relations. Evaluation metrics including precision, recall, and F1-score were used to assess the performance of our models in extracting these relationships accurately. Results The most commonly reported AEs following the primary series of COVID-19 vaccines include arm soreness, fatigue, and headache, while the spectrum of AEs following boosters is more diverse. In relation extraction, fine-tuned GPT-3.5 with Sentence-based Relation Identification achieved the highest precision of 0.94 and a perfect recall of 1, resulting in an impressive F1 score of 0.97. Conclusion This study advances biomedical informatics by showing how large language models and deep learning models can extract relationships between vaccine shots and adverse events from VAERS and social media. These findings improve vaccine safety monitoring and clinical practice by enhancing our understanding of post-vaccination symptoms. The study sets a precedent for future research in natural language processing and biomedical informatics, with potential applications in pharmacovigilance and clinical decision-making.

Abstract Introduction COVID-19, a global health crisis, prompted rapid vaccine development. Bivalent vaccines by Pfizer-BioNTech and Moderna target both original and Omicron strains. Despite vaccination efforts, the persistence of adverse events underscores the importance of ongoing safety monitoring. Method We collected COVID-19 bivalent vaccine adverse event reports from the Vaccine Adverse Event Reporting System (VAERS) (September 15, 2022, to September 1, 2023). The Medical Dictionary for Regulatory Activities (MedDRA) categorized reported symptoms into corresponding concepts. Statistical analyses included zero-truncated Poisson regression for estimating symptom rates, logistic regression for identifying risk factors, and Spearman correlation for assessing non-parametric relationships between adverse events in different System Organ Classes (SOCs). Results Our analysis of COVID-19 bivalent vaccine adverse events identified significant correlations between SOCs. Notably, a positive correlation (ρ = 0.3) linked Infections (SOC 1) with Respiratory disorders (SOC 13), while a negative correlation (ρ=-0.3) connected Investigations (SOC 23) with Injury (SOC 24). Zero-truncated Poisson modeling estimated an average of 6.179 symptoms per individual, varying by age, gender, and vaccine. Gender and age significantly influence reported adverse events associated with COVID-19 bivalent vaccines. Females generally experienced higher odds for most SOCs compared to males, except for cardiac and product-related issues. Older adults showed a higher likelihood of symptoms across various SOCs, while children and adolescents had distinct susceptibility patterns. Additionally, Moderna and Pfizer vaccines were associated with different SOCs, with Moderna linked mainly to immune and skin disorders and Pfizer associated primarily with cardiac and metabolic issues. Conclusion In conclusion, our study aligned self-reported items with widely-accepted medical terminology and rigorously analyzed adverse events following COVID-19 bivalent vaccine using advanced statistical methods. The significant findings provide crucial insights, guiding evidence-based strategies and interventions for vaccine safety.