Citations

Abstract Objective In acupuncture therapy, the accurate location of acupoints is essential for its effectiveness. The advanced language understanding capabilities of large language models (LLMs) like Generative Pre-trained Transformers (GPTs) and Llama present a significant opportunity for extracting relations related to acupoint locations from textual knowledge sources. This study aims to explore the performance of LLMs in extracting acupoint-related location relations and assess the impact of fine-tuning on GPT’s performance. Materials and Methods We utilized the World Health Organization Standard Acupuncture Point Locations in the Western Pacific Region (WHO Standard) as our corpus, which consists of descriptions of 361 acupoints. Five types of relations (“direction_of”, “distance_of”, “part_of”, “near_acupoint”, and “located_near”) (n = 3174) between acupoints were annotated. Four models were compared: pre-trained GPT-3.5, fine-tuned GPT-3.5, pre-trained GPT-4, as well as pretrained Llama 3. Performance metrics included micro-average exact match precision, recall, and F1 scores. Results Our results demonstrate that fine-tuned GPT-3.5 consistently outperformed other models in F1 scores across all relation types. Overall, it achieved the highest micro-average F1 score of 0.92. Discussion The superior performance of the fine-tuned GPT-3.5 model, as shown by its F1 scores, underscores the importance of domain-specific fine-tuning in enhancing relation extraction capabilities for acupuncture-related tasks. In light of the findings from this study, it offers valuable insights into leveraging LLMs for developing clinical decision support and creating educational modules in acupuncture. Conclusion This study underscores the effectiveness of LLMs like GPT and Llama in extracting relations related to acupoint locations, with implications for accurately modeling acupuncture knowledge and promoting standard implementation in acupuncture training and practice. The findings also contribute to advancing informatics applications in traditional and complementary medicine, showcasing the potential of LLMs in natural language processing.

Abstract Alzheimer's disease and related dementias (ADRD) significantly impact patients and their caregivers, causing emotional stress, lack of training, and financial strain for the latter. Addressing the need for effective support, this research developed and validated ADQueryAid, a conversational AI system designed to empower ADRD caregivers. Built on a Large Language Model (LLM) and enriched with authoritative ADRD information through a knowledge graph, ADQueryAid uses Retrieval Augmented Generation (RAG) techniques to retrieve relevant information from both structured and unstructured data sources. Prompt engineering and conversation policies ensure the delivery of informative, empathetic, and personalized responses. Evaluated via a Blind Within-Subjects Design study with 20 ADRD caregivers interacting with both ADQueryAid and a baseline model (ChatGPT 3.5) in fictional caregiving scenarios, the system's usability was assessed using the Chatbot Usability Questionnaire (CUQ). Results showed ADQueryAid significantly outperformed the baseline model across all CUQ metrics, offering more contextually relevant information, accurate guidance, and emotional support, which enhanced the caregiving experience. This study highlights the potential of AI to support ADRD caregivers by providing personalized assistance, with future research focusing on expanding the knowledge base, refining conversational strategies, and exploring the long-term impact on caregiver well-being and patient outcomes.

Abstract Objectives Precise literature recommendation and summarization are crucial for biomedical professionals. While the latest iteration of generative pretrained transformer (GPT) incorporates 2 distinct modes—real-time search and pretrained model utilization—it encounters challenges in dealing with these tasks. Specifically, the real-time search can pinpoint some relevant articles but occasionally provides fabricated papers, whereas the pretrained model excels in generating well-structured summaries but struggles to cite specific sources. In response, this study introduces RefAI, an innovative retrieval-augmented generative tool designed to synergize the strengths of large language models (LLMs) while overcoming their limitations. Materials and Methods RefAI utilized PubMed for systematic literature retrieval, employed a novel multivariable algorithm for article recommendation, and leveraged GPT-4 turbo for summarization. Ten queries under 2 prevalent topics (“cancer immunotherapy and target therapy” and “LLMs in medicine”) were chosen as use cases and 3 established counterparts (ChatGPT-4, ScholarAI, and Gemini) as our baselines. The evaluation was conducted by 10 domain experts through standard statistical analyses for performance comparison. Results The overall performance of RefAI surpassed that of the baselines across 5 evaluated dimensions—relevance and quality for literature recommendation, accuracy, comprehensiveness, and reference integration for summarization, with the majority exhibiting statistically significant improvements (P-values <.05). Discussion RefAI demonstrated substantial improvements in literature recommendation and summarization over existing tools, addressing issues like fabricated papers, metadata inaccuracies, restricted recommendations, and poor reference integration. Conclusion By augmenting LLM with external resources and a novel ranking algorithm, RefAI is uniquely capable of recommending high-quality literature and generating well-structured summaries, holding the potential to meet the critical needs of biomedical professionals in navigating and synthesizing vast amounts of scientific literature.

AbstractBackground and AimEffective clinical event classification is essential for clinical research and quality improvement. The validation of artificial intelligence (AI) models like Generative Pre‐trained Transformer 4 (GPT‐4) for this task and comparison with conventional methods remains unexplored.MethodsWe evaluated the performance of the GPT‐4 model for classifying gastrointestinal (GI) bleeding episodes from 200 medical discharge summaries and compared the results with human review and an International Classification of Diseases (ICD) code‐based system. The analysis included accuracy, sensitivity, and specificity evaluation, using ground truth determined by physician reviewers.ResultsGPT‐4 exhibited an accuracy of 94.4% in identifying GI bleeding occurrences, outperforming ICD codes (accuracy 63.5%, P < 0.001). GPT‐4's accuracy was either slightly lower or statistically similar to individual human reviewers (Reviewer 1: 98.5%, P < 0.001; Reviewer 2: 90.8%, P = 0.170). For location classification, GPT‐4 achieved accuracies of 81.7% and 83.5% for confirmed and probable GI bleeding locations, respectively, with figures that were either slightly lower or comparable with those of human reviewers. GPT‐4 was highly efficient, analyzing the dataset in 12.7 min at a cost of 21.2 USD, whereas human reviewers required 8–9 h each.ConclusionOur study indicates GPT‐4 offers a reliable, cost‐efficient, and faster alternative to current clinical event classification methods, outperforming the conventional ICD coding system and performing comparably to individual expert human reviewers. Its implementation could facilitate more accurate and granular clinical research and quality audits. Future research should explore scalability, prompt and model tuning, and ethical implications of high‐performance AI models in clinical data processing.

Background The rapid evolution of artificial intelligence (AI) in conjunction with recent updates in dual antiplatelet therapy (DAPT) management guidelines emphasizes the necessity for innovative models to predict ischemic or bleeding events after drug‐eluting stent implantation. Leveraging AI for dynamic prediction has the potential to revolutionize risk stratification and provide personalized decision support for DAPT management. Methods and Results We developed and validated a new AI‐based pipeline using retrospective data of drug‐eluting stent‐treated patients, sourced from the Cerner Health Facts data set (n=98 236) and Optum's de‐identified Clinformatics Data Mart Database (n=9978). The 36 months following drug‐eluting stent implantation were designated as our primary forecasting interval, further segmented into 6 sequential prediction windows. We evaluated 5 distinct AI algorithms for their precision in predicting ischemic and bleeding risks. Model discriminative accuracy was assessed using the area under the receiver operating characteristic curve, among other metrics. The weighted light gradient boosting machine stood out as the preeminent model, thus earning its place as our AI‐DAPT model. The AI‐DAPT demonstrated peak accuracy in the 30 to 36 months window, charting an area under the receiver operating characteristic curve of 90% [95% CI, 88%–92%] for ischemia and 84% [95% CI, 82%–87%] for bleeding predictions. Conclusions Our AI‐DAPT excels in formulating iterative, refined dynamic predictions by assimilating ongoing updates from patients' clinical profiles, holding value as a novel smart clinical tool to facilitate optimal DAPT duration management with high accuracy and adaptability.