Zuo, Xu

Abstract Objective The rapid evolution of the COVID-19 virus has led to the development of different vaccine shots, each designed to combat specific variants and enhance overall efficacy. While vaccines have been crucial in controlling the spread of the virus, they can also cause adverse events (AEs). Understanding these relationships is vital for vaccine safety monitoring and surveillance. Methods In our study, we collected data from the Vaccine Adverse Event Reporting System (VAERS) and social media platforms (Twitter and Reddit) to extract relationships between COVID-19 vaccine shots and adverse events. The dataset comprised 771 relation pairs, enabling a comprehensive analysis of adverse event patterns. We employed state-of-the-art GPT models, including GPT-3.5 and GPT-4, alongside traditional models such as Recurrent Neural Networks (RNNs) and BioBERT, to extract these relationships. Additionally, we used two sets of post-processing rules to further refine the extracted relations. Evaluation metrics including precision, recall, and F1-score were used to assess the performance of our models in extracting these relationships accurately. Results The most commonly reported AEs following the primary series of COVID-19 vaccines include arm soreness, fatigue, and headache, while the spectrum of AEs following boosters is more diverse. In relation extraction, fine-tuned GPT-3.5 with Sentence-based Relation Identification achieved the highest precision of 0.94 and a perfect recall of 1, resulting in an impressive F1 score of 0.97. Conclusion This study advances biomedical informatics by showing how large language models and deep learning models can extract relationships between vaccine shots and adverse events from VAERS and social media. These findings improve vaccine safety monitoring and clinical practice by enhancing our understanding of post-vaccination symptoms. The study sets a precedent for future research in natural language processing and biomedical informatics, with potential applications in pharmacovigilance and clinical decision-making.

Abstract Objective In acupuncture therapy, the accurate location of acupoints is essential for its effectiveness. The advanced language understanding capabilities of large language models (LLMs) like Generative Pre-trained Transformers (GPTs) and Llama present a significant opportunity for extracting relations related to acupoint locations from textual knowledge sources. This study aims to explore the performance of LLMs in extracting acupoint-related location relations and assess the impact of fine-tuning on GPT’s performance. Materials and Methods We utilized the World Health Organization Standard Acupuncture Point Locations in the Western Pacific Region (WHO Standard) as our corpus, which consists of descriptions of 361 acupoints. Five types of relations (“direction_of”, “distance_of”, “part_of”, “near_acupoint”, and “located_near”) (n = 3174) between acupoints were annotated. Four models were compared: pre-trained GPT-3.5, fine-tuned GPT-3.5, pre-trained GPT-4, as well as pretrained Llama 3. Performance metrics included micro-average exact match precision, recall, and F1 scores. Results Our results demonstrate that fine-tuned GPT-3.5 consistently outperformed other models in F1 scores across all relation types. Overall, it achieved the highest micro-average F1 score of 0.92. Discussion The superior performance of the fine-tuned GPT-3.5 model, as shown by its F1 scores, underscores the importance of domain-specific fine-tuning in enhancing relation extraction capabilities for acupuncture-related tasks. In light of the findings from this study, it offers valuable insights into leveraging LLMs for developing clinical decision support and creating educational modules in acupuncture. Conclusion This study underscores the effectiveness of LLMs like GPT and Llama in extracting relations related to acupoint locations, with implications for accurately modeling acupuncture knowledge and promoting standard implementation in acupuncture training and practice. The findings also contribute to advancing informatics applications in traditional and complementary medicine, showcasing the potential of LLMs in natural language processing.

PURPOSE The RECIST guidelines provide a standardized approach for evaluating the response of cancer to treatment, allowing for consistent comparison of treatment efficacy across different therapies and patients. However, collecting such information from electronic health records manually can be extremely labor-intensive and time-consuming because of the complexity and volume of clinical notes. The aim of this study is to apply natural language processing (NLP) techniques to automate this process, minimizing manual data collection efforts, and improving the consistency and reliability of the results. METHODS We proposed a complex, hybrid NLP system that automates the process of extracting, linking, and summarizing anticancer therapy and associated RECIST-like responses from narrative clinical text. The system consists of multiple machine learning–/deep learning–based and rule-based modules for diverse NLP tasks such as named entity recognition, assertion classification, relation extraction, and text normalization, to address different challenges associated with anticancer therapy and response information extraction. We then evaluated the system performances on two independent test sets from different institutions to demonstrate its effectiveness and generalizability. RESULTS The system used domain-specific language models, BioBERT and BioClinicalBERT, for high-performance therapy mentions identification and RECIST responses extraction and categorization. The best-performing model achieved a 0.66 score in linking therapy and RECIST response mentions, with end-to-end performance peaking at 0.74 after relation normalization, indicating substantial efficacy with room for improvement. CONCLUSION We developed, implemented, and tested an information extraction system from clinical notes for cancer treatment and efficacy assessment information. We expect this system will support future cancer research, particularly oncologic studies that focus on efficiently assessing the effectiveness and reliability of cancer therapeutics.