Xu, Hua

PURPOSE Manual extraction of treatment outcomes from unstructured oncology clinical notes is a significant challenge for real-world evidence (RWE) generation. This study aimed to develop and evaluate a robust natural language processing (NLP) system to automatically extract cancer treatments and their associated RECIST-based response categories (complete response, partial response, stable disease, and progressive disease) from non–small cell lung cancer (NSCLC) clinical notes. METHODS This retrospective NLP development and validation study used a corpus of 250 NSCLC oncology notes from University of Pittsburgh Medical Center (UPMC) Hillman Cancer Center, annotated by physician experts. An end-to-end NLP pipeline was designed, integrating a rule-based module for entity extraction (treatments and responses) and a machine learning module using biomedical clinical bidirectional encoder representations from transformers for relation classification. The system's performance was evaluated on a held-out test set, with partial external validation for relation extraction on a Mayo Clinic data set. RESULTS The NLP system achieved high overall accuracy. On the UPMC test set (64 notes), the relation classification model attained an area under the receiver operating characteristic curve of 0.94 and an F1 score of 0.92 for linking treatments with documented responses. The rule-based entity extraction demonstrated a macro-averaged F1 score of 0.87 (precision 0.98, recall 0.81). Although precision was high for chemotherapy and most response types (1.00), recall for cancer surgery was 0.45. External validation at Mayo Clinic showed moderate relation extraction F1 scores (range: 0.51-0.64). CONCLUSION The proposed NLP system can reliably extract structured treatment and response information from unstructured NSCLC oncology notes with high accuracy. This automated approach can assist in abstracting critical cancer treatment outcomes from clinical narrative text, thereby streamlining real-world data analysis and supporting the generation of RWE in oncology.

Abstract Background Acupuncture, a key modality in traditional Chinese medicine, is gaining global recognition as a complementary therapy and a subject of increasing scientific interest. However, fragmented and unstructured acupuncture knowledge spread across diverse sources poses challenges for semantic retrieval, reasoning, and in-depth analysis. To address this gap, we developed AcuKG, a comprehensive knowledge graph that systematically organizes acupuncture-related knowledge to support sharing, discovery, and artificial intelligence–driven innovation in the field. Methods AcuKG integrates data from multiple sources, including online resources, guidelines, PubMed literature, ClinicalTrials.gov, and multiple ontologies (SNOMED CT, UBERON, and MeSH). We employed entity recognition, relation extraction, and ontology mapping to establish AcuKG, with human-in-the-loop to ensure data quality. Two cases evaluated AcuKG’s usability: (1) how AcuKG advances acupuncture research for obesity and (2) how AcuKG enhances large language model (LLM) application on acupuncture question-answering. Results AcuKG comprises 1839 entities and 11 527 relations, mapped to 1836 standard concepts in 3 ontologies. Two use cases demonstrated AcuKG’s effectiveness and potential in advancing acupuncture research and supporting LLM applications. In the obesity use case, AcuKG identified highly relevant acupoints (eg, ST25, ST36) and uncovered novel research insights based on evidence from clinical trials and literature. When applied to LLMs in answering acupuncture-related questions, integrating AcuKG with GPT-4o and LLaMA 3 significantly improved accuracy (GPT-4o: 46% → 54%, P = .03; LLaMA 3: 17% → 28%, P = .01). Conclusion AcuKG is an open dataset that provides a structured and computational framework for acupuncture applications, bridging traditional practices with acupuncture research and cutting-edge LLM technologies.

PURPOSE Large language models (LLMs) have demonstrated remarkable versatility in oncology applications, such as cancer staging and survival analysis. Despite their potential, ethical concerns such as data privacy breaches, bias in training data, lack of transparency, and risks associated with erroneous outputs pose significant challenges to their adoption in high-stakes oncology settings. Therefore, we aim to explore the ethical challenges associated with LLM-based applications in oncology and evaluate emerging techniques designed to address these issues. METHODS Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses framework, a systematic review was conducted to evaluate publications related to ethical issues of LLMs in oncology across eight academic databases (eg, PubMed, Web of Science, and Embase) between January 1, 2019, and December 31, 2024. RESULTS The search retrieved 4,319 published articles, of which 65 publications were preserved and included in our analysis. We identified six prevalent ethical challenges in oncology, including trust, equity, privacy, transparency, nonmaleficence, and accountability. We then evaluated emerging technical solutions to mitigate ethical challenges and summarized evaluation metrics used to assess these solutions' effectiveness. CONCLUSION This review provides actionable recommendations for responsibly deploying LLMs in oncology, ensuring adherence to ethical guidelines, and fostering improved patient outcomes. By bridging technical and clinical perspectives, this review offers a foundational framework for advancing ethical artificial intelligence applications in oncology and highlights areas for future research.

Abstract Objective In acupuncture therapy, the accurate location of acupoints is essential for its effectiveness. The advanced language understanding capabilities of large language models (LLMs) like Generative Pre-trained Transformers (GPTs) and Llama present a significant opportunity for extracting relations related to acupoint locations from textual knowledge sources. This study aims to explore the performance of LLMs in extracting acupoint-related location relations and assess the impact of fine-tuning on GPT’s performance. Materials and Methods We utilized the World Health Organization Standard Acupuncture Point Locations in the Western Pacific Region (WHO Standard) as our corpus, which consists of descriptions of 361 acupoints. Five types of relations (“direction_of”, “distance_of”, “part_of”, “near_acupoint”, and “located_near”) (n = 3174) between acupoints were annotated. Four models were compared: pre-trained GPT-3.5, fine-tuned GPT-3.5, pre-trained GPT-4, as well as pretrained Llama 3. Performance metrics included micro-average exact match precision, recall, and F1 scores. Results Our results demonstrate that fine-tuned GPT-3.5 consistently outperformed other models in F1 scores across all relation types. Overall, it achieved the highest micro-average F1 score of 0.92. Discussion The superior performance of the fine-tuned GPT-3.5 model, as shown by its F1 scores, underscores the importance of domain-specific fine-tuning in enhancing relation extraction capabilities for acupuncture-related tasks. In light of the findings from this study, it offers valuable insights into leveraging LLMs for developing clinical decision support and creating educational modules in acupuncture. Conclusion This study underscores the effectiveness of LLMs like GPT and Llama in extracting relations related to acupoint locations, with implications for accurately modeling acupuncture knowledge and promoting standard implementation in acupuncture training and practice. The findings also contribute to advancing informatics applications in traditional and complementary medicine, showcasing the potential of LLMs in natural language processing.

Abstract Despite recent methodology advancements in clinical natural language processing (NLP), the adoption of clinical NLP models within the translational research community remains hindered by process heterogeneity and human factor variations. Concurrently, these factors also dramatically increase the difficulty in developing NLP models in multi-site settings, which is necessary for algorithm robustness and generalizability. Here, we reported on our experience developing an NLP solution for Coronavirus Disease 2019 (COVID-19) signs and symptom extraction in an open NLP framework from a subset of sites participating in the National COVID Cohort (N3C). We then empirically highlight the benefits of multi-site data for both symbolic and statistical methods, as well as highlight the need for federated annotation and evaluation to resolve several pitfalls encountered in the course of these efforts.

Abstract Objective Social determinants of health (SDoH) play critical roles in health outcomes and well-being. Understanding the interplay of SDoH and health outcomes is critical to reducing healthcare inequalities and transforming a “sick care” system into a “health-promoting” system. To address the SDOH terminology gap and better embed relevant elements in advanced biomedical informatics, we propose an SDoH ontology (SDoHO), which represents fundamental SDoH factors and their relationships in a standardized and measurable way. Material and Methods Drawing on the content of existing ontologies relevant to certain aspects of SDoH, we used a top-down approach to formally model classes, relationships, and constraints based on multiple SDoH-related resources. Expert review and coverage evaluation, using a bottom-up approach employing clinical notes data and a national survey, were performed. Results We constructed the SDoHO with 708 classes, 106 object properties, and 20 data properties, with 1,561 logical axioms and 976 declaration axioms in the current version. Three experts achieved 0.967 agreement in the semantic evaluation of the ontology. A comparison between the coverage of the ontology and SDOH concepts in 2 sets of clinical notes and a national survey instrument also showed satisfactory results. Discussion SDoHO could potentially play an essential role in providing a foundation for a comprehensive understanding of the associations between SDoH and health outcomes and paving the way for health equity across populations. Conclusion SDoHO has well-designed hierarchies, practical objective properties, and versatile functionalities, and the comprehensive semantic and coverage evaluation achieved promising performance compared to the existing ontologies relevant to SDoH.

SummaryBackgroundPrevious studies have demonstrated an association between nonselective beta‐blockers (NSBBs) and lower risk of hepatocellular carcinoma (HCC) in cirrhosis. However, there has been no population‐based study investigating the risk of HCC among cirrhotic patients treated using carvedilol.AimsTo determine the risk of HCC among cirrhotic patients with NSBBs including carvedilol.MethodsThis retrospective cohort study utilised the Cerner Health Facts database in the United States from 2000 to 2017. Kaplan‐Meier estimate, Cox proportional hazards regression, and propensity score matching (PSM) were used to test the HCC risk among the carvedilol, nadolol, and propranolol groups compared with no beta‐blocker group.ResultsThe final cohort comprised 107 428 eligible patients. The 100‐month cumulative HCC incidence of NSBBs was significantly lower than the no beta‐blocker group (carvedilol (11.24%) vs no beta‐blocker (15.69%), nadolol (27.55%) vs no beta‐blocker (32.11%), and propranolol (26.17%) vs no beta‐blocker (28.84%) (P values < 0.0001). NSBBs were associated with a significantly lower risk of HCC (Hazard ratio: carvedilol 0.61 (95% CI 0.51‐0.73), nadolol 0.74 (95% CI 0.63‐0.87), propranolol 0.75 (95% CI 0.66‐0.84) after PSM in the multivariate cox analysis. In subgroup analysis, NSBBs reduced the risk of HCC in cirrhosis with complications and non‐alcoholic cirrhosis.ConclusionsNSBBs, including carvedilol, were associated with a significantly decreased risk of HCC in patients with cirrhosis when compared with no beta‐blocker regardless of complications status. Future randomised‐controlled studies comparing the incidence of HCC among NSBBs should elucidate which NSBB would be the best option to prevent HCC in cirrhosis.

Abstract Objective : Developing clinical natural language processing systems often requires access to many clinical documents, which are not widely available to the public due to privacy and security concerns. To address this challenge, we propose to develop methods to generate synthetic clinical notes and evaluate their utility in real clinical natural language processing tasks. Materials and Methods : We implemented 4 state-of-the-art text generation models, namely CharRNN, SegGAN, GPT-2, and CTRL, to generate clinical text for the History and Present Illness section. We then manually annotated clinical entities for randomly selected 500 History and Present Illness notes generated from the best-performing algorithm. To compare the utility of natural and synthetic corpora, we trained named entity recognition (NER) models from all 3 corpora and evaluated their performance on 2 independent natural corpora. Results : Our evaluation shows GPT-2 achieved the best BLEU (bilingual evaluation understudy) score (with a BLEU-2 of 0.92). NER models trained on synthetic corpus generated by GPT-2 showed slightly better performance on 2 independent corpora: strict F1 scores of 0.709 and 0.748, respectively, when compared with the NER models trained on natural corpus (F1 scores of 0.706 and 0.737, respectively), indicating the good utility of synthetic corpora in clinical NER model development. In addition, we also demonstrated that an augmented method that combines both natural and synthetic corpora achieved better performance than that uses the natural corpus only. Conclusions : Recent advances in text generation have made it possible to generate synthetic clinical notes that could be useful for training NER models for information extraction from natural clinical notes, thus lowering the privacy concern and increasing data availability. Further investigation is needed to apply this technology to practice.