Citations

Abstract Clinical empathy is essential for patient care, but physicians need continually balance emotional warmth with factual precision under the cognitive and emotional constraints of clinical practice. This study investigates how large language models (LLMs) can function as empathy editors, refining physicians’ written responses to enhance empathetic tone while preserving underlying medical information. More importantly, we introduce novel quantitative metrics, an Empathy Ranking Score and a MedFactChecking Score to systematically assess both emotional and factual quality of the responses. Experimental results show that LLM-edited responses significantly increase perceived empathy while preserving factual accuracy compared with fully LLM-generated outputs. These findings suggest that using LLMs as editorial assistants, rather than autonomous generators , offers a safer, more effective pathway to empathetic and trustworthy AI-assisted healthcare communication.

428 Background: Prostate cancer management presents a significant healthcare burden, with the need to efficiently triage patients for treatment. Our objective is to leverage large language models to predict physician-recommended treatment plans from unstructured clinical notes. By accurately predicting treatment plans, we aim to risk stratify and triage patients effectively, thereby optimizing the allocation of physician resources. Methods: 448 unstructured initial urology consultation patient notes following first positive prostate cancer biopsy were identified. The recommended and final treatments received were manually annotated to establish ground truth labels (Table 1). The dataset was split 80:20 for training and testing, preprocessed to remove plan sections and formatted into question-answer (QA) format. A domain-specific large language model (LLM) inspired by GPT and a specialized tokenizer (PCa- LLM) for prostate cancer terminology were developed. QA models were built using the PCa-LLM and compared with those using GPT-2 as the backbone to predict recommended and final treatments. Results: For the physician-recommended treatment plans, our LLM (PCa-LLM) showed superior performance with higher AUROC scores for curative vs. non-curative treatments (0.78 vs. 0.65), chemo-hormonal vs. other non-curative treatments (0.89 vs. 0.65), and surveillance vs. all other treatments (0.72 vs. 0.70), while both models achieved the same high AUROC of 0.99 for chemo-hormonal vs. all other treatments. For final treatments, PCa-LLM demonstrated better AUROC for curative vs. non-curative treatments (0.77 vs. 0.74) and chemo-hormonal vs. other non-curative treatments (0.71 vs. 0.66), while GPT2 outperformed PCa-LLM for surveillance vs. all other treatments (0.78 vs. 0.70). Both models achieved an AUROC of 0.99 for chemo-hormonal vs. all other treatments. Conclusions: PCa-LLM accurately predicted most treatment categories better than GPT2, with higher AUROC scores, and can be utilized to triage prostate cancer patients using initial consultation notes. Task Physician -Recommended Treatment Plan Final Treatment Received Curative Prostatectomy/Radiation 228 230 Non-Curative Focal Therapy 22 15 Active Surveillance 40 45 Chemo-hormonal 30 30 Model Predictions (AUROC) GPT2 PCa-LLM GPT2 PCa-LLM Curative vs. non-curative 0.65 0.78 0.74 0.77 Chemohormonal vs. other non-curative 0.65 0.89 0.66 0.71 Chemohormonal vs. all other 0.99 0.99 0.99 0.99 Surveillance vs. other non-curative 0.64 0.60 0.67 0.59 Surveillance vs. all other 0.70 0.72 0.78 0.70

PURPOSE Patient-centered outcomes (PCOs) are pivotal in cancer treatment, as they directly reflect patients' quality of life. Although multiple studies suggest that factors affecting breast cancer–related morbidity and survival are influenced by treatment side effects and adherence to long-term treatment, such data are generally only available on a smaller scale or from a single center. The primary challenge with collecting these data is that the outcomes are captured as free text in clinical narratives written by clinicians. MATERIALS AND METHODS Given the complexity of PCO documentation in these narratives, computerized methods are necessary to unlock the wealth of information buried in unstructured text notes that often document PCOs. Inspired by the success of large language models (LLMs), we examined the adaptability of three LLMs, GPT-2, BioGPT, and PMC-LLaMA, on PCO tasks across three institutions, Mayo Clinic, Emory University Hospital, and Stanford University. We developed an open-source framework for fine-tuning LLM that can directly extract the five different categories of PCO from the clinic notes. RESULTS We found that these LLMs without fine-tuning (zero-shot) struggle with challenging PCO extraction tasks, displaying almost random performance, even with some task-specific examples (few-shot learning). The performance of our fine-tuned, task-specific models is notably superior compared with their non–fine-tuned LLM models. Moreover, the fine-tuned GPT-2 model has demonstrated a significantly better performance than the other two larger LLMs. CONCLUSION Our discovery indicates that although LLMs serve as effective general-purpose models for tasks across various domains, they require fine-tuning when applied to the clinician domain. Our proposed approach has the potential to lead more efficient, adaptable models for PCO information extraction, reducing reliance on extensive computational resources while still delivering superior performance for specific tasks.

Abstract Objective To develop a natural language processing (NLP) algorithm that can accurately extract headache frequency from free‐text clinical notes. Background Headache frequency, defined as the number of days with any headache in a month (or 4 weeks), remains a key parameter in the evaluation of treatment response to migraine preventive medications. However, due to the variations and inconsistencies in documentation by clinicians, significant challenges exist to accurately extract headache frequency from the electronic health record (EHR) by traditional NLP algorithms. Methods This was a retrospective cross‐sectional study with patients identified from two tertiary headache referral centers, Mayo Clinic Arizona and Mayo Clinic Rochester. All neurology consultation notes written by 15 specialized clinicians (11 headache specialists and 4 nurse practitioners) between 2012 and 2022 were extracted and 1915 notes were used for model fine‐tuning (90%) and testing (10%). We employed four different NLP frameworks: (1) ClinicalBERT (Bidirectional Encoder Representations from Transformers) regression model, (2) Generative Pre‐Trained Transformer‐2 (GPT‐2) Question Answering (QA) model zero‐shot, (3) GPT‐2 QA model few‐shot training fine‐tuned on clinical notes, and (4) GPT‐2 generative model few‐shot training fine‐tuned on clinical notes to generate the answer by considering the context of included text. Results The mean (standard deviation) headache frequency of our training and testing datasets were 13.4 (10.9) and 14.4 (11.2), respectively. The GPT‐2 generative model was the best‐performing model with an accuracy of 0.92 (0.91, 0.93, 95% confidence interval [CI]) and R 2 score of 0.89 (0.87, 0.90, 95% CI), and all GPT‐2–based models outperformed the ClinicalBERT model in terms of exact matching accuracy. Although the ClinicalBERT regression model had the lowest accuracy of 0.27 (0.26, 0.28), it demonstrated a high R 2 score of 0.88 (0.85, 0.89), suggesting the ClinicalBERT model can reasonably predict the headache frequency within a range of ≤ ± 3 days, and the R 2 score was higher than the GPT‐2 QA zero‐shot model or GPT‐2 QA model few‐shot training fine‐tuned model. Conclusion We developed a robust information extraction model based on a state‐of‐the‐art large language model, a GPT‐2 generative model that can extract headache frequency from EHR free‐text clinical notes with high accuracy and R 2 score. It overcame several challenges related to different ways clinicians document headache frequency that were not easily achieved by traditional NLP models. We also showed that GPT‐2–based frameworks outperformed ClinicalBERT in terms of accuracy in extracting headache frequency from clinical notes. To facilitate research in the field, we released the GPT‐2 generative model and inference code with open‐source license of community use in GitHub. Additional fine‐tuning of the algorithm might be required when applied to different health‐care systems for various clinical use cases.