Liu, Tianming

Abstract Background Accurate and efficient dose calculation is essential for online adaptive planning in proton therapy. Deep learning (DL) has shown promising dose prediction results for pencil beam scanning proton therapy (PBSPT) in recent years, but existing DL‐based dose prediction methods still suffer from limited generalizability and an inability to effectively handle outlier clinical cases. This may lead to inaccurate dose delivery to targets or excessive irradiation to organs at risk (OARs), thereby compromising the safety and efficacy of online adaptive proton therapy. Purpose To design a physics‐aware and generalizable AI‐based PBSPT dose prediction method that incorporates underlying physics to enhance generalizability, particularly in handling outlier clinical cases. Methods This study analyzed PBSPT plans of 103 prostate (93 for training and 10 for testing) and 78 lung cancer patients (68 for training and 10 for testing) from our institution, with each case comprising CT images and structure sets. Using the doses generated by our Monte Carlo‐based dose engine as the reference standard, we compared three methods: the region of interest (ROI)‐based method, the beam mask and sliding window method, and the proposed noisy probing dose method, which rapidly generates a low‐statistics dose via uniformly weighted spots on an expanded spot‐placement target volume without optimization. To evaluate the generalizability of these methods to rare treatment planning scenarios, 12 cases with uncommon beam angles or prescription doses were used to assess their performance, which was evaluated using dose‐volume histogram (DVH) indices, 3D Gamma passing rates (3%/2 mm/10%), and Dice coefficients for dose agreement, while prediction times were measured to gauge model efficiency. Results The proposed noisy probing dose method consistently outperformed the ROI‐based and beam mask baselines across all evaluated metrics, with more accurate dose agreement and superior generalizability. For DVH indices, the noisy probing dose method achieved the smallest deviation in clinical target volume (CTV) dose coverage: in prostate cancer, CTV D98 deviation was reduced by 45% (from 0.53 ± 0.22 Gy [RBE] for ROI‐based) and 29% (from 0.41 ± 0.28 Gy [RBE] for beam mask) to 0.29 ± 0.06 Gy [RBE]; in lung cancer, similar improvements were observed, with CTV D98 deviations reduced to 0.34 ± 0.12 Gy [RBE]. The 3D Gamma passing rates improved to 99.65% ± 1.15% for prostate targets and 97.04% ± 1.17% for lung targets. The dice coefficients of the 90% iso‐dose lines were also the highest with the noisy probing dose method (prostate: 0.983 ± 0.005; lung: 0.967 ± 0.01). For the 12 outlier cases, the noisy probing dose method maintained superior generalizability, yielding higher 3D Gamma passing rates (prostate targets: 96.79% ± 0.83%, OARs: 94.29% ± 1.01%; lung targets: 93.38% ± 1.34%, OARs: 93.95% ± 1.32%), demonstrating robust generalizability to rare clinical scenarios. The dose predictions for all testing cases were completed within 0.3 seconds. Conclusions A novel noisy probing dose method was proposed for PBSPT dose prediction in prostate and lung cancer patients. By embedding more proton‐specific physics, this method demonstrated an improvement in the generalizability of dose prediction.

Abstract Background: For large language models (LLMs) to reach their potential as information technology tools that make medication use safer, clinically relevant benchmarks capable of automated grading and designed specifically to measure the performance of LLMs for medication tasks are required. The purpose of this study was to design a suite of benchmarking tests reflective of Comprehensive Medication Management (CMM; the standard of care for medication optimization) and quantify the baseline performance of the latest LLMs. Methods: We established six benchmarks representing critical stages of the CMM process: drug formulation matching, drug order (sig) generation, drug route matching, drug-drug interaction identification, renal dose identification, and drug-indication matching. For each benchmark, we curated a clinician-annotated dataset comprising 250 standardized input-output pairs including both inpatient and outpatient medications. We evaluated the clinical knowledge retrieval capabilities of three LLMs: GPT-4o-mini, MedGemma-27B, and LLaMA3-70B. We employed a zero-shot prompting strategy, excluding in-context examples, to assess the models’ internal clinical knowledge rather than their few-shot learning potential. To check reliability, each model was run three times using a temperature of 0.7 (a mid-range value of an LLM setting controlling text generation randomness). Performance was assessed using task-specific evaluation metrics including precision (positive predictive value), recall (sensitivity), F1-score, accuracy, and correctness consistency across trials. Results: Across six benchmarks, LLaMA3-70B demonstrated the highest performance in four tasks: drug-formulation matching (F1, 54.0% [95 CI: 50.1-58]), drug-order generation (accuracy, 88.0%), drug-route identification (F1, 74.3% [95 CI: 71-78]), and drug-indication identification (accuracy, 97.6% [95 CI: 95.6-99.2]). In the drug–drug interaction task, GPT-4o-mini achieved the highest overall accuracy (70.4% [95 CI: 64.8-75.7]). For renal dose–adjustment identification, GPT-4o-mini demonstrated the highest F1 score (83.3% [95 CI: 77.6-88]). Correctness-consistency scores ranged from 8.0% to 97.6% across benchmarks, with no model exhibiting uniformly superior consistency. Conclusions: Model performance varied substantially across medication-related tasks. LLaMA3-70B demonstrated promising baseline performance in tasks involving formulation, ordering, route, and indication. GPT-4o-mini showed potential advantages in drug–drug interaction detection and renal dose adjustment. These findings underscore the need for task-specific evaluation when deploying models for medication-focused clinical decision support.

Abstract Background The purpose of this study was to develop a series of benchmarks specifically for large language model (LLM) performance in the identification of drug-drug interactions (DDIs). Methods We evaluated six LLMs using a clinician-designed dataset of 750 DDI scenarios. Tasks included: (1) a pointwise two-drug classification task (2) a pairwise three-drug discrimination task and (3) a listwise 4–6 drug selection task. Zero-shot prompting was applied and performance was assessed using precision, recall, F1 score, accuracy, and self-consistency. Results Model performance varied by task structure and interaction severity. GPT-5-Chat demonstrated the highest recall and F1 score in the pointwise task (80.7% and 80.2%), whereas DrugGPT achieved superior accuracy in the pairwise (93.7%) and listwise task (92.6%). All models had improved performance with more severe compared to less severe DDIs (accuracy 72.0-99.2% vs. 58.3-86.7%). All models tended to have higher self-consistency in the pairwise and listwise experiments compared to the pointwise experiment (68.8-93.6% vs. 31.0-74.2%). No model exhibited uniformly high reliability across all tasks. Conclusions Current LLMs show inconsistent capabilities in identifying DDIs. Performance degrades as reasoning space expands with limited stability. These findings emphasize the need for multi-format evaluation frameworks and reliability-aware assessment when considering LLMs for medication-safety applications.

Abstract Background The radiation oncology clinical practice involves many steps relying on the dynamic interplay of abundant text data. Large language models have displayed remarkable capabilities in processing complex text information. But their direct applications in specific fields like radiation oncology remain underexplored. Purpose This study aims to investigate whether fine‐tuning LLMs with domain knowledge can improve the performance on Task (1) treatment regimen generation, Task (2) treatment modality selection (photon, proton, electron, or brachytherapy), and Task (3) ICD‐10 code prediction in radiation oncology. Methods Data for 15 724 patient cases were extracted. Cases where patients had a single diagnostic record, and a clearly identifiable primary treatment plan were selected for preprocessing and manual annotation to have 7903 cases of the patient diagnosis, treatment plan, treatment modality, and ICD‐10 code. Each case was used to construct a pair consisting of patient diagnostics details and an answer (treatment regimen, treatment modality, or ICD‐10 code, respectively) for the supervised fine‐tuning of these three tasks. Open source LLaMA2‐7B and Mistral‐7B models were utilized for the fine‐tuning with the Low‐Rank Approximations method. Accuracy and ROUGE‐1 score were reported for the fine‐tuned models and original models. Clinical evaluation was performed on Task (1) by radiation oncologists, while precision, recall, and F‐1 score were evaluated for Task (2) and (3). One‐sided Wilcoxon signed‐rank tests were used to statistically analyze the results. Results Fine‐tuned LLMs outperformed original LLMs across all tasks with p value ≤ 0.001. Clinical evaluation demonstrated that over 60% of the fine‐tuned LLMs‐generated treatment regimens were clinically acceptable. Precision, recall, and F1‐score showed improved performance of fine‐tuned LLMs. Conclusion Fine‐tuned LLMs demonstrated statistically significant improvements over original LLMs upon three clinically important tasks in radiation oncology. This study explored the feasibility of applying fine‐tuned LLMs in radiation oncology, inspiring further development of utilizing LLMs to assist with radiation oncology tasks.

Abstract Background Efficient and accurate delineation of organs at risk (OARs) is a critical procedure for treatment planning and dose evaluation. Deep learning‐based auto‐segmentation of OARs has shown promising results and is increasingly being used in radiation therapy. However, existing deep learning‐based auto‐segmentation approaches face two challenges in clinical practice: generalizability and human‐AI interaction. A generalizable and promptable auto‐segmentation model, which segments OARs of multiple disease sites simultaneously and supports on‐the‐fly human‐AI interaction, can significantly enhance the efficiency of radiation therapy treatment planning. Purpose Meta's segment anything model (SAM) was proposed as a generalizable and promptable model for next‐generation natural image segmentation. We further evaluated the performance of SAM in radiotherapy segmentation. Methods Computed tomography (CT) images of clinical cases from four disease sites at our institute were collected: prostate, lung, gastrointestinal, and head & neck. For each case, we selected the OARs important in radiotherapy treatment planning. We then compared both the Dice coefficients and Jaccard indices derived from three distinct methods: manual delineation (ground truth), automatic segmentation using SAM's ’segment anything’ mode, and automatic segmentation using SAM's ‘box prompt’ mode that implements manual interaction via live prompts during segmentation. Results Our results indicate that SAM's segment anything mode can achieve clinically acceptable segmentation results in most OARs with Dice scores higher than 0.7. SAM's box prompt mode further improves Dice scores by 0.1∼0.5. Similar results were observed for Jaccard indices. The results show that SAM performs better for prostate and lung, but worse for gastrointestinal and head & neck. When considering the size of organs and the distinctiveness of their boundaries, SAM shows better performance for large organs with distinct boundaries, such as lung and liver, and worse for smaller organs with less distinct boundaries, like parotid and cochlea. Conclusions Our results demonstrate SAM's robust generalizability with consistent accuracy in automatic segmentation for radiotherapy. Furthermore, the advanced box‐prompt method enables the users to augment auto‐segmentation interactively and dynamically, leading to patient‐specific auto‐segmentation in radiation therapy. SAM's generalizability across different disease sites and different modalities makes it feasible to develop a generic auto‐segmentation model in radiotherapy.

AbstractBackgroundAccurate and efficient dose calculation is essential for on‐line adaptive planning in proton therapy. Deep learning (DL) has shown promising dose prediction results in photon therapy. However, there is a scarcity of DL‐based dose prediction methods specifically designed for proton therapy. Successful dose prediction method for proton therapy should account for more challenging dose prediction problems in pencil beam scanning proton therapy (PBSPT) due to its sensitivity to heterogeneities.PurposeTo develop a DL‐based PBSPT dose prediction workflow with high accuracy and balanced complexity to support on‐line adaptive proton therapy clinical decision and subsequent replanning.MethodsPBSPT plans of 103 prostate cancer patients (93 for training and the other 10 for independent testing) and 83 lung cancer patients (73 for training and the other 10 for independent testing) previously treated at our institution were included in the study, each with computed tomography scans (CTs), structure sets, and plan doses calculated by the in‐house developed Monte‐Carlo dose engine (considered as the ground truth in the model training and testing). For the ablation study, we designed three experiments corresponding to the following three methods: (1) Experiment 1, the conventional region of interest (ROI) (composed of targets and organs‐at‐risk [OARs]) method. (2) Experiment 2, the beam mask (generated by raytracing of proton beams) method to improve proton dose prediction. (3) Experiment 3, the sliding window method for the model to focus on local details to further improve proton dose prediction. A fully connected 3D‐Unet was adopted as the backbone. Dose volume histogram (DVH) indices, 3D Gamma passing rates with a criterion of 3%/3 mm/10%, and dice coefficients for the structures enclosed by the iso‐dose lines between the predicted and the ground truth doses were used as the evaluation metrics. The calculation time for each proton dose prediction was recorded to evaluate the method's efficiency.ResultsCompared to the conventional ROI method, the beam mask method improved the agreement of DVH indices for both targets and OARs and the sliding window method further improved the agreement of the DVH indices (for lung cancer, CTV D98 absolute deviation: 0.74 ± 0.18 vs. 0.57 ± 0.21 vs. 0.54 ± 0.15 Gy[RBE], ROI vs. beam mask vs. sliding window methods, respectively). For the 3D Gamma passing rates in the target, OARs, and BODY (outside target and OARs), the beam mask method improved the passing rates in these regions and the sliding window method further improved them (for prostate cancer, targets: 96.93% ± 0.53% vs. 98.88% ± 0.49% vs. 99.97% ± 0.07%, BODY: 86.88% ± 0.74% vs. 93.21% ± 0.56% vs. 95.17% ± 0.59%). A similar trend was also observed for the dice coefficients. This trend was especially remarkable for relatively low prescription isodose lines (for lung cancer, 10% isodose line dice: 0.871 ± 0.027 vs. 0.911 ± 0.023 vs. 0.927 ± 0.017). The dose predictions for all the testing cases were completed within 0.25 s.ConclusionsAn accurate and efficient deep learning‐augmented proton dose prediction framework has been developed for PBSPT, which can predict accurate dose distributions not only inside but also outside ROI efficiently. The framework can potentially further reduce the initial planning and adaptive replanning workload in PBSPT.