Holmes, Jason

Abstract Large Language Models (LLMs) demonstrate strong capabilities in healthcare applications. However, recruiting human evaluators to assess usability and effectiveness becomes challenging in the large-scale LLM-generated outcomes. Existing metrics are often outdated for evaluating LLM-based healthcare technologies. This paper proposes a Research through Evaluation (RtE) approach that can refine metrics and improve assessments of LLM-generated outputs. We deploy RtE in a retrospective comparative study of prostate cancer patient inquiries, comparing responses from human clinical care teams with those generated by a GPT-4-based in-basket bot. Through three rounds of co-evaluations with clinical professionals and an LLM-as-graders study with four LLMs, the RtE method shows that starting with small, iterative evaluations helps human graders adjust metrics to suit their daily practices better. Additionally, in the LLM-as-graders study, LLMs exhibit similar behavior, preferring the in-basket bot's responses over the clinical care team. Our methodology and findings demonstrate that the RtE method can enhance the transparency and robustness of evaluation metrics for LLM-generated content, with the potential to generalize to evaluations in other clinical practices.

Abstract Background Accurate and efficient dose calculation is essential for online adaptive planning in proton therapy. Deep learning (DL) has shown promising dose prediction results for pencil beam scanning proton therapy (PBSPT) in recent years, but existing DL‐based dose prediction methods still suffer from limited generalizability and an inability to effectively handle outlier clinical cases. This may lead to inaccurate dose delivery to targets or excessive irradiation to organs at risk (OARs), thereby compromising the safety and efficacy of online adaptive proton therapy. Purpose To design a physics‐aware and generalizable AI‐based PBSPT dose prediction method that incorporates underlying physics to enhance generalizability, particularly in handling outlier clinical cases. Methods This study analyzed PBSPT plans of 103 prostate (93 for training and 10 for testing) and 78 lung cancer patients (68 for training and 10 for testing) from our institution, with each case comprising CT images and structure sets. Using the doses generated by our Monte Carlo‐based dose engine as the reference standard, we compared three methods: the region of interest (ROI)‐based method, the beam mask and sliding window method, and the proposed noisy probing dose method, which rapidly generates a low‐statistics dose via uniformly weighted spots on an expanded spot‐placement target volume without optimization. To evaluate the generalizability of these methods to rare treatment planning scenarios, 12 cases with uncommon beam angles or prescription doses were used to assess their performance, which was evaluated using dose‐volume histogram (DVH) indices, 3D Gamma passing rates (3%/2 mm/10%), and Dice coefficients for dose agreement, while prediction times were measured to gauge model efficiency. Results The proposed noisy probing dose method consistently outperformed the ROI‐based and beam mask baselines across all evaluated metrics, with more accurate dose agreement and superior generalizability. For DVH indices, the noisy probing dose method achieved the smallest deviation in clinical target volume (CTV) dose coverage: in prostate cancer, CTV D98 deviation was reduced by 45% (from 0.53 ± 0.22 Gy [RBE] for ROI‐based) and 29% (from 0.41 ± 0.28 Gy [RBE] for beam mask) to 0.29 ± 0.06 Gy [RBE]; in lung cancer, similar improvements were observed, with CTV D98 deviations reduced to 0.34 ± 0.12 Gy [RBE]. The 3D Gamma passing rates improved to 99.65% ± 1.15% for prostate targets and 97.04% ± 1.17% for lung targets. The dice coefficients of the 90% iso‐dose lines were also the highest with the noisy probing dose method (prostate: 0.983 ± 0.005; lung: 0.967 ± 0.01). For the 12 outlier cases, the noisy probing dose method maintained superior generalizability, yielding higher 3D Gamma passing rates (prostate targets: 96.79% ± 0.83%, OARs: 94.29% ± 1.01%; lung targets: 93.38% ± 1.34%, OARs: 93.95% ± 1.32%), demonstrating robust generalizability to rare clinical scenarios. The dose predictions for all testing cases were completed within 0.3 seconds. Conclusions A novel noisy probing dose method was proposed for PBSPT dose prediction in prostate and lung cancer patients. By embedding more proton‐specific physics, this method demonstrated an improvement in the generalizability of dose prediction.

Abstract Background In the current clinical workflow of radiation oncology, therapists manually summarize the physician‐issued Computed Tomography (CT) simulation orders to prepare patients' CT simulations. This process increases the workload, introduces variability in documentation quality, and is prone to human errors. Purpose This study aims to address these challenges using a large language model (LLM) to automate the generation of summaries from the CT simulation orders and evaluate its performance. Methods A total of 607 CT simulation orders were collected from the Aria database at our institution. A locally hosted LLaMa 3.1 405B model, accessed via the Application Programming Interface (API) service, was used to extract keywords from the CT simulation orders and generate summaries. The downloaded CT simulation orders were categorized into seven groups based on treatment modalities and disease sites. For each group, a customized instruction prompt was developed collaboratively with therapists to guide the LLaMa 3.1 405B model in generating summaries. The ground truth for the corresponding summaries was manually derived by carefully reviewing each CT simulation order and subsequently verified by therapists. The accuracy of the LLM‐generated summaries was evaluated by therapists using the ground truth. Results Over 98% of the LLM‐generated summaries aligned with the ground truth in terms of accuracy. Our evaluations showed an improved consistency in format and enhanced readability from the LLM‐generated summaries compared to the corresponding therapist‐generated summaries. This automated approach demonstrated consistent performance across all groups, regardless of treatment modality or disease site. Conclusions This study demonstrated the high precision and consistency of the LLaMa 3.1 405B model in extracting keywords and summarizing CT simulation orders, suggesting that LLMs have great potential to assist in this task, reduce the workload of CT simulation therapists and improve radiation oncology workflow efficiency.

Abstract Background The radiation oncology clinical practice involves many steps relying on the dynamic interplay of abundant text data. Large language models have displayed remarkable capabilities in processing complex text information. But their direct applications in specific fields like radiation oncology remain underexplored. Purpose This study aims to investigate whether fine‐tuning LLMs with domain knowledge can improve the performance on Task (1) treatment regimen generation, Task (2) treatment modality selection (photon, proton, electron, or brachytherapy), and Task (3) ICD‐10 code prediction in radiation oncology. Methods Data for 15 724 patient cases were extracted. Cases where patients had a single diagnostic record, and a clearly identifiable primary treatment plan were selected for preprocessing and manual annotation to have 7903 cases of the patient diagnosis, treatment plan, treatment modality, and ICD‐10 code. Each case was used to construct a pair consisting of patient diagnostics details and an answer (treatment regimen, treatment modality, or ICD‐10 code, respectively) for the supervised fine‐tuning of these three tasks. Open source LLaMA2‐7B and Mistral‐7B models were utilized for the fine‐tuning with the Low‐Rank Approximations method. Accuracy and ROUGE‐1 score were reported for the fine‐tuned models and original models. Clinical evaluation was performed on Task (1) by radiation oncologists, while precision, recall, and F‐1 score were evaluated for Task (2) and (3). One‐sided Wilcoxon signed‐rank tests were used to statistically analyze the results. Results Fine‐tuned LLMs outperformed original LLMs across all tasks with p value ≤ 0.001. Clinical evaluation demonstrated that over 60% of the fine‐tuned LLMs‐generated treatment regimens were clinically acceptable. Precision, recall, and F1‐score showed improved performance of fine‐tuned LLMs. Conclusion Fine‐tuned LLMs demonstrated statistically significant improvements over original LLMs upon three clinically important tasks in radiation oncology. This study explored the feasibility of applying fine‐tuned LLMs in radiation oncology, inspiring further development of utilizing LLMs to assist with radiation oncology tasks.

Objectives/Goals: Our study’s objective is to evaluate RadOnc-GPT, a GPT-4o powered LLM, in generating responses to in-basket messages related to prostate cancer treatment in the Radiation Oncology department. By integrating it with electronic health record (EHR) systems, the goal is to assess its impact on clinician workload, response quality, and efficiency in healthcare communication. Methods/Study Population: RadOnc-GPT was integrated with patient EHRs from both hospital-wide and radiation-oncology-specific databases. The study examined 158 pre-recorded in-basket message interactions from 90 non-metastatic prostate cancer patients. Quantitative natural language processing analysis and two randomized single-blinded grading studies, involving four clinicians and four nurses, were conducted to evaluate RadOnc-GPT’s response quality in completeness, correctness, clarity, empathy, and estimated editing time. Response times were measured to estimate the time saved for clinicians and nurses. The study population included patient messages across all phases of care (pre-, during, and post-treatment) for those undergoing radiotherapy. Results/Anticipated Results: In the single-blinded grader study, clinician graders evaluated 316 responses (158 from human care teams and 158 from RadOnc-GPT). Results showed RadOnc-GPT outperformed human responses in empathy and clarity, while humans excelled in completeness and correctness. Sentiment analyses using TextBlob and VADER revealed RadOnc-GPT responses had a positive mean score of 0.25, whereas human responses clustered around neutral. VADER analysis indicated a high median score for RadOnc-GPT, nearing 1.0, reflecting predominantly positive sentiment, while human responses displayed a broader sentiment range, indicating sensitivity to context. Clinicians averaged 3.60 minutes (SD 1.44) to respond, compared to 6.39 minutes (SD 4.05) for nurses, highlighting RadOnc-GPT’s efficiency in generating timely responses. Discussion/Significance of Impact: RadOnc-GPT effectively generated responses to individualized patient in-basket messages, comparable to those from radiation oncologists and nurses. While human oversight is still necessary to avoid errors, RadOnc-GPT can speed up response times and reduce pressure on care teams, shifting their role from drafting to reviewing responses.

Abstract Background Inter‐fractional and intra‐fractional positioning and motion monitoring based on implanted fiducials through kilovoltage (kV) imaging is a cost‐effective approach to enhance intact prostate radiotherapy treatment accuracy. However, comprehensive studies for three‐dimensional (3D) position extraction and the corrective action threshold are lacking. Purpose To develop and verify a fiducial motion monitoring system with 3D position information based on triggered kV images taken during treatment delivery, and to comprehensively evaluate the recommended thresholds for corrective action during treatment. Methods An in‐house fiducial triangulation algorithm was developed to monitor fiducial positions using subsequent kV triggered images. The precision of fiducial triangulation and motion detection was validated on a pelvis phantom with four fiducial inserts. A retrospective study was conducted on prostate cancer patients who received either moderately hypofractionated radiotherapy (MHRT, n = 16) or stereotactic body radiotherapy (SBRT, n = 12), categorized by endo‐rectal balloon (ERB) use. Intra‐fractional positions relative to isocenter were computed, and these positions were analyzed in comparison to the original marker locations of the patient's treatment plan. A linear regression fit was used per fraction to determine correlation coefficients between motions in the left–right (LR), superior–inferior (SI), and anterior–posterior (AP) directions. Average fraction time (AFT) of treatment was reported based on the observed average time spent on steps in the workflow and the percentage of fiducials caught outside of the current two‐dimensional (2D) tolerance threshold of 5 mm. The observed 3D offsets were used to estimate AFT for various 3D tolerance thresholds. Results Overall, our triangulation method proved to be very precise for static cases, where phantom measurements revealed a fiducial position triangulation precision of mm for stationary targets, but had a spread of mm for targets with 1 mm of motion. Employing triangulation, true motion exceeding 5 mm was detected above 4.0 mm in magnitude 90.9% of the time, a noticeable improvement in comparison to the 28%–65% successful detection rate reported by 2D projection methods. Most detection errors were attributed to depth disparity. Similar to other reports, correlation coefficients for intra‐fractional motion generally indicated no LR/SI, no LR/AP, and weak positive SI/AP correlations for MHRT and SBRT patients. Offsets beyond a 3D tolerance threshold of 5 mm were observed at a rate of 4.25%–5.25%, while the 5 mm 2D out‐of‐tolerance catch‐rate was 1.6%. The AFT was 8.1–8.2 min using the 2D tolerance threshold of 5 mm. In comparison, the estimated AFT for the proposed 3D tolerance monitoring of offsets beyond 2–5 mm was slightly higher at 8.2–10.9 min due to the higher amount of out‐of‐tolerance instances for the higher precision intra‐fractional motion management. Conclusions Our study showcases a promising subsequent kV‐based triangulation method for intra‐fractional prostate motion monitoring. Acquiring 3D motion information results in higher out‐of‐tolerance catch‐rates particularly in the depth dimension of the kV images, which is perpendicular to the treatment beam. Failure to properly observe and catch these offsets would result in sub‐optimal conformality and accuracy of the dose delivery.

425 Background: Prostate cancer (PC) ranks among the most prevalent cancers in men, with incidence rising due to factors like aging populations and enhanced screening techniques. Effective data extraction from clinical records is crucial for advancing research and improving patient care; however, traditional manual methods can be time-consuming and susceptible to mistakes. This study seeks to evaluate RadOnc-GPT against human annotation for extracting essential clinical information from prostate cancer clinical records. Methods: RadOnc-GPT is a retrieval augmented generation (RAG)-based chatbot with access to a variety of patient data including treatment details, radiology reports, and clinical notes. RadOnc-GPT uses Open AI’s gpt-4o model. Patients who received curative intent radiotherapy as initial treatment for PC in our institution between 2014 and 2023 were randomly selected for evaluation. Relevant Clinical factors were extracted by a human expert annotator on our on-site RedCap database. The extraction of the same variables was performed using RadOnc-GPT. A comparison between the ground truth (human annotation) and RadOnc-GPT, to evaluate the performance of the AI system, was carried out including accuracy, recall and F1 score. Results: A total of 382 PC patients were included in the cohort for data extraction. The clinical variables extracted were the treatment intent, NCCN risk category, T and N stage, Gleason score, radiation volume, and Androgen Deprivation therapy (ADT) use. The overall performance of RadOnc-GPT was excellent, with the highest accuracy achieved for treatment intent (99.7%), Gleason score (98.4%), and Treatment Volume (90.1-99.5%). Conclusions: RadOnc-GPT performed excellent at extracting relevant clinical details from the electronic medical record and radiation oncology information systems with dramatic reduction in time and cost compared with manual review. LLMs could be highly useful for large-scale data extraction in PC research, offering a promising solution for automating the accurate data extraction process. Correct (n) Total (n) Accuracy Recall F1 score RT with curative intent 381 382 99.7 99.9 99.9 Prostate Cancer Risk Category 345 382 90.3 90.4 95.1 Primary tumor staging (T) at diagnosis 368 382 96.3 96.3 97.9 Regional nodal status (N) at diagnosis 352 382 92.1 92.2 95.9 Gleason Score Primary Secondary Total 376372376 382382382 98.497.498.4 98.597.498.5 99.298.799.2 Treated Areas Prostate Pelvic Nodes Para-Aortic Nodes Distant Site 380344378372 382382382382 99.590.19997.4 99.490.199.197.4 99.79599.598.7 ADT Use with RT 365 382 95.5 95.7 97.8

Purpose: This study compares the performance of small, fine-tuned LLMs with the large original models on two radiation oncology clinical tasks: treatment modality selection and ICD-10 code prediction. Methods: We used data of 403 treatment modality selection cases and 377 ICD-10 code prediction cases. The models used were fine-tuned LLaMA 2 7B and Mistral 7B, as well as the larger LLaMA 3.1 70B and 405B models for comparison. We evaluated the accuracy of all models on these two tasks. Results: Both fine-tuned LLaMA 2 7B and Mistral 7B outperformed LLaMA 3.1 70B and 405B in treatment modality selection task. The fine-tuned LLaMA 2 7B and Mistral 7B models achieved accuracy comparable to the larger models in ICD-10 code prediction. Conclusions: Small models, fine-tuned with radiation oncology data, can achieve performance comparable to-or even surpass-that of larger models.

Abstract Background Efficient and accurate delineation of organs at risk (OARs) is a critical procedure for treatment planning and dose evaluation. Deep learning‐based auto‐segmentation of OARs has shown promising results and is increasingly being used in radiation therapy. However, existing deep learning‐based auto‐segmentation approaches face two challenges in clinical practice: generalizability and human‐AI interaction. A generalizable and promptable auto‐segmentation model, which segments OARs of multiple disease sites simultaneously and supports on‐the‐fly human‐AI interaction, can significantly enhance the efficiency of radiation therapy treatment planning. Purpose Meta's segment anything model (SAM) was proposed as a generalizable and promptable model for next‐generation natural image segmentation. We further evaluated the performance of SAM in radiotherapy segmentation. Methods Computed tomography (CT) images of clinical cases from four disease sites at our institute were collected: prostate, lung, gastrointestinal, and head & neck. For each case, we selected the OARs important in radiotherapy treatment planning. We then compared both the Dice coefficients and Jaccard indices derived from three distinct methods: manual delineation (ground truth), automatic segmentation using SAM's ’segment anything’ mode, and automatic segmentation using SAM's ‘box prompt’ mode that implements manual interaction via live prompts during segmentation. Results Our results indicate that SAM's segment anything mode can achieve clinically acceptable segmentation results in most OARs with Dice scores higher than 0.7. SAM's box prompt mode further improves Dice scores by 0.1∼0.5. Similar results were observed for Jaccard indices. The results show that SAM performs better for prostate and lung, but worse for gastrointestinal and head & neck. When considering the size of organs and the distinctiveness of their boundaries, SAM shows better performance for large organs with distinct boundaries, such as lung and liver, and worse for smaller organs with less distinct boundaries, like parotid and cochlea. Conclusions Our results demonstrate SAM's robust generalizability with consistent accuracy in automatic segmentation for radiotherapy. Furthermore, the advanced box‐prompt method enables the users to augment auto‐segmentation interactively and dynamically, leading to patient‐specific auto‐segmentation in radiation therapy. SAM's generalizability across different disease sites and different modalities makes it feasible to develop a generic auto‐segmentation model in radiotherapy.

Abstract Purpose . To enhance an in-house graphic-processing-unit accelerated virtual particle (VP)-based Monte Carlo (MC) proton dose engine (VPMC) to model aperture blocks in both dose calculation and optimization for pencil beam scanning proton therapy (PBSPT)-based stereotactic radiosurgery (SRS). Methods and materials . A module to simulate VPs passing through patient-specific aperture blocks was developed and integrated in VPMC based on simulation results of realistic particles (primary protons and their secondaries). To validate the aperture block module, VPMC was first validated by an opensource MC code, MCsquare, in eight water phantom simulations with 3 cm thick brass apertures: four were with aperture openings of 1, 2, 3, and 4 cm without a range shifter, while the other four were with same aperture opening configurations with a range shifter of 45 mm water equivalent thickness. Then, VPMC was benchmarked with MCsquare and RayStation MC for 10 patients with small targets (average volume 8.4 c.c. with range of 0.4–43.3 c.c.). Finally, 3 typical patients were selected for robust optimization with aperture blocks using VPMC. Results . In the water phantoms, 3D gamma passing rate (2%/2 mm/10%) between VPMC and MCsquare was 99.71 ± 0.23%. In the patient geometries, 3D gamma passing rates (3%/2 mm/10%) between VPMC/MCsquare and RayStation MC were 97.79 ± 2.21%/97.78 ± 1.97%, respectively. Meanwhile, the calculation time was drastically decreased from 112.45 ± 114.08 s (MCsquare) to 8.20 ± 6.42 s (VPMC) with the same statistical uncertainties of ~0.5%. The robustly optimized plans met all the dose–volume-constraints (DVCs) for the targets and OARs per our institutional protocols. The mean calculation time for 13 influence matrices in robust optimization by VPMC was 41.6 s and the subsequent on-the-fly ‘trial-and-error’ optimization procedure took only 71.4 s on average for the selected three patients. Conclusion . VPMC has been successfully enhanced to model aperture blocks in dose calculation and optimization for the PBSPT-based SRS.