Patel, Samir H.

Abstract Large Language Models (LLMs) demonstrate strong capabilities in healthcare applications. However, recruiting human evaluators to assess usability and effectiveness becomes challenging in the large-scale LLM-generated outcomes. Existing metrics are often outdated for evaluating LLM-based healthcare technologies. This paper proposes a Research through Evaluation (RtE) approach that can refine metrics and improve assessments of LLM-generated outputs. We deploy RtE in a retrospective comparative study of prostate cancer patient inquiries, comparing responses from human clinical care teams with those generated by a GPT-4-based in-basket bot. Through three rounds of co-evaluations with clinical professionals and an LLM-as-graders study with four LLMs, the RtE method shows that starting with small, iterative evaluations helps human graders adjust metrics to suit their daily practices better. Additionally, in the LLM-as-graders study, LLMs exhibit similar behavior, preferring the in-basket bot's responses over the clinical care team. Our methodology and findings demonstrate that the RtE method can enhance the transparency and robustness of evaluation metrics for LLM-generated content, with the potential to generalize to evaluations in other clinical practices.

Abstract Background In the current clinical workflow of radiation oncology, therapists manually summarize the physician‐issued Computed Tomography (CT) simulation orders to prepare patients' CT simulations. This process increases the workload, introduces variability in documentation quality, and is prone to human errors. Purpose This study aims to address these challenges using a large language model (LLM) to automate the generation of summaries from the CT simulation orders and evaluate its performance. Methods A total of 607 CT simulation orders were collected from the Aria database at our institution. A locally hosted LLaMa 3.1 405B model, accessed via the Application Programming Interface (API) service, was used to extract keywords from the CT simulation orders and generate summaries. The downloaded CT simulation orders were categorized into seven groups based on treatment modalities and disease sites. For each group, a customized instruction prompt was developed collaboratively with therapists to guide the LLaMa 3.1 405B model in generating summaries. The ground truth for the corresponding summaries was manually derived by carefully reviewing each CT simulation order and subsequently verified by therapists. The accuracy of the LLM‐generated summaries was evaluated by therapists using the ground truth. Results Over 98% of the LLM‐generated summaries aligned with the ground truth in terms of accuracy. Our evaluations showed an improved consistency in format and enhanced readability from the LLM‐generated summaries compared to the corresponding therapist‐generated summaries. This automated approach demonstrated consistent performance across all groups, regardless of treatment modality or disease site. Conclusions This study demonstrated the high precision and consistency of the LLaMa 3.1 405B model in extracting keywords and summarizing CT simulation orders, suggesting that LLMs have great potential to assist in this task, reduce the workload of CT simulation therapists and improve radiation oncology workflow efficiency.

Abstract Background The radiation oncology clinical practice involves many steps relying on the dynamic interplay of abundant text data. Large language models have displayed remarkable capabilities in processing complex text information. But their direct applications in specific fields like radiation oncology remain underexplored. Purpose This study aims to investigate whether fine‐tuning LLMs with domain knowledge can improve the performance on Task (1) treatment regimen generation, Task (2) treatment modality selection (photon, proton, electron, or brachytherapy), and Task (3) ICD‐10 code prediction in radiation oncology. Methods Data for 15 724 patient cases were extracted. Cases where patients had a single diagnostic record, and a clearly identifiable primary treatment plan were selected for preprocessing and manual annotation to have 7903 cases of the patient diagnosis, treatment plan, treatment modality, and ICD‐10 code. Each case was used to construct a pair consisting of patient diagnostics details and an answer (treatment regimen, treatment modality, or ICD‐10 code, respectively) for the supervised fine‐tuning of these three tasks. Open source LLaMA2‐7B and Mistral‐7B models were utilized for the fine‐tuning with the Low‐Rank Approximations method. Accuracy and ROUGE‐1 score were reported for the fine‐tuned models and original models. Clinical evaluation was performed on Task (1) by radiation oncologists, while precision, recall, and F‐1 score were evaluated for Task (2) and (3). One‐sided Wilcoxon signed‐rank tests were used to statistically analyze the results. Results Fine‐tuned LLMs outperformed original LLMs across all tasks with p value ≤ 0.001. Clinical evaluation demonstrated that over 60% of the fine‐tuned LLMs‐generated treatment regimens were clinically acceptable. Precision, recall, and F1‐score showed improved performance of fine‐tuned LLMs. Conclusion Fine‐tuned LLMs demonstrated statistically significant improvements over original LLMs upon three clinically important tasks in radiation oncology. This study explored the feasibility of applying fine‐tuned LLMs in radiation oncology, inspiring further development of utilizing LLMs to assist with radiation oncology tasks.

ImportanceThe De-escalated Adjuvant Radiation Therapy (DART) phase 3 randomized clinical trial (RCT) showed that in patients with human papillomavirus (HPV)−associated oropharyngeal squamous cell carcinoma, postoperative minimal residual disease (MRD), detected through circulating tumor HPV DNA (ctHPVDNA), was associated with a higher risk of disease progression. When considered along with pathologic factors, postoperative ctHPVDNA assessment may improve patient selection for adjuvant treatment de-escalation; however, more data are needed to demonstrate how it may be used in personalizing treatment intensity.ObjectiveTo determine the association of postoperative MRD status with progression-free survival (PFS) after surgery for HPV-associated oropharyngeal squamous cell carcinoma.Design, Setting, and ParticipantsThis was a secondary analysis of the DART RCT, which was conducted from October 2016 to August 2020 in multiple sites in the US. Participants from the de-escalated adjuvant radiation therapy group and the standard of care group with available blood specimen data were included. Reports from 3-month posttreatment surveillance visits were used to assess associations and outcomes. Data analyses were performed from March 2023 to March of 2025.InterventionsThe DART group received 30 to 36 Gy of radiation therapy in 1.5 to 1.8 Gy twice daily, plus docetaxel, 15 mg/m2, on days 1 and 8. The standard of care group received 60 Gy with or without weekly cisplatin, 40 mg/m2.Main Outcome and MeasurePFS.ResultsThe analysis included 140 patients (mean [SD] age, 59.1 [8.4] years; 12 [8.6%] females and 128 [91.4%] males; 97 [69.3%] with no smoking history); characteristics were similar to the overall DART RCT population. Of these, 17 patients (12.1%) had postoperative MRD (13 of 96 [13.5%] receiving DART and 4 of 44 [9.1%] receiving standard of care). For all patients, postoperative MRD positivity was strongly associated with worsened PFS at 24 months (MRD positivity, 69.5%; MRD negativity, 95.9%; hazard ratio [HR], 0.19; 95% CI, 0.06-0.59). MRD positivity was associated with PFS when evaluating only those patients in the DART group, where 24-month PFS was 68.4% compared to 92.6% for MRD-negative patients (HR, 0.28; 95% CI, 0.08-0.93). Three months after completion of all treatment, 8 of 117 patients (6.8%) had detectable ctHPVDNA, whereas 109 of 117 (93.2%) did not, and detection was highly associated with PFS (HR, 20.48; 95% CI, 6.91-60.67).Conclusions and RelevanceThis secondary analysis of the DART RCT found that patients with detectable ctHPVDNA after surgery had a higher risk of disease progression. When added to the pathologic factors considered, ctHPVDNA assessment may improve selection of patients for treatment de-escalation. In addition, the 3-month posttreatment time point, early in surveillance, may identify a sizable portion of patients with progression and may guide intervention and surveillance after surgery for HPV-associated oropharyngeal squamous cell carcinoma.Trial RegistrationClinicalTrials.gov Identifier: NCT02908477

ABSTRACTBackgroundIn sinonasal cancer (SNC), treatment with proton therapy (PT) provides excellent local control, especially after gross total resection. Because of the heterogeneity and rarity of this disease site, a comprehensive assessment of toxicity, survival, and control rates is lacking. Our primary objective was to assess the toxicity outcomes of PT in SNC patients, with a secondary aim of assessing survival and tumor control after PT.MethodsPubMed, Embase, EBSCO, Scopus, Science Direct, Web of Science, Ovid, Proquest, and Cochrane Library were searched from inception to August 2024 reporting PT acute and late toxicity, survival, and tumor control outcomes in SNC patients. A random‐effect meta‐analysis was used to assess the pooled safety, survival, and tumor control outcomes. The primary analysis was to report acute and late toxicity. The secondary aims included overall survival (OS), disease‐free survival (DFS), local control (LC), regional control (RC), and distant metastasis control (DMC) rate.ResultsFourteen studies were included for qualitative analysis. We pooled data from 756 patients who received PT for SNC. Among acute toxicity (AT), there was a 31.9% occurrence rate of grade ≥ 3 events, whereas within late toxicity (LT), grade ≥ 3 events occurred at a rate of 35.3%. Most LT (62.1%) were classified as grade 2, with the most frequent being ocular (24.8%) or neurological (18.4%) toxicities. The most common grade ≥ 3 toxicities were mucositis (15.3%) in AT and ocular toxicity (9.6%) in LT. The pooled 5‐year OS, DFS, LC, RC, and DMC were 36.8%, 34.2%, 35.6%, 28.6%, and 54.3%, respectively.ConclusionOur analysis demonstrates that PT‐treated SNC patients experience acceptable rates of acute and LT consistent with other published outcomes with highly conformal radiation techniques. PT demonstrates favorable OS and DFS. Further prospective and comparative effectiveness research is needed to better quantify the magnitude of the benefit of PT or other forms of radiation modalities.

ABSTRACTBackgroundExternal and internal head and neck cancer related lymphedema (HNCRL) is a common consequence of radiation treatment (RT). Currently, internal HNCRL can be clinically assessed endoscopically using a visuoperceptual, ordinal rating scale. Use of fluoroscopy to identify and measure internal HNCRL has been explored but is not widely used in clinical practice. Advanced Pneumatic Compression (APC) has shown external HNCRL reduction after a single treatment, whereas internal HNCRL reduction has not yet been assessed.MethodsFluoroscopic lateral scouts and TIMS Review software (TIMS Medical) were used to assess immediate internal HNCRL measures of 30 HNC patients with RT history following a single APC treatment. Pre and post external measures were also obtained. Paired t‐test was used to assess changes in pre and post measures. A post‐treatment survey was completed.ResultsAll 30 patients had both immediate external and internal HNCRL reductions, and all reductions were statistically significant. All reported positive benefit.ConclusionsA single, external APC treatment immediately impacted the extent of pharyngeal and laryngeal edema in post‐radiated HNC survivors. Fluoroscopy can be a clinically useful and complementary modality to detect, surveil, and measure internal HNCRL.

AbstractBackgroundBolus materials have been used for decades in radiotherapy. Most frequently, these materials are utilized to bring dose closer to the skin surface to cover superficial targets optimally. While cavity filling, such as nasal cavities, is desirable, traditional commercial bolus is lacking, requiring other solutions. Recently, investigators have worked on utilizing 3D printing technology, including commercially available solutions, which can overcome some challenges with traditional bolus.PurposeTo utilize failure modes and effects analysis (FMEA) to successfully implement a comprehensive 3D printed bolus solution to replace commercial bolus in our clinic using a series of open‐source (or free) software products.Methods3D printed molds for bespoke bolus were created by exporting the DICOM structures of the bolus designed in the treatment planning system and manipulated to create a multipart mold for 3D printing. A silicone (Ecoflex 00–30) mixture is poured into the mold and cured to form the bolus. Molds for sheet bolus of five thicknesses were also created. A comprehensive FMEA was performed to guide workflow adjustments and QA steps.ResultsThe process map identified 39 and 30 distinct steps for the bespoke and flat sheet bolus workflows, respectively. The corresponding FMEA highlighted 119 and 86 failure modes, with 69 shared between the processes. Misunderstanding of plan intent was a potential cause for most of the highest‐scoring failure modes, indicating that physics and dosimetry involvement early in the process is paramount.ConclusionFMEA informed the design and implementation of QA steps to guarantee a safe and high‐quality comprehensive implementation of silicone bolus from 3D printed molds. This approach allows for greater adaptability not afforded by traditional bolus, as well as potential dissemination to other clinics due to the open‐source nature of the workflow.

Purpose: This study aimed to develop a retrained large language model (LLM) tailored to the needs of HN cancer patients treated with radiotherapy, with emphasis on symptom management and survivorship care. Methods: A comprehensive external database was curated for training ChatGPT-4, integrating expert-identified consensus guidelines on supportive care for HN patients and correspondences from physicians and nurses within our institution’s electronic medical records for 90 HN patients. The performance of our model was evaluated using 20 patient post-treatment inquiries that were then assessed by three Board certified radiation oncologists (RadOncs). The rating of the model was assessed on a scale of 1 (strongly disagree) to 5 (strongly agree) based on accuracy, clarity of response, completeness s, and relevance. Results: The average scores for the 20 tested questions were 4.25 for accuracy, 4.35 for clarity, 4.22 for completeness, and 4.32 for relevance, on a 5-point scale. Overall, 91.67% (220 out of 240) of assessments received scores of 3 or higher, and 83.33% (200 out of 240) received scores of 4 or higher. Conclusion: The custom-trained model demonstrates high accuracy in providing support to HN patients offering evidence-based information and guidance on their symptom management and survivorship care.

6006 Background: IMPT has unique biologic and physical properties compared with IMRT, limits radiation dose beyond the targeted tumor volumes, and is a novel de-intensification strategy for the management of head and neck tumors. This study was designed to compare the outcomes for patients with OPC after chemoradiation therapy (CRT) with IMRT vs IMPT. Methods: This is a multi-center, randomized, phase III non-inferiority OPC trial Stage III/IV (AJCC 7th) squamous cell carcinoma stratified patients by human papillomavirus status, smoking status, and receipt of induction chemotherapy (IC). The primary endpoint was the rate of progression-free survival (PFS) rate at 3 years, where progression was defined as disease recurrence or death. Under the null hypothesis, H0: r ≥ 1.535 established the margin for non-inferiority of IMPT. Secondary endpoints include overall survival (OS), treatment-related malnutrition, and gastrostomy-tube dependence. Analyses were conducted on intent-to-treat (ITT; n=440), per-protocol (PP; n=296), and as-treated (AT; n=397) populations. Results: Patients (n=440) were randomized to undergo IMRT(n=219) or IMPT (n=221) at 21 institutions. The median age was 61 years and HPV/p16 was positive in 95%. IC was the initial treatment in 13% of patients. All patients were treated with CRT to 70 Gy in 33 fx with bilateral neck treatment, and post-CRT surgical lymph node dissection occurred in 8%. The median follow-up was 3.14 years. In the ITT analysis, the hazard ratio (HR) for disease progression or death at 3 y was 0.87 (95%CI 0.56,1.35); p=0.006 and the corresponding HR for death (OS) was 0.63 (95%CI 0.36-1.10) suggesting a protective affect with IMPT. In PP analysis, the PFS HR was 0.85 (95%CI 0.52,1.38); p=0.009 and HR for death (OS) was 0.60 (95%CI 0.32-1.12). In the AT analysis, PFS HR was 0.88 (95%CI 0.56,1.37); p=0.007 and the corresponding HR for death (OS) was 0.70 (95%CI 0.40-1.22). For each analysis above, the null hypothesis was rejected and IMPT was non-inferior to IMRT. PP gastrostomy-tube dependence decreased with IMPT vs. IMRT from 42% to 28% (p=0.019), and more IMPT patients sustained their nutrition with end of treatment weight loss < 5% from baseline: 24% vs 14% (p=0.037). Conclusions: IMPT is non-inferior to IMRT and has emerged as a standard of care CRT approach for OPC that reduces malnutrition and gastrostomy-tube dependence. Clinical trial information: NCT01893307 .