Citations

Abstract Large Language Models (LLMs) demonstrate strong capabilities in healthcare applications. However, recruiting human evaluators to assess usability and effectiveness becomes challenging in the large-scale LLM-generated outcomes. Existing metrics are often outdated for evaluating LLM-based healthcare technologies. This paper proposes a Research through Evaluation (RtE) approach that can refine metrics and improve assessments of LLM-generated outputs. We deploy RtE in a retrospective comparative study of prostate cancer patient inquiries, comparing responses from human clinical care teams with those generated by a GPT-4-based in-basket bot. Through three rounds of co-evaluations with clinical professionals and an LLM-as-graders study with four LLMs, the RtE method shows that starting with small, iterative evaluations helps human graders adjust metrics to suit their daily practices better. Additionally, in the LLM-as-graders study, LLMs exhibit similar behavior, preferring the in-basket bot's responses over the clinical care team. Our methodology and findings demonstrate that the RtE method can enhance the transparency and robustness of evaluation metrics for LLM-generated content, with the potential to generalize to evaluations in other clinical practices.

Importance Standard adjuvant therapy for human papillomavirus–associated oropharyngeal squamous cell carcinoma (HPV-OPSCC) is radiation therapy 60 Gy with or without cisplatin delivered in 30 fractions during 6 weeks. A phase 3 randomized clinical trial demonstrated decreased toxic effects and improved patient-reported outcomes with de-escalated adjuvant radiation therapy (DART) compared with standard therapy; however, outcomes for patients treated in the clinical practice setting following trial adoption remain unknown. Objective To compare characteristics and oncologic outcomes of patients treated with DART during clinical trials (on study) and in clinical practice settings (off study) and report progression-free survival (PFS) by risk category. Design, Setting, and Participants This retrospective cohort study included patients treated with DART in the MC1273 and MC1675 trials and in clinical practice after study completion at a multisite tertiary care center. Patients with surgically resectable HPV-OPSCC who underwent margin-negative resection between August 20, 2013, and March 21, 2023, and were treated with DART were included in the analysis. Those with prior head and neck cancer, pT0 or pT4 disease, unknown HPV status, metastatic disease at presentation, synchronous primary tumors, induction chemotherapy, or treatment at an outside institution were excluded. Data were analyzed from July 1, 2024, to July 18, 2025. Interventions DART consisting of 30 to 36 Gy in 1.5- to 1.8-Gy fractions twice daily with docetaxel 15 mg/m 2 on days 1 and 8. Main Outcomes and Measures Progression-free survival as measured via Kaplan-Meier analysis. Results A total of 282 patients were included in the analysis: 176 (62.4%) in the on-study group and 106 (37.6%) in the off-study group (252 [89.4%] male; mean [SD] age, 59.1 [9.3] years). Both groups were predominantly male (157 [89.2%] and 95 [89.6%], respectively) and consisted mostly of never smokers (118 [67.0%] and 62 [58.5%], respectively). Pathologic features were similar between groups except for nodal stage (pN1: 155 [88.1%] on study vs 102 [96.2%] off study; pN2: 21 [11.9%] on study vs 4 [3.8%] off study; P = .02). The median follow-up times were 5.2 years (95% CI, 5.2-5.8 years) in the on-study group and 2.2 years (95% CI, 1.9-2.4 years) in the off-study group. Two-year PFS was similar (93% [95% CI, 88%-97%] on study vs 93% [95% CI, 88%-99%] off study). For patients with pN1 and no extranodal extension (ENE), 2- and 5-year PFS were 97% (95% CI, 93%-100%) and 96% (95% CI, 92%-100%), respectively. Patients with pN1 and ENE had a 2- and 5-year PFS of 93% (95% CI, 89%-98%) and 87% (95% CI, 81%-94%), respectively. Conclusions and Relevance In this retrospective cohort study of patients receiving DART on and off study, oncologic outcomes were similar between the 2 groups. Fewer patients with pN2 received DART off study, given the interim results of MC1675. Patients with pN1 and no ENE treated with DART had excellent 2- and 5-year PFS, supporting DART as a compelling treatment option in well-selected patients. An additional study that is currently underway will evaluate biomarker risk-stratified treatment of patients with pN1 and ENE receiving DART.

Abstract Purpose: Head and neck (HN) radiotherapy contour quality directly impacts local control (LC) and survival; however, few departments peer review (PR) contours prior to radiotherapy planning (RTP). This study reports outcomes of a single institution’s formal HN contour PR process. Methods: A formal HN contour PR process was implemented. Prior to RTP, HN radiation oncologists reviewed contours, provided feedback and assigned contour grades as follows: R0 (no change), R1 (minor revision, not high risk) or R2 (major revision; high risk). The PR task was completed and the contour grade was recorded. Cochran-Armitage trend test was performed. Results: Pilot PR process was performed over a 7-month period for 88 patients, followed by a maintenance phase. Contours were graded as follows: R0 ( N = 51), R1 ( N = 20) and R2 ( N = 17). Over time, the number of R2 revisions decreased ( p = 0.0001); month 1 ( N = 7), month 2 ( N = 3), month 3 ( N = 5), month 4 ( N = 2) and months 5–7 ( N = 0). Conversely, the number of R0 revisions improved over time ( p = 0.0203); month 1 ( N = 5), months 2–3 ( N = 9), month 4 ( N = 5), month 5 ( N = 8), month 6 ( N = 12) and month 7 ( N = 3). Each radiation oncologist demonstrated reduction in R2 revisions during the pilot. During maintenance, all 3 radiation oncologists demonstrated low rates of R2 revisions of less than 3 cases per year. Conclusion: Incorporation of HN contour PR into routine clinical workflow is feasible. The collective experience of multiple high-volume HN radiation oncologists led to improved contour quality in the pilot for each radiation oncologist and continued to ensure high quality in the maintenance phase.

Abstract Background In the current clinical workflow of radiation oncology, therapists manually summarize the physician‐issued Computed Tomography (CT) simulation orders to prepare patients' CT simulations. This process increases the workload, introduces variability in documentation quality, and is prone to human errors. Purpose This study aims to address these challenges using a large language model (LLM) to automate the generation of summaries from the CT simulation orders and evaluate its performance. Methods A total of 607 CT simulation orders were collected from the Aria database at our institution. A locally hosted LLaMa 3.1 405B model, accessed via the Application Programming Interface (API) service, was used to extract keywords from the CT simulation orders and generate summaries. The downloaded CT simulation orders were categorized into seven groups based on treatment modalities and disease sites. For each group, a customized instruction prompt was developed collaboratively with therapists to guide the LLaMa 3.1 405B model in generating summaries. The ground truth for the corresponding summaries was manually derived by carefully reviewing each CT simulation order and subsequently verified by therapists. The accuracy of the LLM‐generated summaries was evaluated by therapists using the ground truth. Results Over 98% of the LLM‐generated summaries aligned with the ground truth in terms of accuracy. Our evaluations showed an improved consistency in format and enhanced readability from the LLM‐generated summaries compared to the corresponding therapist‐generated summaries. This automated approach demonstrated consistent performance across all groups, regardless of treatment modality or disease site. Conclusions This study demonstrated the high precision and consistency of the LLaMa 3.1 405B model in extracting keywords and summarizing CT simulation orders, suggesting that LLMs have great potential to assist in this task, reduce the workload of CT simulation therapists and improve radiation oncology workflow efficiency.

Abstract Background The radiation oncology clinical practice involves many steps relying on the dynamic interplay of abundant text data. Large language models have displayed remarkable capabilities in processing complex text information. But their direct applications in specific fields like radiation oncology remain underexplored. Purpose This study aims to investigate whether fine‐tuning LLMs with domain knowledge can improve the performance on Task (1) treatment regimen generation, Task (2) treatment modality selection (photon, proton, electron, or brachytherapy), and Task (3) ICD‐10 code prediction in radiation oncology. Methods Data for 15 724 patient cases were extracted. Cases where patients had a single diagnostic record, and a clearly identifiable primary treatment plan were selected for preprocessing and manual annotation to have 7903 cases of the patient diagnosis, treatment plan, treatment modality, and ICD‐10 code. Each case was used to construct a pair consisting of patient diagnostics details and an answer (treatment regimen, treatment modality, or ICD‐10 code, respectively) for the supervised fine‐tuning of these three tasks. Open source LLaMA2‐7B and Mistral‐7B models were utilized for the fine‐tuning with the Low‐Rank Approximations method. Accuracy and ROUGE‐1 score were reported for the fine‐tuned models and original models. Clinical evaluation was performed on Task (1) by radiation oncologists, while precision, recall, and F‐1 score were evaluated for Task (2) and (3). One‐sided Wilcoxon signed‐rank tests were used to statistically analyze the results. Results Fine‐tuned LLMs outperformed original LLMs across all tasks with p value ≤ 0.001. Clinical evaluation demonstrated that over 60% of the fine‐tuned LLMs‐generated treatment regimens were clinically acceptable. Precision, recall, and F1‐score showed improved performance of fine‐tuned LLMs. Conclusion Fine‐tuned LLMs demonstrated statistically significant improvements over original LLMs upon three clinically important tasks in radiation oncology. This study explored the feasibility of applying fine‐tuned LLMs in radiation oncology, inspiring further development of utilizing LLMs to assist with radiation oncology tasks.

ImportanceThe De-escalated Adjuvant Radiation Therapy (DART) phase 3 randomized clinical trial (RCT) showed that in patients with human papillomavirus (HPV)−associated oropharyngeal squamous cell carcinoma, postoperative minimal residual disease (MRD), detected through circulating tumor HPV DNA (ctHPVDNA), was associated with a higher risk of disease progression. When considered along with pathologic factors, postoperative ctHPVDNA assessment may improve patient selection for adjuvant treatment de-escalation; however, more data are needed to demonstrate how it may be used in personalizing treatment intensity.ObjectiveTo determine the association of postoperative MRD status with progression-free survival (PFS) after surgery for HPV-associated oropharyngeal squamous cell carcinoma.Design, Setting, and ParticipantsThis was a secondary analysis of the DART RCT, which was conducted from October 2016 to August 2020 in multiple sites in the US. Participants from the de-escalated adjuvant radiation therapy group and the standard of care group with available blood specimen data were included. Reports from 3-month posttreatment surveillance visits were used to assess associations and outcomes. Data analyses were performed from March 2023 to March of 2025.InterventionsThe DART group received 30 to 36 Gy of radiation therapy in 1.5 to 1.8 Gy twice daily, plus docetaxel, 15 mg/m2, on days 1 and 8. The standard of care group received 60 Gy with or without weekly cisplatin, 40 mg/m2.Main Outcome and MeasurePFS.ResultsThe analysis included 140 patients (mean [SD] age, 59.1 [8.4] years; 12 [8.6%] females and 128 [91.4%] males; 97 [69.3%] with no smoking history); characteristics were similar to the overall DART RCT population. Of these, 17 patients (12.1%) had postoperative MRD (13 of 96 [13.5%] receiving DART and 4 of 44 [9.1%] receiving standard of care). For all patients, postoperative MRD positivity was strongly associated with worsened PFS at 24 months (MRD positivity, 69.5%; MRD negativity, 95.9%; hazard ratio [HR], 0.19; 95% CI, 0.06-0.59). MRD positivity was associated with PFS when evaluating only those patients in the DART group, where 24-month PFS was 68.4% compared to 92.6% for MRD-negative patients (HR, 0.28; 95% CI, 0.08-0.93). Three months after completion of all treatment, 8 of 117 patients (6.8%) had detectable ctHPVDNA, whereas 109 of 117 (93.2%) did not, and detection was highly associated with PFS (HR, 20.48; 95% CI, 6.91-60.67).Conclusions and RelevanceThis secondary analysis of the DART RCT found that patients with detectable ctHPVDNA after surgery had a higher risk of disease progression. When added to the pathologic factors considered, ctHPVDNA assessment may improve selection of patients for treatment de-escalation. In addition, the 3-month posttreatment time point, early in surveillance, may identify a sizable portion of patients with progression and may guide intervention and surveillance after surgery for HPV-associated oropharyngeal squamous cell carcinoma.Trial RegistrationClinicalTrials.gov Identifier: NCT02908477

ABSTRACTBackgroundIn sinonasal cancer (SNC), treatment with proton therapy (PT) provides excellent local control, especially after gross total resection. Because of the heterogeneity and rarity of this disease site, a comprehensive assessment of toxicity, survival, and control rates is lacking. Our primary objective was to assess the toxicity outcomes of PT in SNC patients, with a secondary aim of assessing survival and tumor control after PT.MethodsPubMed, Embase, EBSCO, Scopus, Science Direct, Web of Science, Ovid, Proquest, and Cochrane Library were searched from inception to August 2024 reporting PT acute and late toxicity, survival, and tumor control outcomes in SNC patients. A random‐effect meta‐analysis was used to assess the pooled safety, survival, and tumor control outcomes. The primary analysis was to report acute and late toxicity. The secondary aims included overall survival (OS), disease‐free survival (DFS), local control (LC), regional control (RC), and distant metastasis control (DMC) rate.ResultsFourteen studies were included for qualitative analysis. We pooled data from 756 patients who received PT for SNC. Among acute toxicity (AT), there was a 31.9% occurrence rate of grade ≥ 3 events, whereas within late toxicity (LT), grade ≥ 3 events occurred at a rate of 35.3%. Most LT (62.1%) were classified as grade 2, with the most frequent being ocular (24.8%) or neurological (18.4%) toxicities. The most common grade ≥ 3 toxicities were mucositis (15.3%) in AT and ocular toxicity (9.6%) in LT. The pooled 5‐year OS, DFS, LC, RC, and DMC were 36.8%, 34.2%, 35.6%, 28.6%, and 54.3%, respectively.ConclusionOur analysis demonstrates that PT‐treated SNC patients experience acceptable rates of acute and LT consistent with other published outcomes with highly conformal radiation techniques. PT demonstrates favorable OS and DFS. Further prospective and comparative effectiveness research is needed to better quantify the magnitude of the benefit of PT or other forms of radiation modalities.

ABSTRACTBackgroundExternal and internal head and neck cancer related lymphedema (HNCRL) is a common consequence of radiation treatment (RT). Currently, internal HNCRL can be clinically assessed endoscopically using a visuoperceptual, ordinal rating scale. Use of fluoroscopy to identify and measure internal HNCRL has been explored but is not widely used in clinical practice. Advanced Pneumatic Compression (APC) has shown external HNCRL reduction after a single treatment, whereas internal HNCRL reduction has not yet been assessed.MethodsFluoroscopic lateral scouts and TIMS Review software (TIMS Medical) were used to assess immediate internal HNCRL measures of 30 HNC patients with RT history following a single APC treatment. Pre and post external measures were also obtained. Paired t‐test was used to assess changes in pre and post measures. A post‐treatment survey was completed.ResultsAll 30 patients had both immediate external and internal HNCRL reductions, and all reductions were statistically significant. All reported positive benefit.ConclusionsA single, external APC treatment immediately impacted the extent of pharyngeal and laryngeal edema in post‐radiated HNC survivors. Fluoroscopy can be a clinically useful and complementary modality to detect, surveil, and measure internal HNCRL.

AbstractBackgroundBolus materials have been used for decades in radiotherapy. Most frequently, these materials are utilized to bring dose closer to the skin surface to cover superficial targets optimally. While cavity filling, such as nasal cavities, is desirable, traditional commercial bolus is lacking, requiring other solutions. Recently, investigators have worked on utilizing 3D printing technology, including commercially available solutions, which can overcome some challenges with traditional bolus.PurposeTo utilize failure modes and effects analysis (FMEA) to successfully implement a comprehensive 3D printed bolus solution to replace commercial bolus in our clinic using a series of open‐source (or free) software products.Methods3D printed molds for bespoke bolus were created by exporting the DICOM structures of the bolus designed in the treatment planning system and manipulated to create a multipart mold for 3D printing. A silicone (Ecoflex 00–30) mixture is poured into the mold and cured to form the bolus. Molds for sheet bolus of five thicknesses were also created. A comprehensive FMEA was performed to guide workflow adjustments and QA steps.ResultsThe process map identified 39 and 30 distinct steps for the bespoke and flat sheet bolus workflows, respectively. The corresponding FMEA highlighted 119 and 86 failure modes, with 69 shared between the processes. Misunderstanding of plan intent was a potential cause for most of the highest‐scoring failure modes, indicating that physics and dosimetry involvement early in the process is paramount.ConclusionFMEA informed the design and implementation of QA steps to guarantee a safe and high‐quality comprehensive implementation of silicone bolus from 3D printed molds. This approach allows for greater adaptability not afforded by traditional bolus, as well as potential dissemination to other clinics due to the open‐source nature of the workflow.