Waddle, Mark

Abstract Large Language Models (LLMs) demonstrate strong capabilities in healthcare applications. However, recruiting human evaluators to assess usability and effectiveness becomes challenging in the large-scale LLM-generated outcomes. Existing metrics are often outdated for evaluating LLM-based healthcare technologies. This paper proposes a Research through Evaluation (RtE) approach that can refine metrics and improve assessments of LLM-generated outputs. We deploy RtE in a retrospective comparative study of prostate cancer patient inquiries, comparing responses from human clinical care teams with those generated by a GPT-4-based in-basket bot. Through three rounds of co-evaluations with clinical professionals and an LLM-as-graders study with four LLMs, the RtE method shows that starting with small, iterative evaluations helps human graders adjust metrics to suit their daily practices better. Additionally, in the LLM-as-graders study, LLMs exhibit similar behavior, preferring the in-basket bot's responses over the clinical care team. Our methodology and findings demonstrate that the RtE method can enhance the transparency and robustness of evaluation metrics for LLM-generated content, with the potential to generalize to evaluations in other clinical practices.

Abstract Background Vorasidenib is Food and Drug Administration‐approved for treatment of grade 2 isocitrate dehydrogenase (IDH)–mutant gliomas and has demonstrated a benefit in progression‐free survival compared to a placebo. This study assessed the cost‐effectiveness of this treatment strategy compared to standard treatment: observation or chemoradiation (CRT). Methods The authors used an integrated Markov model to compare quality‐adjusted life‐years (QALY) gained and cost‐effectiveness of patients with IDH‐mutated low‐grade glioma (LGG) treated with vorasidenib, observation, or CRT. Outcomes were expressed as incremental cost‐effectiveness ratios (ICERs) per QALY gained. Treatment is considered cost‐effective if it meets the commonly accepted willingness‐to‐pay threshold of $100,000 per QALY gained. Results Based on model parameters, vorasidenib demonstrated an ICER of $356,756 per QALY when compared to CRT. When compared to observation, vorasidenib demonstrated an improved ICER of $215,191 per QALY. On one‐way sensitivity analysis compared to observation, median cost of vorasidenib and median overall survival were most likely to affect ICER with only marginal changes on analysis of other model factors. For ICER to become ≤$100,000 compared to observation, the cost of vorasidenib would need to drop to below $14,072 monthly or have a median survival gain of at least 11.9 years. The probabilistic sensitivity analysis demonstrated that vorasidenib was favored in none of 5000 Monte Carlo iterations. Conclusion Using the best available data, the current price of vorasidenib does not meet commonly accepted cost‐effectiveness thresholds in treatment of IDH‐mutated LGGs.

Retrieval-augmented generation (RAG) is rapidly emerging as a transformative paradigm for large language models (LLMs), especially in high-stakes domains like oncology that demand precision, factual grounding, and up-to-date knowledge. By pairing LLMs with external knowledge repositories, RAG systems explicitly ground model outputs in relevant retrieved documents, helping to reduce hallucinations and ensure responses reflect current evidence. In oncology, where clinical knowledge evolves continually with new research and drug approvals, RAG offers a way to integrate the latest data (e.g., trial results, guidelines, genomic databases) into decision-making. This review synthesizes the technical foundations of RAG, including its architecture and key components, and examines current applications in oncology such as clinical decision support, patient education, radiology reporting, pathology analysis, and genomics-driven precision medicine. We highlight recent studies that demonstrate RAG’s potential—for instance, improving treatment recommendations by incorporating genetic profiles and literature, and enhancing diagnostic accuracy by integrating guidelines. We also discuss emerging developments like multimodal RAG (combining text with imaging or other data), ensemble model approaches, and new explainability tools that trace model outputs to sources. Finally, we critically analyze the limitations and challenges of deploying RAG in healthcare, including computational costs, retrieval errors, noise or conflicts in retrieved information, and ethical and regulatory considerations. While RAG-based systems show promise in augmenting oncologists’ expertise with timely knowledge, careful implementation, high-quality curation of knowledge bases, and human oversight will be crucial for safe and effective adoption in clinical practice.

PURPOSE The purpose of this study was to determine if dose de-escalation from 60 to 66 Gy to 30 to 36 Gy of adjuvant radiotherapy (RT) for selected patients with human papillomavirus–associated oropharyngeal squamous cell carcinoma could maintain historical rates for disease control while reducing toxicity and preserving swallow function and quality of life (QOL). PATIENTS AND METHODS MC1273 was a single-arm phase II trial testing an aggressive course of RT de-escalation after surgery. Eligibility criteria included patients with p16-positive oropharyngeal squamous cell carcinoma, smoking history of 10 pack-years or less, and negative margins. Cohort A (intermediate risk) received 30 Gy delivered in 1.5-Gy fractions twice per day over 2 weeks along with 15 mg/m 2 docetaxel once per week. Cohort B included patients with extranodal extension who received the same treatment plus a simultaneous integrated boost to nodal levels with extranodal extension to 36 Gy in 1.8-Gy fractions twice per day. The primary end point was locoregional tumor control at 2 years. Secondary end points included 2-year progression-free survival, overall survival, toxicity, swallow function, and patient-reported QOL. RESULTS Accrual was from September 2013 to June 2016 (N = 80; cohort A, n = 37; cohort B, n = 43). Median follow-up was 36 months, with a minimum follow-up of 25 months. The 2-year locoregional tumor control rate was 96.2%, with progression-free survival of 91.1% and overall survival of 98.7%. Rates of grade 3 or worse toxicity at pre-RT and 1 and 2 years post-RT were 2.5%, 0%, and 0%. Swallowing function improved slightly between pre-RT and 12 months post-RT, with one patient requiring temporary feeding tube placement. CONCLUSION Aggressive RT de-escalation resulted in locoregional tumor control rates comparable to historical controls, low toxicity, and little decrement in swallowing function or QOL.

It is an exciting time for big data efforts in radiation oncology. The use of big data to help aid both outcomes and decision‐making research is becoming a reality. However, there are true challenges that exist in the space of gathering and utilizing performance and outcomes data. Here, we summarize the current state of big data in radiation oncology with respect to outcomes and discuss some of the efforts and challenges in radiation oncology big data.