Citations

Abstract Large Language Models (LLMs) demonstrate strong capabilities in healthcare applications. However, recruiting human evaluators to assess usability and effectiveness becomes challenging in the large-scale LLM-generated outcomes. Existing metrics are often outdated for evaluating LLM-based healthcare technologies. This paper proposes a Research through Evaluation (RtE) approach that can refine metrics and improve assessments of LLM-generated outputs. We deploy RtE in a retrospective comparative study of prostate cancer patient inquiries, comparing responses from human clinical care teams with those generated by a GPT-4-based in-basket bot. Through three rounds of co-evaluations with clinical professionals and an LLM-as-graders study with four LLMs, the RtE method shows that starting with small, iterative evaluations helps human graders adjust metrics to suit their daily practices better. Additionally, in the LLM-as-graders study, LLMs exhibit similar behavior, preferring the in-basket bot's responses over the clinical care team. Our methodology and findings demonstrate that the RtE method can enhance the transparency and robustness of evaluation metrics for LLM-generated content, with the potential to generalize to evaluations in other clinical practices.

PURPOSE Cancer recurrence is a critical outcome for patients and physicians. Retrospective cancer recurrence data can evaluate recurrence-directed treatment and generate novel interventions targeting recurrent cancers. However, large cancer databases do not provide recurrence-related information, stymying study at scale, and consequently require significant manual record review. Automated evaluation of records may allow for the rapid generation of easily analyzed data sets, accelerating the evaluation of recurrence altogether. METHODS Patients treated with radiation therapy at one tertiary referral center from 2010 to 2018 with a verified cancer status (cancer recurrence v no cancer recurrence) were identified. Patients with recurrent disease were initially identified through manual record review, and the associated pathology report was collected. Google Automated Machine Learning with Natural Language Processing (AutoNLP) and Google Gemini 1.5 Pro were used to generate a model for binary classification, with comparison to the gold-standard manually developed data set. RESULTS A total of 7,054 patients were identified. 3,431 (48.6%) were female, with a median age of 64 years. Head and neck (1,482, 21%), breast (1,480, 21%), upper GI (1,307, 18.5%), and lung/thorax (1,107, 15.7%) were the most common disease sites. Recurrence was verified for 1,546 patients (21.9%) using pathology reports, of which 1,249 positive cases were paired with 651 negative pathology reports for model development. Google Gemini 1.5 Pro consistently outperformed AutoNLP across all measurements of accuracy, generating a greater absolute difference in precision, recall, negative predictive value, and specificity, and a higher likelihood of correct classification at the individual level, rendering Gemini superior in recurrence status extraction. CONCLUSION AutoNLP and Google Gemini 1.5 Pro are promising tools for identifying recurrence from pathology reports, with the latter demonstrating superior overall performance, making it particularly suitable for clinical translation.

357 Background: Prostate cancer (PCa) is the most incident cancer in adult males in the United States, and the second highest in the world. CTCAE is the standardized method to grade the severity of treatment-related adverse events (AEs), but are tedious to collect and subject to inter-observer variability. This study aimed to evaluate the performance of LLMs in automatically extracting and grading CTCAE toxicities from clinical notes and patient-reported outcomes of PCa patients from a clinical trial. Methods: A total of 55 patients from NCT02874014 trial, undergoing proton therapy (PT) for localized PCa, were included. The ground truth of CTCAE graded toxicities was obtained from the trial’s results. Clinical notes within 3 days, and the most recent patient questionnaires within 90 days of toxicity assessment were retrieved from the electronic health record. A comprehensive prompt was developed to extract and CTCAE grade toxicities from the retrieved records. LLMs used for this analysis were OpenAI GPT-4o and Gemini 2.0 Flash. Phase 1 evaluated both LLMs' performance against the ground truth. To ensure the metrics reflected the models’ performance based only on the information available in the text provided, cases where both LLMs contradicted the ground truth were manually reviewed. Phase 2 recalculated LLMs' performance metrics using this adapted ground truth. A conservative stepwise ensemble model was developed to leverage the complementary strengths of both LLMs. Results: In the conservative stepwise ensemble model, Gemini 2.0 Flash served as the initial screening platform to identify AEs in step 1. All cases with identified toxicity events were then reviewed and confirmed using OpenAI GPT-4o in step 2. This effectively reduced false positive rates while preserving high sensitivity. The ensemble model achieved strong performance metrics: binary accuracy (0.954), sensitivity (0.961), specificity (0.953), F1 score (0.836), and grade accuracy (0.932). Conclusions: This study demonstrates the feasibility of the use of LLMs for extraction and CTCAE toxicity grading in PCa patients treated with RT. The ensemble model achieved strong performance metrics. Further validation is needed for other cancer diagnosis and treatment modalities. Phase Model Binary Accuracy Precision Sensitivity Specificity F1 Score Grade Accuracy 1 Gemini 0.946 0.672 1.000 0.939 0.804 0.919 1 OpenAI 0.940 0.785 0.637 0.978 0.703 0.895 2 Gemini 0.956 0.735 1.000 0.950 0.848 0.927 2 OpenAI 0.954 0.917 0.679 0.991 0.780 0.903

Abstract Background Partial breast irradiation (PBI) is an effective treatment for early-stage breast cancer. However, evidence comparing different PBI modalities is limited. Methods This trial emulation uses electronic health records from a multi-state large tertiary health system. Three PBI modalities were compared: proton radiation therapy, photon radiation therapy, and applicator-based brachytherapy. Inverse probability weighting (IPW) was used to balance confounders. The primary outcome was ipsilateral breast recurrence (IBR). Results Between 2013 and 2023, 1041 women with early-stage breast cancer were treated with PBI. Of these, 1026 patients were included in the analyses. The median age was 66 years; 22.51% had ductal carcinoma in situ (DCIS), and 97.56% were estrogen receptor positive. With median follow-up of 38.2 months, the rate of IBR in the overall group was 3.05% (95% confidence interval [CI] = 1.99% to 4.65%) at 3 years. After IPW, compared with photon PBI, the hazard ratio for IBR for proton PBI was 2.11 (95% CI = 0.70 to 6.33) and for brachytherapy, 3.86 (95% CI = 1.39 to 10.69). Compared with proton PBI, the hazard ratio for IBR for brachytherapy was 1.83 (95% CI = 0.88 to 3.81). IBR risk was similar across PBI modalities for patients with tumor size ≤ 10 mm and in patients treated for DCIS. Among patients with tumor size >10 mm, the hazard ratio for IBR for brachytherapy was 7.64 (95% CI = 1.64 to 35.58) and 3.59 (95% CI = 1.22 to 10.56) relative to photons and protons, respectively. Conclusion PBI with applicator-based brachytherapy was associated with higher IBR than photon or proton PBI in patients with tumor size > 10 mm, suggesting the need for more careful PBI patient selection for this modality.

Background: Approximately 75% of all head and neck cancer patients are treated with radiotherapy (RT). RT to the oral cavity results in acute and late adverse events which can be severe and detrimental to a patient’s quality of life and function. The purpose of this study was to explore associations between RT dose to a defined oral cavity organ-at-risk (OAR) avoidance structure, provider- and patient-reported outcomes (PROs), opioid use, and hospitalization. Methods: This was a retrospective analysis of prospectively obtained outcomes using multivariable modeling. The study included 196 patients treated with RT involving the oral cavity for a head and neck tumor. A defined oral cavity OAR avoidance structure was used in all patients for RT treatment planning. Validated PROs were collected prospectively. Opioid use and hospitalization were abstracted electronically from medical records. Results: Multivariable modeling revealed the mean dose to the oral cavity OAR was significantly associated with opioid use (p = 0.0082) and hospitalization (p = 0.0356) during and within 30 days of completing RT. Conclusions: The findings of this study may be valuable in RT treatment planning for patients with tumors of the head and neck region to reduce the need for opioid use and hospitalization during treatment.

In this study, we investigated whether radiomics features from pre-treatment positron emission tomography (PET) images could be used to predict disease progression in patients with HPV-positive oropharyngeal cancer treated with definitive proton or x-ray radiotherapy. Machine learning models were built using a dataset from Mayo Clinic, Rochester, Minnesota (n = 72) and tested on a dataset from Mayo Clinic, Phoenix, Arizona (n = 22). A total of 71 clinical and radiomics features were considered. The Mann–Whitney U test was used to identify the top 2 clinical and top 20 radiomics features that were significantly different between progression and progression-free patients. Two dimensionality reduction methods were used to define two feature sets (manually filtered or machine-driven). A forward feature selection scheme was conducted on each feature set to build models of increased complexity (number of input features from 1 to 6) and evaluate model robustness and overfitting. The machine-driven features had superior performance and were less prone to overfitting compared to the manually filtered features. The four-variable Gaussian Naïve Bayes model using the ‘Radiation Type’ clinical feature and three machine-driven features achieved a training accuracy of 79% and testing accuracy of 77%. These results demonstrate that radiomics features can provide risk stratification beyond HPV-status to formulate individualized treatment and follow-up strategies.

AbstractBackgroundThis study was aimed at developing and validating a decision‐making tool predictive of overall survival (OS) for patients receiving stereotactic body radiation therapy (SBRT) for spinal metastases.MethodsThree hundred sixty‐one patients at one institution were used for the training set, and 182 at a second institution were used for external validation. Treatments most commonly involved one or three fractions of spine SBRT. Exclusion criteria included proton therapy and benign histologies.ResultsThe final model consisted of the following variables and scores: Spinal Instability Neoplastic Score (SINS) ≥ 6 (1), time from primary diagnosis < 21 months (1), Eastern Cooperative Oncology Group (ECOG) performance status = 1 (1) or ECOG performance status > 1 (2), and >1 organ system involved (1). Each variable was an independent predictor of OS (p < .001), and each 1‐point increase in the score was associated with a hazard ratio of 2.01 (95% confidence interval [CI], 1.79–2.25; p < .0001). The concordance value was 0.75 (95% CI, 0.71–0.78). The scores were discretized into three groups—favorable (score = 0–1), intermediate (score = 2), and poor survival (score = 3–5)—with 2‐year OS rates of 84% (95% CI, 79%–90%), 46% (95% CI, 36%–59%), and 21% (95% CI, 14%–32%), respectively (p < .0001 for each). In the external validation set (182 patients), the score was also predictive of OS (p < .0001). Increasing SINS<zaq;6> was predictive of decreased OS as a continuous variable (p < .0001).ConclusionsThis novel score is proposed as a decision‐making tool to help to optimize patient selection for spine SBRT. SINS may be an independent predictor of OS.

AbstractRealizing the potential of user‐developed automation software interacting with a treatment planning system (TPS) requires rigorous testing to ensure patient safety and data integrity. We developed an automated test platform to allow comparison of the treatment planning database before and after the execution of a write‐enabled script interacting with a commercial TPS (Eclipse, Varian Medical Systems, Palo Alto, CA) using the vendor‐provided Eclipse Scripting Application Programming Interface (ESAPI). The C#‐application known as Write‐Enable Script Testing Engine (WESTE) serializes the treatment planning objects (Patient, Structure Set, PlanSetup) accessible through ESAPI, and then compares the serialization acquired before and after the execution of the script being tested, documenting identified differences to highlight the changes made to the treatment planning data. The first two uses of WESTE demonstrated that the testing platform could acquire and analyze the data quickly (<4 s per test case) and facilitate the clinical implementation of write‐enabled scripts.

AbstractPurposeTo investigate the dependency of dose‐volume histogram (DVH) behavior and precision on underlying discretization using shapes and dose distributions with known analytical DVHs for five commercial DVH calculators.MethodsDVHs and summary metrics were extracted from all five systems using synthetic cone and cylinder objects for which the true volume and DVH curves were known. Trends in the curves and metrics were explored by varying the underlying voxelization of the CT image, structure set, and dose grid as well by varying the geometry of the structure and direction of a linear dose gradient. Using synthetic structures allowed for comparison with ground truth DVH curves to assess their accuracy while an algorithm was additionally developed to assess the precision of each system. The precision was calculated with a novel algorithm that treats any “stair step” behavior in a DVH curve as an uncertainty band and calculates the width, characterized as a percent difference, of the band for various DVH metrics. The underlying voxelization was additionally changed and DVHs were extracted for two clinical examples. The details of how each system calculated DVHs were also investigated and tendencies in the calculated curves, metrics, and precision were related to choices made in the calculation methodology.ResultsCalculation methodology differences that had a noticeable impact on the DVH curves and summary metrics include supersampling beyond the input grids and interpretation of the superior and inferior ends of the structures. Among the systems studied, the median precision ranged from 0.902% to 3.22%, and interquartile ranges varied from 1.09% to 3.91%.ConclusionsCommercial dose‐evaluation solutions can calculate different DVH curves, structure volume measures, and dose statistics for the same input data due to differences in their calculation methodologies. This study highlights the importance of understanding and investigating the DVH calculation when considering a new clinical system and when using more than one system for data transfer.