Hallemeier, Chris L.

Abstract Large Language Models (LLMs) demonstrate strong capabilities in healthcare applications. However, recruiting human evaluators to assess usability and effectiveness becomes challenging in the large-scale LLM-generated outcomes. Existing metrics are often outdated for evaluating LLM-based healthcare technologies. This paper proposes a Research through Evaluation (RtE) approach that can refine metrics and improve assessments of LLM-generated outputs. We deploy RtE in a retrospective comparative study of prostate cancer patient inquiries, comparing responses from human clinical care teams with those generated by a GPT-4-based in-basket bot. Through three rounds of co-evaluations with clinical professionals and an LLM-as-graders study with four LLMs, the RtE method shows that starting with small, iterative evaluations helps human graders adjust metrics to suit their daily practices better. Additionally, in the LLM-as-graders study, LLMs exhibit similar behavior, preferring the in-basket bot's responses over the clinical care team. Our methodology and findings demonstrate that the RtE method can enhance the transparency and robustness of evaluation metrics for LLM-generated content, with the potential to generalize to evaluations in other clinical practices.

Objective: To develop and validate a risk score for predicting post-transplant recurrence in perihilar cholangiocarcinoma (pCCA) patients. Background: pCCA is an aggressive malignancy with a poor prognosis. Although neoadjuvant chemoradiation (CRT) followed by liver transplantation (LT) offers a potential cure for patients with unresectable, early-stage de novo or primary sclerosing cholangitis (PSC)-associated pCCA, post-transplant recurrence negatively impacts survival. Currently, no risk stratification tools exist. Methods: This retrospective study included consecutive patients with unresectable de novo or PSC-associated pCCA who underwent neoadjuvant CRT followed by LT at Mayo Clinic (1993–2024). Predictors of post-transplant recurrence were identified using multivariable Cox regression with LASSO variable selection. The Perihilar Cholangiocarcinoma Risk Estimation of Tumor Recurrence after Transplant (PRETREAT) risk score was created by assigning points from model coefficients. Score performance was evaluated by discrimination (c-statistic), internally validated with 10-fold cross-validation, and externally validated in a separate cohort. Survival outcomes were compared across 3 PRETREAT risk groups and between 2 time periods. Results: This study included 399 patients (development: 301; validation: 98). The 5-year cumulative incidence of post-transplant recurrence was 29.7% in the development cohort and 27.6% in the validation cohort. Multivariable analysis identified 4 independent predictors: macroscopic residual tumor on explant [hazard ratio (HR): 12.4], vascular encasement (HR: 2.18), lymphovascular invasion (HR: 2.04), and radial tumor diameter (HR: 1.02/mm). The PRETREAT score (0–22), based on these factors, demonstrated excellent performance in both internal and external validation (C-index 0.83 and 0.85, respectively). Patients were stratified into low-risk (0–7), moderate-risk (8–15), and high-risk (16–22) groups. The 5-year recurrence-free survival rates significantly differed across low-risk, moderate-risk, and high-risk groups (89.0%, 38.3%, and 15.4%, respectively; P<0.001). Over the last decade, overall survival improved significantly only in moderate-risk patients (70.4% vs 46.9%, P=0.024), without significant improvement in recurrence-free survival. Conclusions: The PRETREAT score provides a valuable tool for guiding post-transplant management of pCCA, enabling risk-stratified surveillance, and laying the foundation for future clinical trials.

Abstract Background Anal squamous cell carcinoma is a rare cancer with increased occurrence of multiple cancers before and after the anal squamous cell carcinoma diagnosis. However, there are limited data on this aspect. This multi-institutional analysis aimed to define the occurrence of malignancies before and after anal squamous cell carcinoma, time trends, and impact on survival and to identify prognostic factors. Methods Initial primary malignancy was defined as a malignancy occurring before the anal squamous cell carcinoma. Second primary malignancy was defined as a distinct primary cancer that developed after anal squamous cell carcinoma diagnosis. Retrospective multi-institutional chart review was done. Progression-free survival (PFS), overall survival, and prognostic factors were evaluated. Results A total of 647 patients with anal squamous cell carcinoma treated with curative intent were analyzed. Median age was 61.2 years with 72% as females. Of these, 150 (23.3%) patients had multiple malignancies with initial primary malignancy in 16% and second primary malignancy in 8%. Patients without prior cancer had better 5-year PFS (81.2% vs 67.2%, P = .011) and overall survival (81% vs 69%, P = .008) compared with those with prior cancer. Second primary malignancies had a statistically significant adverse impact on PFS (hazard ratio [HR] = 4.22) and overall survival (HR = 3.56). Females had better 5-year PFS (82% vs 70%, P = .016) as compared with males. The median time interval for developing anal squamous cell carcinoma (as second primary malignancy) after initial primary malignancy was 9.32 years. Conclusions Anal squamous cell carcinoma patients have an increased risk of multiple malignancies. These patients who have prior cancers have inferior outcomes. Second primary malignancy is a poor prognostic factor in patients with anal cancer. Second primary malignancy can develop years after treatment of primary anal squamous cell carcinoma.

Abstract Purpose: This phase Ib/2 trial investigated pembrolizumab-containing trimodality therapy in patients with gastroesophageal junction (GEJ) adenocarcinoma. Patients and Methods: Patients with GEJ adenocarcinoma (cT1–3NanyM0) received neoadjuvant pembrolizumab-containing chemoradiation (CROSS regimen) followed by surgical resection and adjuvant pembrolizumab. The primary endpoints were tolerability in the first 16 patients and pathologic complete response [pCR (ypT0N0)]. Secondary endpoints included progression-free survival (PFS) and overall survival (OS). An independent propensity-score-matched cohort (treated with CROSS without immunotherapy) was used for comparison. Exploratory analyses included immune biomarkers in the tumor microenvironment (TME) and plasma. Results: We enrolled 31 eligible patients, of whom 29 received all expected doses of neoadjuvant pembrolizumab and 28 underwent R0 resection. Safety endpoints were met. The primary efficacy endpoint was not met [7/31 (22.6%) achieved pCR]. Patients with high [i.e., combined positive score (CPS) ≥ 10] baseline expression of programmed death (PD)-L1 in the TME had a significantly higher pCR rate than those with low expression [50.0% (4/8) vs. 13.6% (3/22); P = 0.046]. Patients with high PD-L1 expression also experienced longer PFS and OS than propensity-score-matched patients. Among trial patients with PD-L1 CPS &lt; 10, unprespecified analysis explored whether extracellular vesicles (EV) could identify further responders: an elevated plasma level of PD-L1–expressing EVs was significantly associated with higher pCR. Conclusions: Adding pembrolizumab to trimodality therapy showed acceptable tolerability but did not meet the pre-specified pCR endpoint. Exploratory analyses suggested that high PD-L1 expression in the TME and/or on EVs may identify patients most likely to achieve tumor response.

AbstractObjectivesInhibitors to the checkpoint proteins cytotoxic T‐lymphocyte‐associated protein 4 (CTLA‐4) and programmed cell death protein 1 (PD‐1) are becoming widely used in cancer treatment. However, a lack of understanding of the patient response to treatment limits accurate identification of potential responders to immunotherapy.MethodsIn this study, we assessed the expression of PD‐1 and CTLA‐4 on 19 leucocyte populations in the peripheral blood of 74 cancer patients. A reference data set for PD‐1 and CTLA‐4 was established for 40 healthy volunteers to determine the normal expression patterns for these checkpoint proteins.ResultsUnsupervised hierarchical clustering found four immune profiles shared across the solid tumor types, while chronic lymphocytic leukaemia patients had an immune profile largely unique to them. Furthermore, we measured these leucocyte populations on an additional cohort of 16 cancer patients receiving the PD‐1 inhibitor pembrolizumab in order to identify differences between responders and non‐responders, as well as compared to healthy volunteers (n = 20). We observed that cancer patients had pre‐treatment PD‐1 and CTLA‐4 expression on their leucocyte populations at different levels compared to healthy volunteers and identified two leucocyte populations positive for CTLA‐4 that had not been previously described. We found higher levels of PD‐1+ CD3+ CD4− CD8− cells in patients with progressive disease and have identified it as a potential biomarker of response, as well as identifying other significant differences in phenotypes between responders and non‐responders.ConclusionThese results are suggestive that categorisation of patients based on immune profiles may differentiate responders from non‐responders to immunotherapy for solid tumors.