Leibovich, Bradley C.

Background: Large language models (LLMs) are increasingly used in patient-facing clinical applications, creating a need for scalable methods to evaluate the accuracy, safety, and appropriateness of their responses. Although physician review remains the reference standard, it is resource-intensive and difficult to scale. Objective: To develop and clinically validate an LLM-as-a-judge framework for evaluating responses to patients’ periprocedural questions. Methods: We retrospectively evaluated 1285 patient–AIVA interactions from 112 patients across two Mayo Clinic sites and multiple surgical specialties. Physician reviewers classified interactions using a predefined true-positive (TP), false-negative (FN), true-negative (TN), and false-positive (FP) framework. The finalized LLM judge independently evaluated the same interactions. Agreement was assessed using four-class and category-specific agreement, Cohen’s kappa, and a secondary binary analysis of response correctness. Results: Overall, four-class agreement was 93.1% (95% CI, 91.7–94.5%), with Cohen’s κ = 0.852 (95% CI, 0.823–0.882). Category-specific agreement was 94.1% for TP, 90.7% for FN, 95.5% for TN, and 84.9% for FP classifications. In the binary correctness analysis, accuracy was 93.4% (95% CI, 92.0–94.7%), sensitivity 94.6%, specificity 90.2%, precision 96.2%, and F1-score 0.954. McNemar’s test demonstrated no significant asymmetry between paired classifications (p = 0.11). Conclusions: A clinically grounded LLM judge closely approximated physician evaluation of patient-facing AI responses. These findings support its potential as a scalable assistive monitoring tool while preserving physician oversight for uncertain or safety-sensitive interactions.

496 Background: Approximately 10% of patients with renal cell carcinoma (RCC) have an associated venous tumor thrombus (VTT), the presence of which impacts surgical management and increases perioperative risk. Historical systemic therapy regimens had limited success in downsizing VTT; however, the impact of immune checkpoint inhibitors (ICIs) on VTT shrinkage is unknown. Our primary objective is to assess VTT shrinkage comparing responses to ICI versus non-ICI regimens. Methods: This multicenter retrospective cohort study included patients with RCC and VTT from 2006-2024 who received upfront systemic therapy. Patients were evaluated at tertiary referral centers across Mayo Clinic Rochester, Florida, and Arizona. Systemic therapies were categorized as ICI versus non-ICI regimens. VTT length was quantified by distance relative to the superior border of the renal vein ostium. The primary outcome was change in VTT length on serial imaging as measured by absolute length reduction (cm), RECIST version 1.1 criteria, and Mayo VTT level reclassification. Linear mixed-effect models (LMMs) identified associations with VTT shrinkage and clinicopathologic features. Results: A total of 108 patients with median age 65 years (IQR 57-71) were included, of whom 49 (45%) received ICI regimens. ICI regimens were associated with greater VTT shrinkage (1cm vs 0.1 cm, p=0.025), increased RECIST responses (51% vs 32%, p=0.046), and a higher rate downstaging per Mayo VTT level (33% vs 12%, p=0.008) compared to non-ICI regimens. On multivariable LMM analysis, ICI regimens remained more likely to result in VTT shrinkage compared to non-ICI regimens (-0.12 cm/month, 95% CI –0.24-0.00, p=0.046). Absence of abdominal lymphadenopathy was also associated with VTT shrinkage (-0.18cm/month, 95% CI -0.06- -0.30, p=0.004). Initial Mayo VTT level, IMDC risk, and number of metastatic sites were not associated with VTT shrinkage. Conclusions: Treatment with ICI-regimens resulted in VTT shrinkage and Mayo VTT level reduction. This is a particularly important area of exploration as radical nephrectomy and tumor thrombectomy carry up to a 40% risk of peri-operative major complications. These data support using ICI regimens for patients with RCC with VTT who are not candidates for upfront surgery and prioritizing ICI regimens in neoadjuvant clinical trials.

This cohort study examines the oncological, neurological, and serological profiles of adult male patients with testicular germ cell tumors and paraneoplastic neurologic syndromes.

Abstract Purpose: Germ cell tumors (GCTs) pose significant diagnostic challenges due to the limited sensitivity and specificity of current tumor markers. Hereby we utilize phage immunoprecipitation sequencing (PhIP-Seq) technique to develop a unique immunosignature panel (iSIGN) with high sensitivity and specificity to improve the diagnosis and differentiation of GCTs. Patients and Methods: Serum samples from patients with confirmed gonadal or extragonadal GCTs and controls were collected from Mayo Clinic repositories. Controls included benign testicular lesions, autoimmune diseases, neurodegenerative disorders, and other cancers. Samples were randomized into development and validation cohorts. PhIP-Seq identified antigen-autoantibody interactions, and enrichment scores were calculated to develop a biomarker panel distinguishing GCT from non-GCT cases. A second biomarker panel was developed to predict seminoma from nonseminoma cases. Results: A total of 427 serum samples were analyzed, including 150 from GCT patients and 277 controls. For our primary model to distinguish GCT cases (GCT-iSIGN), 24 peptides from 16 antigens were identified based on significant differential autoantibody binding enrichment in GCT patients (p < 0.05), ≥2 peptides per antigen, and enrichment scores ≥ 2.5-fold. GCT-iSIGN was highly sensitive (93%) and specific (99%) for identifying GCT cases from controls with an area under curve (AUC) of 0.93 and 9% Unknown Case Rate (UCR). The secondary panel (Sem-iSIGN) included 17 peptides corresponding to 5 unique proteins with high specificity (93%), moderate sensitivity (65%) and AUC of 0.77 to distinguish seminoma from nonseminoma GCT cases. Conclusion: This study demonstrates the potential use of PhIP-Seq to identify unique immunosignature panels that can serve as reliable biomarkers for GCT. These panels address the limitations of traditional markers and other proteomic techniques. This immunosignature model represents a cost-effective, stable, and scalable approach to enhancing GCT diagnosis and management. Citation Format: M Bakri Hammami, Andrew Knight, Haidara Kherbek, Brian Costello, Silvana De Lorenzo, Bradley C. Leibovich, John C. Cheville, Yong Guo, Jessica Sagen, Janet E. Olson, Alicia Algeciras-Schimnich, Sean J. Pittock, John R. Mills, Surendra Dasari, Divyanshu Dubey. iSIGN panel identifies germ cell tumors via whole-proteome phage immunoprecipitation sequencing [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Mechanisms of Cancer Immunity and Cancer-related Autoimmunity; 2025 Sep 24-27; Montreal, QC, Canada. Philadelphia (PA): AACR; Cancer Immunol Res 2025;13(9 Suppl):Abstract nr B027.

Abstract Background The mechanisms underlying metastatic latency in renal cell carcinoma (RCC) remain poorly understood. Methods This study evaluated 2 large independent cohorts for differences in tumor biology between patients who developed metastases early (≤1 year after nephrectomy) and those with late onset (>3 years). Results In the discovery cohort (n = 161), late metastatic RCC was associated with clear cell histology (88.9% vs 78.7%), lower pathological stage (pT1-2; 40.3% vs 18.0%), and favorable histopathological features including low grade (40.0% vs 2.3%), less sarcomatoid (5.6% vs 21.8%), and reduced necrosis (37.7% vs 78.3%; all P < .02). Late metastatic RCC tumors exhibited increased angiogenesis (63.5% vs 19.4%) and reduced inflammation (78.8% vs 50.0%; all P < .02) profiles. Genomic driver analyses revealed comparable rates of PBRM1 and SETD2 loss in late and early metastatic RCC, while BAP1 loss was significantly less common in late metastatic RCC (7.5% vs 27.1%; P < .02). In multivariable models, BAP1/PBRM1/SETD2 status and tumor necrosis emerged as key discriminators of late metastatic RCCs. These findings were confirmed in the second cohort (n = 307). Late metastatic RCC was enriched for fatty acid oxidation and angiogenesis pathways, supporting a less aggressive phenotype. This was further evidenced by a lower engraftment rate in murine models (0% vs 36.5%; P < .001) and significantly longer overall survival from the time of metastasis (median survival doubled, P < .001). Interestingly, late metastatic RCC shared genomic and phenotypic features with RCC that metastasizes to the pancreas, suggesting a common underlying biology influencing both metastatic latency and pancreatic tropism. Conclusions Overall, these findings advocate for recognition of late metastatic RCC because of its distinct biology and improved prognosis.

Accurate identification of surgical instruments is crucial for efficient workflows and patient safety within the operating room, particularly in preventing complications such as retained surgical instruments. Artificial Intelligence (AI) models have shown the potential to automate this process. This study evaluates the accuracy of publicly available Large Language Models (LLMs)—ChatGPT-4, ChatGPT-4o, and Gemini—and a specialized commercial mobile application, Surgical-Instrument Directory (SID 2.0), in identifying surgical instruments from images. The study utilized a dataset of 92 high-resolution images of 25 surgical instruments (retractors, forceps, scissors, and trocars) photographed from multiple angles. Model performance was evaluated using accuracy, weighted precision, recall, and F1 score. ChatGPT-4o exhibited the highest accuracy (89.1%) in categorizing instruments (e.g., scissors, forceps). SID 2.0 (77.2%) and ChatGPT-4 (76.1%) achieved comparable accuracy, while Gemini (44.6%) demonstrated lower accuracy in this task. For precise subtype identification of instrument names (like “Mayo scissors” or “Kelly forceps”), all models had low accuracy, with SID 2.0 having an accuracy of 39.1%, followed by ChatGPT-4o (33.69%). Subgroup analysis revealed ChatGPT-4 and 4o recognized trocars in all instances. Similarly, Gemini identified surgical scissors in all instances. In conclusion, publicly available LLMs can reliably identify surgical instruments at the category level, with ChatGPT-4o demonstrating an overall edge. However, precise subtype identification remains a challenge for all models. These findings highlight the potential of AI-driven solutions to enhance surgical-instrument management and underscore the need for further refinements to improve accuracy and support patient safety.