Leibovich, Bradley C

Background: Large language models (LLMs) are increasingly used in patient-facing clinical applications, creating a need for scalable methods to evaluate the accuracy, safety, and appropriateness of their responses. Although physician review remains the reference standard, it is resource-intensive and difficult to scale. Objective: To develop and clinically validate an LLM-as-a-judge framework for evaluating responses to patients’ periprocedural questions. Methods: We retrospectively evaluated 1285 patient–AIVA interactions from 112 patients across two Mayo Clinic sites and multiple surgical specialties. Physician reviewers classified interactions using a predefined true-positive (TP), false-negative (FN), true-negative (TN), and false-positive (FP) framework. The finalized LLM judge independently evaluated the same interactions. Agreement was assessed using four-class and category-specific agreement, Cohen’s kappa, and a secondary binary analysis of response correctness. Results: Overall, four-class agreement was 93.1% (95% CI, 91.7–94.5%), with Cohen’s κ = 0.852 (95% CI, 0.823–0.882). Category-specific agreement was 94.1% for TP, 90.7% for FN, 95.5% for TN, and 84.9% for FP classifications. In the binary correctness analysis, accuracy was 93.4% (95% CI, 92.0–94.7%), sensitivity 94.6%, specificity 90.2%, precision 96.2%, and F1-score 0.954. McNemar’s test demonstrated no significant asymmetry between paired classifications (p = 0.11). Conclusions: A clinically grounded LLM judge closely approximated physician evaluation of patient-facing AI responses. These findings support its potential as a scalable assistive monitoring tool while preserving physician oversight for uncertain or safety-sensitive interactions.

Abstract Background The mechanisms underlying metastatic latency in renal cell carcinoma (RCC) remain poorly understood. Methods This study evaluated 2 large independent cohorts for differences in tumor biology between patients who developed metastases early (≤1 year after nephrectomy) and those with late onset (>3 years). Results In the discovery cohort (n = 161), late metastatic RCC was associated with clear cell histology (88.9% vs 78.7%), lower pathological stage (pT1-2; 40.3% vs 18.0%), and favorable histopathological features including low grade (40.0% vs 2.3%), less sarcomatoid (5.6% vs 21.8%), and reduced necrosis (37.7% vs 78.3%; all P < .02). Late metastatic RCC tumors exhibited increased angiogenesis (63.5% vs 19.4%) and reduced inflammation (78.8% vs 50.0%; all P < .02) profiles. Genomic driver analyses revealed comparable rates of PBRM1 and SETD2 loss in late and early metastatic RCC, while BAP1 loss was significantly less common in late metastatic RCC (7.5% vs 27.1%; P < .02). In multivariable models, BAP1/PBRM1/SETD2 status and tumor necrosis emerged as key discriminators of late metastatic RCCs. These findings were confirmed in the second cohort (n = 307). Late metastatic RCC was enriched for fatty acid oxidation and angiogenesis pathways, supporting a less aggressive phenotype. This was further evidenced by a lower engraftment rate in murine models (0% vs 36.5%; P < .001) and significantly longer overall survival from the time of metastasis (median survival doubled, P < .001). Interestingly, late metastatic RCC shared genomic and phenotypic features with RCC that metastasizes to the pancreas, suggesting a common underlying biology influencing both metastatic latency and pancreatic tropism. Conclusions Overall, these findings advocate for recognition of late metastatic RCC because of its distinct biology and improved prognosis.

Accurate identification of surgical instruments is crucial for efficient workflows and patient safety within the operating room, particularly in preventing complications such as retained surgical instruments. Artificial Intelligence (AI) models have shown the potential to automate this process. This study evaluates the accuracy of publicly available Large Language Models (LLMs)—ChatGPT-4, ChatGPT-4o, and Gemini—and a specialized commercial mobile application, Surgical-Instrument Directory (SID 2.0), in identifying surgical instruments from images. The study utilized a dataset of 92 high-resolution images of 25 surgical instruments (retractors, forceps, scissors, and trocars) photographed from multiple angles. Model performance was evaluated using accuracy, weighted precision, recall, and F1 score. ChatGPT-4o exhibited the highest accuracy (89.1%) in categorizing instruments (e.g., scissors, forceps). SID 2.0 (77.2%) and ChatGPT-4 (76.1%) achieved comparable accuracy, while Gemini (44.6%) demonstrated lower accuracy in this task. For precise subtype identification of instrument names (like “Mayo scissors” or “Kelly forceps”), all models had low accuracy, with SID 2.0 having an accuracy of 39.1%, followed by ChatGPT-4o (33.69%). Subgroup analysis revealed ChatGPT-4 and 4o recognized trocars in all instances. Similarly, Gemini identified surgical scissors in all instances. In conclusion, publicly available LLMs can reliably identify surgical instruments at the category level, with ChatGPT-4o demonstrating an overall edge. However, precise subtype identification remains a challenge for all models. These findings highlight the potential of AI-driven solutions to enhance surgical-instrument management and underscore the need for further refinements to improve accuracy and support patient safety.

Significance Statement Nephrosclerosis (glomerulosclerosis, interstitial fibrosis, and tubular atrophy) is the defining pathology of both kidney aging and CKD. Optimal thresholds for nephrosclerosis that identify persons with a progressive disease are unknown. This study determined a young-age threshold (18–29 years) and age-based 95th percentile thresholds for nephrosclerosis on the basis of morphometry of kidney biopsy sections from normotensive living kidney donors. These thresholds were 7.1-fold to 36-fold higher in older (70 years or older) versus younger (aged 18–29 years) normotensive donors. Age-based thresholds, but not young-age threshold, were prognostic for determining risk of progressive CKD among patients who underwent a radical nephrectomy or a for-cause native kidney biopsy, suggesting that age-based thresholds are more useful than a single young-age threshold for identifying CKD on biopsy. Background Nephrosclerosis, defined by globally sclerotic glomeruli (GSG) and interstitial fibrosis and tubular atrophy (IFTA), is a pathology of both kidney aging and CKD. A comparison of risk of progressive CKD using aged-based thresholds for nephrosclerosis versus a single young-adult threshold is needed. Methods We conducted morphometric analyses of kidney biopsy images for %GSG, %IFTA, and IFTA foci density among 3020 living kidney donors, 1363 patients with kidney tumor, and 314 patients with native kidney disease. Using normotensive donors, we defined young-age thresholds (18–29 years) and age-based (roughly by decade) 95th percentile thresholds. We compared age-adjusted risk of progressive CKD (kidney failure or 40% decline in eGFR) between nephrosclerosis that was “normal compared with young,” “normal for age but abnormal compared with young,” and “abnormal for age” in patients with tumor and patients with kidney disease. Results The 95th percentiles in the youngest group (18–29 years) to the oldest group (70 years or older) ranged from 1.7% to 16% for %GSG, 0.18% to 6.5% for %IFTA, and 8.2 to 59.3 per cm 2 for IFTA foci density. Risk of progressive CKD did not differ between persons with nephrosclerosis “normal compared with young” versus “normal for age but abnormal compared with young.” Risk of progressive CKD was significantly higher with %GSG, %IFTA, or IFTA foci density that was abnormal versus normal for age in both cohorts. Conclusions Given that increased risk of progressive CKD occurs only when nephrosclerosis is abnormal for age, age-based thresholds for nephrosclerosis seem to be better than a single young-age threshold for identifying clinically relevant CKD.

Abstract The intrinsic and acquired resistance to PD-1/PD-L1 immune checkpoint blockade is an important challenge for patients and clinicians because no reliable tool has been developed to predict individualized response to immunotherapy. In this study, we demonstrate the translational relevance of an ex vivo functional assay that measures the tumor cell killing ability of patient-derived CD8 T and NK cells (referred to as “cytotoxic lymphocytes,” or CLs) isolated from the peripheral blood of patients with renal cell carcinoma. Patient-derived PBMCs were isolated before and after nephrectomy from patients with renal cell carcinoma. We compared the efficacy of U.S. Food and Drug Administration (FDA)-approved PD-1/PD-L1 inhibitors (pembrolizumab, nivolumab, atezolizumab) and a newly developed PD-L1 inhibitor (H1A Ab) in eliciting cytotoxic function. CL activity was improved at 3 mo after radical nephrectomy compared with baseline, and it was associated with higher circulating levels of tumor-reactive effector CD8 T cells (CD11ahighCX3CR1+GZMB+). Treatment of PBMCs with FDA-approved PD-1/PD-L1 inhibitors enhanced tumor cell killing activity of CLs, but a differential response was observed at the individual-patient level. H1A demonstrated superior efficacy in promoting CL activity compared with FDA-approved PD-1/PD-L1 inhibitors. PBMC immunophenotyping by mass cytometry revealed enrichment of effector CD8 T and NK cells in H1A-treated PBMCs and immunosuppressive regulatory T cells in atezolizumab-treated samples. Our study lays the ground for future investigation of the therapeutic value of H1A as a next-generation immune checkpoint inhibitor and the potential of measuring CTL activity in PBMCs as a tool to predict individual response to immune checkpoint inhibitors in patients with advanced renal cell carcinoma.