Citations

Background: Large language models (LLMs) are increasingly used in patient-facing clinical applications, creating a need for scalable methods to evaluate the accuracy, safety, and appropriateness of their responses. Although physician review remains the reference standard, it is resource-intensive and difficult to scale. Objective: To develop and clinically validate an LLM-as-a-judge framework for evaluating responses to patients’ periprocedural questions. Methods: We retrospectively evaluated 1285 patient–AIVA interactions from 112 patients across two Mayo Clinic sites and multiple surgical specialties. Physician reviewers classified interactions using a predefined true-positive (TP), false-negative (FN), true-negative (TN), and false-positive (FP) framework. The finalized LLM judge independently evaluated the same interactions. Agreement was assessed using four-class and category-specific agreement, Cohen’s kappa, and a secondary binary analysis of response correctness. Results: Overall, four-class agreement was 93.1% (95% CI, 91.7–94.5%), with Cohen’s κ = 0.852 (95% CI, 0.823–0.882). Category-specific agreement was 94.1% for TP, 90.7% for FN, 95.5% for TN, and 84.9% for FP classifications. In the binary correctness analysis, accuracy was 93.4% (95% CI, 92.0–94.7%), sensitivity 94.6%, specificity 90.2%, precision 96.2%, and F1-score 0.954. McNemar’s test demonstrated no significant asymmetry between paired classifications (p = 0.11). Conclusions: A clinically grounded LLM judge closely approximated physician evaluation of patient-facing AI responses. These findings support its potential as a scalable assistive monitoring tool while preserving physician oversight for uncertain or safety-sensitive interactions.

Introduction: Large language models (LLMs) match or exceed physician accuracy on benchmark diagnostic tasks. Whether higher accuracy reflects human-like diagnostic behavior, which bears on how safely clinicians can supervise these systems, is rarely assessed. Objectives: To determine whether diagnostic accuracy and alignment with human reader responses move together across widely accessible LLMs under default, point-of-care conditions. Methods: This cross-sectional evaluation submitted the 29 most recent New England Journal of Medicine (NEJM) case challenges to 5 publicly accessible LLMs (GPT-5, Claude, Google Gemini, DeepSeek, and Grok) through each model’s web interface (November 5-8, 2025) at default settings, one query per case, no system prompt. Each case offered a clinical vignette and 6 candidate diagnoses; published human reader response distributions (4386-17 714 per case; median, 7732) were the comparator. Diagnostic accuracy against the published gold-standard diagnosis, compared with the human reader majority by the exact McNemar test; agreement by Cohen κ; alignment by a human-error similarity index (the share of model errors on cases the reader majority also missed) and by the Jensen-Shannon divergence (JSD) between each model’s response and the reader distribution; and ordinary least squares regression of per-case JSD on normalized reader-disagreement entropy. Results: Reader majority accuracy was 48.3% (95% CI, 31.4-65.6). All 5 reached or exceeded this value, from 51.7% for DeepSeek (95% CI, 34.4-68.6) to 82.8% for Grok (95% CI, 65.5-92.4); Grok (P = .02), GPT-5 (P = .004), and Gemini (P = .04) significantly exceeded the reader majority, whereas Claude and DeepSeek did not. Grok, the most accurate model, showed the weakest alignment (similarity index, 0.40; mean JSD, 0.50 bits), with 3 of 5 errors on cases readers answered correctly. GPT-5 showed the strongest alignment (index, 1.000), failing only on cases that also challenged readers. For 4 of 5 models, divergence rose with reader-disagreement entropy; DeepSeek was the exception. Conclusions and Relevance: In this evaluation, diagnostic accuracy and alignment with human reader responses diverged across LLMs. Because a model can outperform clinicians in aggregate yet fail where oversight is least likely to detect it, evaluation of clinical AI should weigh error structure alongside accuracy.

Background: Reliable access to postoperative guidance between clinic visits remains inconsistent, and the usefulness of large language models depends on how clinical knowledge is structured and retrieved. Knowledge graph-only (KG-only), vector retrieval-augmented generation (RAG-only), and hybrid graph-retrieval (GraphRAG) architectures have not been directly compared under controlled conditions for postoperative care. Objective: It was to compare these three pipelines on a curated postoperative question-answering benchmark using automated grounding metrics and blinded clinician review. Methods: A benchmark of 250 expert-authored questions (200 in-scope, 20 emergency-intent, 30 out-of-scope) was evaluated on an identical knowledge corpus using Gemini 2.5 Flash for generation and Gemini Embedding-001 for retrieval. Three blinded clinicians rated all 750 outputs; the primary outcome was majority-consensus accuracy across all 250 questions. GraphRAG is a multi-component pipeline that also includes query expansion, a second retrieval stage, evidence fusion, and a monotonic RAG fallback, and is therefore evaluated as a complete hybrid system rather than as graph retrieval in isolation. Results: Overall accuracy was 96.8% (95% CI: 93.8–98.4) for GraphRAG, 94.8% (95% CI: 91.3–96.9) for RAG-only, and 56.0% (95% CI: 49.8–62.0) for KG-only. Both retrieval-augmented systems outperformed KG-only (adjusted p < 0.001). The GraphRAG versus RAG-only difference was 2.0 percentage points, arose from five discordant question pairs and did not reach the Bonferroni-adjusted threshold (exact McNemar p = 0.062; adjusted p = 0.188). The monotonic RAG fallback determined 44.8% of GraphRAG outputs (112 of 250 queries). GraphRAG showed higher context recall (0.851 versus 0.796; adjusted p < 0.001), whereas RAG-only showed higher context precision. Estimated hallucination risk, derived from an LLM-as-a-judge procedure using Gemini 2.5 Flash rather than independent clinical adjudication, was 0.108, 0.132, and 0.436, respectively; the GraphRAG versus RAG-only difference was not significant (adjusted p = 0.771). Inter-rater agreement was high (Fleiss’ kappa = 0.854). Conclusions: Retrieval augmentation, rather than the choice of retrieval architecture, accounted for most of the difference observed. The incremental advantage of GraphRAG over a well-implemented RAG-only system was small, was not statistically significant after correction and reflects the complete hybrid system rather than graph augmentation alone. Emergency-intent accuracy was 75.0% for both retrieval-augmented systems; the architecture is therefore suited to supervised clinical workflows with clinician review, not autonomous patient-facing triage. Prospective validation on authentic patient messages and formal component ablations are required.

Introduction: Frontotemporal lobar degeneration (FTLD) is a devastating disease that commonly results in early onset dementia, yet its rarity and heterogeneity limit large-scale research and clinical trials. Remote monitoring via low-burden digital health tools may overcome these barriers. We evaluated clinical utility of passive smartphone monitoring in FTLD using a multi-domain mobile assessment platform. Methods: Participants were 567 adults (53% clinically normal, 21% prodromal FTLD, 26% symptomatic FTLD) who completed an in-person study visit and downloaded the ALLFTD Mobile App on their personal smartphones. The app delivered unsupervised cognitive tests and passively collected continuous data on battery percentage (proxy for smartphone use) and step count. Longitudinal follow-up included smartphone monitoring and annual in-person study visits. Primary analyses examined associations between passive smartphone features and markers of disease severity at baseline and longitudinally, and tested whether passive features added incremental value beyond app-based cognitive testing. Results: Passive smartphone features were feasible to collect and showed excellent reliability with <2 weeks of monitoring (ICCs>0.9). Features capturing smartphone use and movement demonstrated sensitivity to gold-standard clinical measures of disease severity both at baseline and longitudinally. A classification model combining passive features, the app-based cognitive composite score, and demographics detected longitudinal functional decline (AUC=0.89); passive features alone (AUC=0.84) performed comparably to a cognitive screener (AUC=0.85). Discussion: Passive smartphone monitoring is a valid, zero-burden measure of disease severity and progression in FTLD that adds modest incremental value beyond app-based cognitive testing. Findings support its use as a scalable digital endpoint in longitudinal FTLD research.

Importance The incidence of human papillomavirus type 16 (HPV16–positive) recurrent/metastatic (R/M) head and neck squamous cell carcinoma (HNSCC) is rising. Approved therapies offer a durable response to a minority of patients. Objective To determine the efficacy and safety of a therapeutic HPV16-targeted immunotherapy, PDS0101, in combination with pembrolizumab in participants with HPV16-positive R/M HNSCC, with separate evaluation of immune checkpoint inhibitor (ICI)–naive and ICI-resistant cohorts. Design, Setting, and Participants VERSATILE-002 was a phase 2, open-label, multicenter nonrandomized clinical trial conducted at 26 oncology centers from March 29, 2021, to May 15, 2025, across the US, UK, and Ireland using the Simon 2-stage optimal design. Efficacy analyses were performed on modified intent-to-treat (mITT) population; safety analyses included all participants who received 1 or more doses. Eighty-eight participants with histologically confirmed R/M HPV16-positive HNSCC were enrolled, of whom 87 (98.9%) received treatment and composed the safety population. Seventy five (85.2%) were in the mITT population used for efficacy analyses, of whom 53 (70.7%) were ICI-naive with a programmed cell death ligand 1 combined positive score of 1 or greater and 22 (29.3%) were ICI-resistant. Exposures Pembrolizumab (200 mg, intravenously, every 3 weeks for up to 35 cycles) plus subcutaneous PDS0101 (5 scheduled doses per protocol). Main Outcomes and Measures The primary end point was objective response rate (ORR) per the Response Evaluation Criteria in Solid Tumors , version 1.1, by a masked independent central reviewer. Secondary end points included progression-free survival (PFS), median overall survival (mOS), and safety, with exploratory end points of duration of response, disease control rate, and response rates by programmed cell death ligand 1 status. Results Of the 75 patients in the mITT population, the mean (SD) age was 64.0 (8.3) years; 4 individuals (5%) were female and 71 (95%) were male; and 2 individuals (3%) were Asian, 1 (1%) was Black, 5 (7%) were Hispanic, and 71 (95%) were White. Among 53 ICI-naive participants in the mITT cohort, ORR by the central reviewer was 34.0%, median PFS was 5.3 months (95% CI, 2.10-9.00), and mOS was 39.3 months (95% CI, 23.90 to not evaluable). Among 22 ICI-resistant patients in the mITT cohort, no objective responses were observed and the primary end point (ORR by central reviewer) was not met, but mOS was 14.8 months (95% CI, 8.50-26.00). The treatment was well tolerated, with 13.8% experiencing grade 3 or higher treatment-related adverse events. Conclusions and Relevance The results of this nonrandomized clinical trial suggest that PDS0101 plus pembrolizumab showed antitumor activity in ICI-naive HPV16-positive R/M HNSCC, with favorable safety, response rates, and survival outcomes. The findings support further evaluation in a first-line setting. Trial Registration ClinicalTrials.gov Identifier: NCT04260126

Surgeons are at high risk of developing long-term Work-Related Musculoskeletal Disorders (WMSDs) due to awkward sustained postures. Passive spine-supporting exoskeletons are a potential intervention to improve posture and reduce fatigue. The NekSpine ® was evaluated by 21 surgeons (15 male) across two hospital sites within a quaternary academic hospital system. Each surgeon performed at least two comparable procedures, one with the exoskeleton and one without. Surgeons completed pre- and postoperative surveys assessing NASA-TLX workload, usability, body-part discomfort, overall fatigue, and workflow disruptions. Eighty-two cases were evaluated (43 exoskeleton cases). Compared with baseline cases, surgeons reported reduced neck and upper back discomfort and lower overall fatigue when using the exoskeleton. The NekSpine ® exoskeleton may reduce spinal discomfort and fatigue during surgery, supporting its potential as an ergonomic intervention for surgeons.

Abstract Background and Objectives Cryoprecipitated antihemophilic factor (cryoprecipitate), fibrinogen concentrate (FC), and pathogen‐reduced cryoprecipitated fibrinogen complex (PR cryoprecipitate) are sources of fibrinogen replacement, and each has advantages and disadvantages in terms of cost, efficacy, safety, and logistics of administration. This study examines the respective utilization frequencies of these different products at institutions in North America and the factors influencing the decision on which one(s) to adopt. Materials and Methods A survey was distributed via email to contacts for Advancement of Blood and Biotherapies (AABB)‐accredited blood banks/transfusion services. Survey questions covered topics including the availability and usage of FC, cryoprecipitate, and PR cryoprecipitate, as well as the perceived advantages of each product. Results A total of 53 responses were received from surveyed institutions (estimated 25% response rate). Over half (31/53, 58.5%) of respondents exclusively used cryoprecipitate, 30.2% (16/53) used two sources of fibrinogen replacement, and 7.5% (4/53) used all three (cryoprecipitate, FC, and PR cryoprecipitate). More than half of the institutions restricted FC to specific indications, primarily for cost containment. The most commonly cited advantages of FC over cryoprecipitate were storage convenience and rapid reconstitution. Conclusion Surveyed institutions in the United States and Canada primarily use cryoprecipitate for fibrinogen replacement, with variable usage of FC and limited usage of PR cryoprecipitate. The perception of FC being more expensive than cryoprecipitate predominated. Further studies comparing available fibrinogen replacement options would be valuable, particularly with consideration of product acquisition cost, wastage, impacts on utilization of other blood products, and patient outcomes.

Objective To assess the effect of Moses™ 2.0 (Lumenis Be Ltd., Yokneam Illit, Israel; Boston Scientific, Marlborough, MA, USA) for lithotripsy of renal stones of 8–20 mm in diameter on operative time, surgeon workload, and retropulsion. Subjects/Patients and Methods In our multicentre, randomised, double‐blind clinical trial, patients were randomised 1:1 to standard or Moses 2.0 laser lithotripsy and stratified by stone size. Patients and surgeons were blinded to laser modality. Primary outcomes were total operative time as well as surgeon workload assessed by the National Aeronautics and Space Administration‐Task Load Index (NASA‐TLX). Secondary outcomes were stone retropulsion evaluated by blinded video review and postoperative clinical outcomes. Linear regression modelling was used for analysis. Results A total of 153 patients (79 Standard, 74 Moses 2.0) with similar clinical, stone, and operative characteristics were evaluated. There was no significant difference in operative times between the Standard and Moses 2.0 groups ( P = 0.73). Moses 2.0 was associated with lower effort, frustration, mental, physical, and temporal workload, as well as higher rated overall performance compared to the standard laser. Video review demonstrated no difference in rates of retropulsion ( P = 0.13). Complication rates (Standard: 13%; Moses 2.0: 10%) and stone‐free rates (Standard: 51%; Moses: 2.0 55%) were comparable. Conclusion Our study evaluated the impact of Moses 2.0 technology on high‐powered renal stone dusting, constituting the largest double‐blind trial of its kind to date. There was insufficient evidence to suggest any difference in operative times or clinical outcomes when using Moses 2.0 compared to standard laser lithotripsy. Moses 2.0 was associated with reduced surgeon workload, which has implicit impact on surgeon wellness.

Abstract The management of diverticular disease has undergone a major evolution over the past two decades, shifting away from episode-count triggers for surgery toward individualized strategies informed by patient-reported outcomes, recurrence risk, and high-quality randomized clinical trials. Pivotal decision analyses and randomized trials have reshaped the surgical treatment landscape, while parallel trials of antibiotic therapy have transformed the nonoperative management of uncomplicated disease. In this study, we provide a comprehensive synthesis of historical perspectives and modern evidence regarding the surgical and nonoperative management of diverticulitis, summarize key randomized trials across the disease spectrum, describe the evolution of operative techniques, and translate these findings into actionable guidance, including a patient-centered decision aid. A structured narrative review of pivotal trials, major guidelines, and multicenter randomized studies were performed, encompassing trials of antibiotic therapy for uncomplicated diverticulitis, laparoscopic lavage for perforated disease, operative strategy for complicated disease, elective resection versus conservative therapy for recurrent or persistent symptoms, and the evolution of minimally invasive and robotic surgical techniques. Trial design, eligibility, interventions, outcomes, and limitations were extracted and synthesized. The traditional “two-episode rule” has been supplanted by individualized, symptom- and quality of life (QoL)-driven decision-making. Routine antibiotic therapy for uncomplicated diverticulitis is not supported by randomized evidence and is now reserved for select patients. Laparoscopic lavage, initially promising for perforated diverticulitis, demonstrates variable outcomes and is now reserved for selected patients with Hinchey III disease; when resection is required, primary anastomosis with diverting ileostomy is increasingly favored over Hartmann's procedure because of markedly higher stoma reversal rates. Randomized trials of elective sigmoid resection show reduced recurrence and improved QoL for patients with persistent symptoms, though not universally across all patient groups, and minimally invasive-particularly robotic-surgical techniques continue to reduce perioperative morbidity. Contemporary management of diverticulitis requires individualized evaluation anchored in patient-reported outcomes, disease phenotype, comorbidity, and shared decision-making. This study synthesizes the current evidence and provides clinic-ready decision aid for patient discussions.