Santoro, Domenico

Introduction: Large language models (LLMs) match or exceed physician accuracy on benchmark diagnostic tasks. Whether higher accuracy reflects human-like diagnostic behavior, which bears on how safely clinicians can supervise these systems, is rarely assessed. Objectives: To determine whether diagnostic accuracy and alignment with human reader responses move together across widely accessible LLMs under default, point-of-care conditions. Methods: This cross-sectional evaluation submitted the 29 most recent New England Journal of Medicine (NEJM) case challenges to 5 publicly accessible LLMs (GPT-5, Claude, Google Gemini, DeepSeek, and Grok) through each model’s web interface (November 5-8, 2025) at default settings, one query per case, no system prompt. Each case offered a clinical vignette and 6 candidate diagnoses; published human reader response distributions (4386-17 714 per case; median, 7732) were the comparator. Diagnostic accuracy against the published gold-standard diagnosis, compared with the human reader majority by the exact McNemar test; agreement by Cohen κ; alignment by a human-error similarity index (the share of model errors on cases the reader majority also missed) and by the Jensen-Shannon divergence (JSD) between each model’s response and the reader distribution; and ordinary least squares regression of per-case JSD on normalized reader-disagreement entropy. Results: Reader majority accuracy was 48.3% (95% CI, 31.4-65.6). All 5 reached or exceeded this value, from 51.7% for DeepSeek (95% CI, 34.4-68.6) to 82.8% for Grok (95% CI, 65.5-92.4); Grok (P = .02), GPT-5 (P = .004), and Gemini (P = .04) significantly exceeded the reader majority, whereas Claude and DeepSeek did not. Grok, the most accurate model, showed the weakest alignment (similarity index, 0.40; mean JSD, 0.50 bits), with 3 of 5 errors on cases readers answered correctly. GPT-5 showed the strongest alignment (index, 1.000), failing only on cases that also challenged readers. For 4 of 5 models, divergence rose with reader-disagreement entropy; DeepSeek was the exception. Conclusions and Relevance: In this evaluation, diagnostic accuracy and alignment with human reader responses diverged across LLMs. Because a model can outperform clinicians in aggregate yet fail where oversight is least likely to detect it, evaluation of clinical AI should weigh error structure alongside accuracy.

Background Nephritis is a common manifestation of IgA vasculitis and is morphologically indistinguishable from IgA nephropathy. While MEST-C scores are predictive of kidney outcomes in IgA nephropathy, their value in IgA vasculitis nephritis has not been investigated in large multiethnic cohorts. Methods Biopsies from 262 children and 99 adults with IgA vasculitis nephritis ( N =361) from 23 centers in North America, Europe, and Asia were independently scored by three pathologists. MEST-C scores were assessed for correlation with eGFR/proteinuria at biopsy. Because most patients ( N =309, 86%) received immunosuppression, risk factors for outcomes were evaluated in this group using latent class mixed models to identify classes of eGFR trajectories over a median follow-up of 2.7 years (interquartile range, 1.2–5.1). Clinical and histologic parameters associated with each class were determined using logistic regression. Results M, E, T, and C scores were correlated with either eGFR or proteinuria at biopsy. Two classes were identified by latent class mixed model, one with initial improvement in eGFR followed by a late decline (class 1, N =91) and another with stable eGFR (class 2, N =218). Class 1 was associated with a higher risk of an established kidney outcome (time to ≥30% decline in eGFR or kidney failure; hazard ratio, 5.84; 95% confidence interval, 2.37 to 14.4). Among MEST-C scores, only E1 was associated with class 1 by multivariable analysis. Other factors associated with class 1 were age 18 years and younger, male sex, lower eGFR at biopsy, and extrarenal noncutaneous disease. Fibrous crescents without active changes were associated with class 2. Conclusions Kidney outcome in patients with biopsied IgA vasculitis nephritis treated with immunosuppression was determined by clinical risk factors and endocapillary hypercellularity (E1) and fibrous crescents, which are features that are not part of the International Study of Diseases of Children classification.