Gorenshtein, Alon

Abstract Most clinical large language model (LLM) benchmarks rely on clean, concise vignettes that do not reflect the noisy, long-form documentation typical of real clinical records. How LLM performance degrades under realistic chart conditions remains poorly characterised. Here we test whether structured retrieval workflows protect National Institutes of Health Stroke Scale (NIHSS) scoring accuracy under systematic context stress. Using 100 de-identified acute stroke cases and a fully crossed 4 x 4 x 3 x 3 condition matrix (144 conditions per case), we vary context acquisition method, document length, distractor load and critical-information position across four Gemini models (57,047 retained runs). Structured retrieval reduces mean absolute error (MAE) from 4.58 to 2.96 points relative to non-agentic baselines (mean gain 1.62 MAE points; 95% CI 1.57–1.67; 35% relative reduction), with consistent gains across all 36 stress combinations. Lower-cost models show disproportionately larger gains (2.76 versus 0.45 MAE points). Tool-retrieved pipelines outperform retrieval-augmented generation in 33 of 36 combinations. These findings indicate that retrieval architecture, rather than model scale alone, is a tractable lever for robust, equitable clinical LLM deployment.

Introduction: We aimed to prove integration of advanced machine learning methods within a robust ensemble framework can enhance clinical decision-support for neurologists managing patients in the emergency department (ED). Methods: We engineered an ensemble framework leveraging the capabilities of the Gemini 1.5-pro-002 large language model (LLM). The model was enhanced using prompt engineering and retrieval-augmented generation (RAG). Predictive modeling achieved by combining eXtreme Gradient Boosting (XGBoost) and logistic regression for optimal accuracy in clinical decision-making. Key clinical outcomes, such as admission and mortality, were assessed. A random subset of 100 cases was reviewed by three senior neurologists to evaluate the alignment of the AI’s predictions with expert clinical judgment. Results: We retrospectively analyzed 1368 consecutive ED patients who underwent neurological consultations, assessing their clinical features, diagnostic tests, and admission outcomes. Patients admitted were typically older and had higher mortality rates, shorter intervals to neurological evaluation, and a higher incidence of acute stroke compared to those discharged. For the primary analysis (n = 250), the Neuro artificial intelligence (AI) model demonstrated significant performance metrics, achieving an area under the curve (AUC) of 0.88 for general admission predictions in comparison to actual outcomes, an AUC of 0.86 for neurological department admissions, 0.93 for long-term mortality risk, and 1 for 48 h mortality risk. Our Neuro AI model predictions showed a strong correlation with expert consensus (Pearson correlation 0.79, p < 0.001), indicating its ability to provide consistent support amid divergent clinical opinions. Conclusions: Our Neuro AI model accurately predicted hospital admissions (AUC = 0.88) and neurological department admissions (AUC = 0.86), demonstrating strong alignment with expert clinical judgment.