Citations

Abstract Background Preparing tumor board patient summaries is time intensive. Large-language-model based systems may automate summarization but require real-world evaluation prior to clinical use. We performed an exploratory retrospective evaluation of the Microsoft Healthcare Agent Orchestrator (HAO), deployed in a Mayo Clinic–controlled staged environment, to generate tumor board–style patient summaries from retrospective Electronic Health Record (EHR) notes. Methods HAO generated summaries for breast, hepatobiliary, and neuro-oncology tumor board cases using up to the most recent 1,000 clinical notes. Clinician reviewers evaluated outputs via REDCap surveys across perceived factuality, completeness, clarity/conciseness, temporal cohesion, comparative performance, safety, and clinical utility (0–4 Likert scale). Reviewers were permitted to query the HAO chat interface to address missing details. Automated factuality was assessed using TBFact (bidirectional entailment), reporting precision and recall against available reference summaries. Results Among 57 survey responses from 5 different physicians, mean scores exceeded 2.8 across domains, with medians of 3 for most axes. In an exploratory comparison, oncology fellows required less time to review HAO-generated summaries than to manually generate patient summaries (mean difference 13.57 minutes per patient, p<0.001), although this difference may be influenced by prior familiarity with the same cases; 96% of survey responses indicated that HAO would save time. TBFact evaluations showed higher recall than precision across domains, consistent with broad capture of reference content alongside additional content that was not present in gold-standard summaries. Attribution was viewed favorably but showed issues with primary-source specificity and link reliability. Conclusions In a controlled Mayo environment, HAO demonstrated moderate performance and was associated with reduced review time for tumor board preparation. These findings are promising but preliminary and do not establish clinical safety, noninferiority to manual review, or readiness for routine clinical use. Limitations, including verbosity, specialty-specific content gaps, and inconsistent attribution, highlight the need for iterative refinement and further evaluation. Graphical Abstract Created using ChatGPT

Abstract Scientific foundation models learn high-dimensional representations from diverse data modalities, yet what they encode and how to extract that knowledge remain open questions. Here we show that probing the internal representations of Evo 2, a 7-billion-parameter genomic foundation model, enables accurate and interpretable genetic variant effect prediction. We introduce a covariance-based probe that captures second-order structure from Evo 2 sequence embeddings to predict variant pathogenicity across variant types and functional consequences, matching or exceeding specialized predictors within their domains. To ground these predictions in known biological mechanisms, we train a complementary panel of probes on existing annotations to detect which genomic properties are disrupted by a variant. This categorized evidence is then integrated with each variant’s genomic context through a language model to generate variant-specific mechanistic hypotheses. Our pathogenicity predictions correlate with experimental measures of variant function, clinical penetrance, and biobank disease associations while the mechanistic hypotheses are consistent with expert reviews, known mechanism classes, and downstream molecular readouts. We release pathogenicity scores, disruption profiles, and contextualized interpretations for 4.2 million variants from the ClinVar database as an open resource through the Evo Variant Effect Explorer (EVEE). More broadly, this structured probing approach offers a general framework for interrogating foundation models across scientific disciplines and grounding their outputs in existing domain concepts.

Background Large language models (LLMs) are entering clinical workflows, yet their alignment with medical-ethical principles is unclear. Objective To evaluate how LLMs reconcile deontological and utilitarian reasoning in medically adapted “Trolley dilemma” scenarios. Methods We tested 14 LLMs, including GPT-o1-preview and DeepSeek-R1-Distill-Llama, using five medically adapted Trolley dilemmas sourced from philosophical and real-world scenarios. Each model generated 10 independent answers per case (overall 700 responses). Responses were forced to “yes” (utilitarian) or “no” (deontological). The primary outcome was the proportion of utilitarian choices. χ 2 tests compared models. Results LLM responses varied. Some models, such as GPT-4 and Gemma, selected the utilitarian option in 20% of cases, while others, like Qwen-2-7B, reached 80% ( p < .001). GPT-o1-preview and DeepSeek-R1-Distill-Llama selected utilitarian responses in 44% and 38% of cases, respectively. Some models selected the consequence-maximizing option in vignettes where doing so required overriding consent or endorsing intentional harm within the scenario framing. Seven of the fourteen models tested recommended impermissible medical actions, including nonconsensual limb amputation, killing a healthy person for organ harvesting, and transferring a patient to hospice against their wishes, in 44 of 700 outputs (6.3 %). Conclusions In this constrained forced-choice stress test, models occasionally endorsed boundary-violating actions within the vignette framing. These results do not generalize to routine clinical decision-making, but motivate further benchmarked evaluation of refusal behavior and boundary constraints in ethically sensitive medical contexts. We must carefully define how these models influence clinical decisions, establish robust ethical guidelines for their use, and ensure that fundamental ethical norms are not compromised.

Background Artificial intelligence (AI)-generated text to video is emerging in medical education, but its effectiveness, accuracy and safety remain uncertain. We aimed to synthesise empirical studies evaluating these tools in learner or patient education. Methods A comprehensive search was conducted in MEDLINE/PubMed, Google Scholar, Scopus, Cochrane Review and Web of Science for studies published up to January 2025. Eligible studies evaluated AI-generated text to video for medical or patient education, reporting both quantitative and qualitative outcomes. Two reviewers screened and extracted data. The review adhered to Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. Results Out of 103 identified studies, five studies met the inclusion criteria: four evaluated patient education and one evaluated physician training. Clinical areas were ophthalmology (2/5), plastic surgery (1/5), dysphagia rehabilitation (1/5) and neurosurgical training (1/5). In ophthalmology, control materials outperformed AI-generated videos on image/script accuracy (p<0.005), with similar script-image alignment. In dysphagia rehabilitation, a randomised trial reported improvements in swallowing function and related outcomes with an AI-assisted video game intervention (p<0.001). A plastic surgery study reported greater user preference for a video avatar tool compared with a text chatbot (63.5% vs 28.1%). Across the reviewed studies, samples were small and CIs were rarely reported. Outcome measures were heterogeneous, and meta-analysis was not feasible. Conclusion AI-generated videos can enhance engagement or selected outcomes in certain contexts, yet concerns about accuracy and inconsistent measurement persist. Current evidence is sparse and mixed. Currently, these tools can complement, rather than replace, standard resources until non-inferiority is demonstrated for the prespecified primary outcomes. PROSPERO registration number CRD42025640042.

Abstract Radiological protocol selection is a critical but time-consuming step in clinical workflow, requiring radiologists to match patient indications with the appropriate MRI or CT protocol. Manual selection can be prone to delays or potential errors, and automated approaches must contend with substantial data imbalance, site-specific variation, and evolving nomenclature. We investigated whether a large language model (LLM) can support reliable protocol selection at scale and whether retrieval-augmented generation (RAG) offers operational advantages over direct fine-tuning. Using 498,228 patient reports collected across three Mayo Clinic sites (Arizona, Florida, and Rochester) spanning six radiological divisions, we trained site-specific Llama 3.2 3B models for use with and without retrieval augmentation. Division-scoped Facebook AI Similarity Search (FAISS) indexes constructed from procedure and diagnosis text were used to supply contextual evidence in the RAG framework. Both fine-tuned and RAG-augmented models achieved strong performance across sites, with F1 scores of 0.88–0.90. RAG matched or modestly trailed direct fine-tuning overall but delivered consistent gains in specific divisions (e.g., musculoskeletal imaging). Importantly, the RAG model introduced abstention behavior (Not Enough Information), which concentrated in linguistically diverse divisions and provided an interpretable signal of uncertainty. These findings suggest that RAG-based models are viable for division-scoped protocol selection and offer practical advantages. Retrieval indexes can be refreshed far more easily and with fewer resources than retraining LLMs, enabling continual adaptation to evolving clinical workflows. Future prospective deployment will evaluate real-time accuracy, agreement with practitioners, and the role of abstention as a safety mechanism in clinical decision support.

Large language models (LLMs) have the potential to transform healthcare by assisting in documentation, diagnosis, patient communication, and medical education. However, their integration into clinical practice remains a challenge. This perspective explores the barriers to implementation by synthesizing recent evidence across five challenge domains: workflow misalignment and diagnostic safety, bias and equity, regulatory and legal governance, technical vulnerabilities such as hallucinations or data poisoning, and the preservation of patient trust and human connection. While the perspective focuses on barriers, LLM capabilities and mitigation strategies are advancing rapidly, raising the likelihood of near-term clinical impact. Drawing on recent empirical studies, we propose a framework for understanding the key technical, ethical, and practical challenges associated with deploying LLMs in clinical environments and provide directions for future research, governance, and responsible deployment.

Abstract Objectives Large language models (LLMs) offer potential in natural language processing tasks in healthcare. Due to the need for high accuracy, understanding their limitations is essential. The purpose of this study was to evaluate the performance of LLMs in classifying radiology reports for the presence of pulmonary embolism (PE) under various conditions, including different prompt designs and data perturbations. Materials and Methods In this retrospective, institutional review board approved study, we evaluated 3 Google’s LLMs including Gemini-1.5-Pro, Gemini-1.5-Flash-001, and Gemini-1.5-Flash-002, in classifying 11 999 pulmonary CT angiography radiology reports for PE. Ground truth labels were determined by concordance between a computer vision-based PE detection (CVPED) algorithm and multiple LLM runs under various configurations. Discrepancies between algorithms’ classifications were aggregated and manually reviewed. We evaluated the effects of prompt design, data perturbations, and repeated analyses across geographic cloud regions. Performance metrics were calculated. Results Of 11 999 reports, 1296 (10.8%) were PE-positive. Accuracy across LLMs ranged between 0.953 and 0.996. The highest recall rate for a prompt modified after a review of the misclassified cases (up to 0.997). Few-shot prompting improved recall (up to 0.99), while chain-of-thought generally degraded performance. Gemini-1.5-Flash-002 demonstrated the highest robustness against data perturbations. Geographic cloud region variability was minimal for Gemini-1.5+-Pro, while the Flash models showed stable performance. Discussion and Conclusion LLMs demonstrated high performance in classifying radiology reports, though results varied with prompt design and data quality. These findings underscore the need for systematic evaluation and validation of LLMs for clinical applications, particularly in high-stakes scenarios.

Artificial Intelligence (AI) has emerged as a transformative force in orthopedic oncology, offering significant advances in the diagnosis, classification, and prediction of treatment response for primary malignant bone tumors (PBT). Through machine learning and deep learning techniques, AI leverages computational algorithms and large datasets to enhance medical imaging interpretation and support clinical decision-making. The integration of radiomics with AI enables the extraction of quantitative features from medical images, allowing for precise tumor characterization and the development of personalized therapeutic strategies. Notably, convolutional neural networks have demonstrated exceptional capabilities in pattern recognition, significantly improving tumor detection, segmentation, and differentiation. This narrative review synthesizes the evolving applications of AI in PBTs, focusing on early tumor detection, imaging analysis, therapy response prediction, and histological classification. AI-driven radiomics and predictive models have yielded promising results in assessing chemotherapy efficacy, optimizing preoperative imaging, and predicting treatment outcomes, thereby advancing the field of precision medicine. Innovative segmentation techniques and multimodal imaging models have further enhanced healthcare efficiency by reducing physician workload and improving diagnostic accuracy. Despite these advancements, challenges remain. The rarity of PBTs limits the availability of robust, high-quality datasets for model development and validation, while the lack of standardized imaging protocols complicates reproducibility. Ethical considerations, including data privacy and the interpretability of complex AI algorithms, also warrant careful attention. Future research should prioritize multicenter collaborations, external validation of AI models, and the integration of explainable AI systems into clinical practice. Addressing these challenges will unlock AI’s full potential to revolutionize PBT management, ultimately improving patient outcomes and advancing personalized care.

Objectives: The aims of the study are to develop a prostate cancer risk prediction model that combines clinical and magnetic resonance imaging (MRI)–related findings and to assess the impact of adding Prostate Imaging-Reporting and Data System (PI-RADS) ≥3 lesions-level findings on its diagnostic performance. Methods: This 3-center retrospective study included prostate MRI examinations performed with clinical suspicion of clinically significant prostate cancer (csPCa) between 2018 and 2022. Pathological diagnosis within 1 year after the MRI was used to diagnose csPCa. Seven clinical, 3 patient-level MRI-related, and 4 lesion-level MRI-related findings were extracted. After feature selection, 2 logistic regression models with and without lesions-level findings were created using data from facility I and II (development cohort). The area under the receiver operating characteristic curve (AUC) between the 2 models was compared in the PI-RADS ≥3 population in the development cohort and Facility III (validation cohort) using the Delong test. Interfacility differences of the selected predictive variables were evaluated using the Kruskal-Wallis test or chi-squared test. Results: Selected lesion-level features included the peripheral zone involvement and apparent diffusion coefficient (ADC) values. The model with lesions-level findings had significantly higher AUC than the model without in 655 examinations in the development cohort (0.81 vs 0.79, respectively, P = 0.005), but not in 553 examinations in the validation cohort (0.77 vs 0.76, respectively). Large interfacility differences were seen in the ADC distribution (P < 0.001) and csPCa proportion in PI-RADS 3–5 (P < 0.001). Conclusions: Adding lesions-level findings improved the csPCa discrimination in the development but not the validation cohort. Interfacility differences impeded model generalization, including the distribution of reported ADC values and PI-RADS score-level csPCa proportion.

The performance of a deep learning model was not different from that of radiologists in the detection of clinically significant prostate cancer at MRI, and gradient-weighted class activation maps localized the tumor.