Celi, Leo Anthony

Abstract Background Large language models (LLMs) are increasingly used in medication-related tasks despite limited evidence supporting their accuracy and safety. LLMs are vulnerable to adversarial attacks and may confabulate in response to fabricated information in the prompt. Given the substantial risk of harm from medication errors, the purpose of this study was to establish a benchmark task to evaluate LLM performance in distinguishing fabricated from real medications. Methods Two datasets (brand and generic), each consisting of 250 medication lists were developed, each including 4-6 medications and one fabricated medication (specifically a Pokémon character), with complete dosing information provided. Six LLMs were evaluated (GPT-5-Chat, GPT-4o-mini, DrugGPT, Gemma-3-27B-IT, Llama-3.3-70B-Instruct, and Qwen3-32B). For each medication list, models were queried for dosing information and disease indication across three experimental conditions: standard decoding, standard decoding with mitigation prompting, and deterministic decoding (temperature = 0). Each experiment was performed in triplicate. Outputs were deemed confabulations if the LLM failed to recognize and reject the fabricated medication. The primary outcome was the rate of confabulations. Exact paired-permutation tests were used to compare confabulation rates across LLMs and prompting approaches. Results Confabulation rates under standard conditions ranged from 2.7% to 99.6%. DrugGPT demonstrated the lowest baseline confabulation rates (2.7-6.4%), whereas other models demonstrated substantially higher rates (47.2-99.6%). Mitigation prompting significantly reduced confabulation rates compared to standard and deterministic decoding across all tasks and datasets (p < 0.001): GPT-5-Chat achieved confabulation rates of 0% with mitigation prompting from baseline rates of 66.4-88.5%. Deterministic decoding did not substantially impact confabulation rates. Conclusions LLMs performing medication-related tasks are susceptible to confabulation in response to invalid or fabricated inputs, raising important safety concerns for clinical use. While mitigation prompting reduced confabulation rates, this study underscores the importance of robust safeguards when deploying LLMs in medication-related decision-making.

*Nature Reviews Nephrology* welcomes you to this webinar, in which our two speakers — Leo Celi and Kianoush Kashani — will present on the theme of Digital Determinants of Health, with a focus on artificial intelligence (AI) and the need for equity-focused innovation. The webinar will be held on 14th of March 2024 — World Kidney Day — which this year focuses on “Advancing equitable access to care and optimal medication practice”. In their [Comment](https://www.nature.com/articles/s41581-023-00763-4.epdf?sharing_token=ihZn7Tzp1ixBfK7a3X_6x9RgN0jAjWel9jnR3ZoTv0NaWQBzIFRPdPKT7AS3IVVHez49nwrLVjZJyLsCUt5034qu70IjW6yS7U3gUuTEQjm0O_u1j1PiMZLzRs7qKAyY_28RvtE3kXcXmwKeyufyUyKP715Wx73c6p6Bfp6w6Sg%3D) “Digital determinants of health: opportunities and risks amidst health inequities”, Leo Celi and colleagues argue that given the current digital revolution, digital tools and AI applications in health care must be developed with equity at the forefront to avoid perpetuating historical biases and inequities. With their [Consensus Statement](https://www.nature.com/articles/s41581-023-00744-7) on “Digital health and acute kidney injury”, Kianoush Kashani and colleagues highlight some of the potential applications of digital health tools in nephrology, as well as the need for equitable digitization. Artificial Intelligence: The digital determinant of health to rule them all Artificial intelligence (AI) is poised to be the biggest digital determinant of health. From healthcare, to education, to law enforcement, AI can theoretically augment every human task that involves thinking. We will see every decision-making infiltrated by AI. This is why AI is different. Most technologies are much narrower in scope, and therefore, easier to regulate. In addition, the data that we use to build AI reflects everything about the systems we would like to disrupt, both good and bad. Behind the façade of AI lies our legacy systems with all their flaws. The biggest threat to AI delivering its promise stems from the risk of cementing the structural inequities that permeate every aspect of society. AI provides an opportunity to inch us closer to a better care delivery system, but it will require us to design an equity-focused sociotechnical ecosystem. Engineering AI in a vacuum that does not build capacity nor include cultural transformation of how we learn and how we work together will be a waste of time and money, and a huge opportunity cost. To avert the disaster from AI, we need to diversify those sitting at the table. We need to diversify those who are developing and deploying AI. Digital health in acute kidney injury management Acute kidney injury (AKI) is a highly prevalent clinical syndrome that substantially impacts patient outcomes. It is accepted by the clinical communities that the management of AKI is time sensitive. Unfortunately, despite growing proof of its preventability, AKI management remains suboptimal in community, acute care, and post-acute care settings. Digital health solutions comprise various tools and models to improve care processes and patient outcomes in multiple medical fields. AKI development, progression, recovery, or lack thereof, offers tremendous opportunities for developing, validating, and implementing digital health solutions in multiple settings.

Abstract Introduction Delirium occurrence is common and preventive strategies are resource intensive. Screening tools can prioritize patients at risk. Using machine learning, we can capture time and treatment effects that pose a challenge to delirium prediction. We aim to develop a delirium prediction model that can be used as a screening tool. Methods From the eICU Collaborative Research Database (eICU-CRD) and the Medical Information Mart for Intensive Care version III (MIMIC-III) database, patients with one or more Confusion Assessment Method-Intensive Care Unit (CAM-ICU) values and intensive care unit (ICU) length of stay greater than 24 h were included in our study. We validated our model using 21 quantitative clinical parameters and assessed performance across a range of observation and prediction windows, using different thresholds and applied interpretation techniques. We evaluate our models based on stratified repeated cross-validation using 3 algorithms, namely Logistic Regression, Random Forest, and Bidirectional Long Short-Term Memory (BiLSTM). BiLSTM represents an evolution from recurrent neural network-based Long Short-Term Memory, and with a backward input, preserves information from both past and future. Model performance is measured using Area Under Receiver Operating Characteristic, Area Under Precision Recall Curve, Recall, Precision (Positive Predictive Value), and Negative Predictive Value metrics. Results We evaluated our results on 16 546 patients (47% female) and 6294 patients (44% female) from eICU-CRD and MIMIC-III databases, respectively. Performance was best in BiLSTM models where, precision and recall changed from 37.52% (95% confidence interval [CI], 36.00%–39.05%) to 17.45 (95% CI, 15.83%–19.08%) and 86.1% (95% CI, 82.49%–89.71%) to 75.58% (95% CI, 68.33%–82.83%), respectively as prediction window increased from 12 to 96 h. After optimizing for higher recall, precision and recall changed from 26.96% (95% CI, 24.99%–28.94%) to 11.34% (95% CI, 10.71%–11.98%) and 93.73% (95% CI, 93.1%–94.37%) to 92.57% (95% CI, 88.19%–96.95%), respectively. Comparable results were obtained in the MIMIC-III cohort. Conclusions Our model performed comparably to contemporary models using fewer variables. Using techniques like sliding windows, modification of threshold to augment recall and feature ranking for interpretability, we addressed shortcomings of current models.

Abstract Background Optimization of mechanical ventilation (MV) in patients with cardiac arrest (CA) may help improve outcomes in these patients. We sought to investigate the association between hyperoxia, PCO2, and positive end-expiratory pressure (PEEP) with mortality in patients with CA. Methods Patients admitted to our medical center CICU from 2001 through 2012 (MIMIC-III database) who received MV with available information on MV parameters and had arterial blood gases sampling were included. Hyperoxia was defined as time-weighted mean of PaO2 &gt;120 mmHg and non-hyperoxia as PaO2 ≤120 mmHg, while Hypercarbia was defined as PCO2 &gt;35 mmHg during CICU admission. The primary outcome was in-hospital mortality. Multivariable logistic regression was used to assess the association between hyperoxia and in-hospital mortality adjusted for age, female sex, Oxford Acute Severity of Illness Score, creatinine, lactate, pH, PaO2/FiO2 ratio, PCO2, PEEP, and time spent on PEEP. Results Among 136 patients, PaO2 = 139±55 mmHg, PCO2 = 39±10 mmHg, and PEEP = 6.4±2.2cmH2O. Unadjusted mortality was higher in the hyperoxic group (51.4%) compared to the non-hyperoxic group (29.0%) (long rank test p=0.0034, figure). In multivariable analysis, hyperoxia was independently associated with higher in-hospital mortality (OR 4.046, 95% CI: 1.501–10.907, p=0.0057). Additionally, there was no association between the presence of hypercarbia and in-hospital mortality (OR 0.896, 95% CI: 0.319 to 2.521, p=0.836) nor when PCO2 was analyzed as a continuous variable (OR 1.063 per 1 mmHg increase in CO2, 95% CI: 0.111–10.145, p=0.957). Similarly, there was no assocation between PEEP and in-hospital mortality (OR 1.012 per 1cmH2O increase, 95% CI: 0.807 to 1.270, p=0.917). Post-hoc analysis with PaO2 as a continuous variable was consistent with the primary analysis (OR 1.214 per 10 mmHg increase in PaO2, 95% CI: 1.059–1.391, p=0.005). Conclusions In patients with CA, hyperoxia was associated with increased mortality, while PCO2 and PEEP levels were not. Optimal MV parameters are important in the management of patients with CA. Further research is warranted to confirm this association and explore the mechanisms behind these observations. These studies can help establish the best MV strategies for patients with CA. Funding Acknowledgement Type of funding source: None