Citations

Abstract Purpose Large language models (LLMs) have proven performance for certain diagnostic tasks; however, limited studies have evaluated their consistency in recommending appropriate medication regimens for a given diagnosis. Medication management is a complex task that requires synthesis of drug formulation and complete order instructions for safe use. Here, the performance of GPT-4o, an LLM available with OpenAI’s ChatGPT, was tested on 3 medication management tasks. Methods GPT-4o performance was tested on 3 medication tasks: identifying available formulations for a given generic drug name, identifying drug-drug interactions (DDIs) for a given medication regimen, and preparing a medication order for a given generic drug name. For each experiment, the model’s raw text response was captured exactly as returned and evaluated using clinician evaluation in addition to standard LLM metrics, including Term Frequency-Inverse Document Frequency (TF-IDF) vectors, normalized Levenshtein similarity, and Recall-Oriented Understudy for Gisting Evaluation (ROUGE-1/ROUGE-L) F1 score between each response and its reference string. Results For the first task of drug-formulation matching, GPT-4o had 49% accuracy for generic medications being matched to all available formulations, with an average of 1.23 omissions per medication and 1.14 hallucinations per medication. For the second task of drug-drug interaction identification, the accuracy was 54.7% for identifying the DDI pair. For the third task, GPT-4o generated order sentences containing no medication or abbreviation errors in 65.8% of the cases. Conclusion Model performance for basic medication tasks was consistently poor. This evaluation highlights the need for domain-specific training through clinician-annotated datasets and a comprehensive evaluation framework for benchmarking performance.

Abstract Background The accuracy and safety of generating medication orders by large language models (LLMs) must be demonstrated. Without standardization, performance evaluation is limited to time and resource-intensive clinician grading. This evaluation aimed to develop a standardized medication format that supports automated performance evaluation (MedMatch). Methods First, a survey of 40 medication prompts was given to clinicians to assess agreement in medication order communication. Second, a clinician panel developed a standardized medication format (MedMatch) for oral and intravenous medications. Third, a clinician-annotated dataset of medication prompts and standardized answers in the MedMatch format was developed for LLM testing. Finally, LLMs were retested with the same dataset, adjusted to exclude route information, to evaluate the appropriate categorization of medication route. Results The formal medication orders consistently showed low omission rates and high overlap for all entities, compared to the verbal and brief written communication types. Lexical overlap results demonstrated pattern norms amongst clinicians with entities appearing most commonly in positions 1-5 in the order of drug name, dose, unit, route, and frequency. In the second survey, the formal written group performed the highest with 78.3% of prompts considered appropriate as a computer-generated response. LLM accuracy on MedMatch order standardization was highest in oral solid (64.2-72.5%), intravenous intermittent (72.5-84.3%), and intravenous push (62.7-74.5%) categories. LLMs performed the worst at categorizing medication orders accurately into intravenous push (18-61%) and intravenous intermittent (51-100%) routes. Conclusions Standardized format for computer-based outputs may support automated performance analysis and enhance the clarity of medication communication.

Abstract Background Large language models (LLMs) are increasingly used in medication-related tasks despite limited evidence supporting their accuracy and safety. LLMs are vulnerable to adversarial attacks and may confabulate in response to fabricated information in the prompt. Given the substantial risk of harm from medication errors, the purpose of this study was to establish a benchmark task to evaluate LLM performance in distinguishing fabricated from real medications. Methods Two datasets (brand and generic), each consisting of 250 medication lists were developed, each including 4-6 medications and one fabricated medication (specifically a Pokémon character), with complete dosing information provided. Six LLMs were evaluated (GPT-5-Chat, GPT-4o-mini, DrugGPT, Gemma-3-27B-IT, Llama-3.3-70B-Instruct, and Qwen3-32B). For each medication list, models were queried for dosing information and disease indication across three experimental conditions: standard decoding, standard decoding with mitigation prompting, and deterministic decoding (temperature = 0). Each experiment was performed in triplicate. Outputs were deemed confabulations if the LLM failed to recognize and reject the fabricated medication. The primary outcome was the rate of confabulations. Exact paired-permutation tests were used to compare confabulation rates across LLMs and prompting approaches. Results Confabulation rates under standard conditions ranged from 2.7% to 99.6%. DrugGPT demonstrated the lowest baseline confabulation rates (2.7-6.4%), whereas other models demonstrated substantially higher rates (47.2-99.6%). Mitigation prompting significantly reduced confabulation rates compared to standard and deterministic decoding across all tasks and datasets (p < 0.001): GPT-5-Chat achieved confabulation rates of 0% with mitigation prompting from baseline rates of 66.4-88.5%. Deterministic decoding did not substantially impact confabulation rates. Conclusions LLMs performing medication-related tasks are susceptible to confabulation in response to invalid or fabricated inputs, raising important safety concerns for clinical use. While mitigation prompting reduced confabulation rates, this study underscores the importance of robust safeguards when deploying LLMs in medication-related decision-making.

Abstract Background Outpatient parenteral antimicrobial therapy (OPAT) is effective and cost-efficient, but 12%–63% of patients experience adverse events, requiring close monitoring. Therapeutic drug monitoring (TDM) is routinely used for vancomycin and aminoglycosides but is underutilized for beta-lactams (BLs) due to limited access, logistical barriers, and clinician inexperience. This study evaluated its use and clinical impact within a high-volume OPAT program. Methods This retrospective, single-center cohort included adult patients (≥18 years) enrolled in an OPAT program who received cefepime (CEF), piperacillin/tazobactam (TZP), or meropenem (MEN) and underwent BL TDM between April 2023 and April 2025. BL TDM was not protocolized within the OPAT program; however, institutional guidance supported its use in carefully selected patients. We aimed to describe TDM indications, lab trends, microbiological profiles, and the impact of TDM on treatment. Results During the study period, 25 OPAT patients underwent BL TDM. TDM was performed in patients receiving CEF (64%), TZP (20%), and MEN (16%). The most common indication for TDM was suspected beta-lactam toxicity (32%) (Figure 1). Post-TDM therapy adjustments were made in 52% (n = 13), including changes in dosing frequency (38.5%) and dosage strength (30.8%). Additional changes included infusion strategy modification or agent substitution (15.4% each). Subtherapeutic levels were the most common reason for adjustment (46.2%), followed by adverse effects, supratherapeutic levels, and changes in renal function (23.1% each). Skin, soft tissue, and musculoskeletal infections were most common (Figure 2). Positive cultures were present in 80%, with Pseudomonas spp most frequently isolated. Therapy was completed in 17 patients; 7 discontinued early, and 1 had treatment ongoing. Reasons for therapy discontinuation are detailed in Table 1. Conclusion BL TDM is contributing to a paradigm shift in OPAT practice by improving both efficacy and safety. While inpatient TDM typically targets efficacy, safety concerns more commonly motivate TDM in the OPAT setting. Although this review involved a small cohort, our experience adds to the growing body of evidence supporting BL TDM as a tool to enhance patient outcomes in the OPAT setting. Disclosures All Authors: No reported disclosures

Abstract Background Beta-lactam pharmacokinetics can be significantly altered in patients with malignancy. Therapeutic drug monitoring (TDM; i.e., drug level testing) may facilitate pharmacokinetic/pharmacodynamic (PK/PD) target attainment, but the data are limited. This study aimed to characterize PK/PD target attainment of anti-pseudomonal beta-lactams in hematology/oncology patients.Table 1:Case DescriptionGraph 1:Distribution of Beta-Lactam TDM Indications Methods We conducted a retrospective review of beta-lactam TDM performed between June 2022 and June 2024 in hospitalized adults with hematologic or solid organ malignancies. We included patients who underwent beta-lactam TDM during clinical care, typically for extracorporeal membrane oxygenation (ECMO), extremes of body weight, suspected augmented renal clearance, multidrug-resistant infection, or suspected dose-dependent beta-lactam toxicity. Primary outcomes were (1) the proportion achieving target beta-lactam concentrations and (2) the proportion undergoing antibiotic dose adjustment based on TDM results.Table 2:Institutional Dosing Guidelines Results There were 47 TDM episodes in 20 unique patients. Considering only the first TDM per patient, 10 (50%) patients achieved target concentrations, 2 (10%) were below target, and 6 (30%) were above target. Antibiotic dose adjustments were made in 8 (40%) patients in response to TDM results. The most common indication for beta-lactam TDM was concerns regarding accurate renal clearance, followed by extremes of weight. Conclusion Retrospective findings suggest that beta-lactam TDM facilitated individualization of therapy in 40% of patients with either hematological or solid organ malignancy, with dose adjustments often intended to improve antibiotic safety. Future studies should refine the criteria for beta-lactam TDM use in this population and determine the need for tailored PK/PD targets in patients with impaired host response to infection. Disclosures All Authors: No reported disclosures