Frazee, Erin N.

Abstract Background: For large language models (LLMs) to reach their potential as information technology tools that make medication use safer, clinically relevant benchmarks capable of automated grading and designed specifically to measure the performance of LLMs for medication tasks are required. The purpose of this study was to design a suite of benchmarking tests reflective of Comprehensive Medication Management (CMM; the standard of care for medication optimization) and quantify the baseline performance of the latest LLMs. Methods: We established six benchmarks representing critical stages of the CMM process: drug formulation matching, drug order (sig) generation, drug route matching, drug-drug interaction identification, renal dose identification, and drug-indication matching. For each benchmark, we curated a clinician-annotated dataset comprising 250 standardized input-output pairs including both inpatient and outpatient medications. We evaluated the clinical knowledge retrieval capabilities of three LLMs: GPT-4o-mini, MedGemma-27B, and LLaMA3-70B. We employed a zero-shot prompting strategy, excluding in-context examples, to assess the models’ internal clinical knowledge rather than their few-shot learning potential. To check reliability, each model was run three times using a temperature of 0.7 (a mid-range value of an LLM setting controlling text generation randomness). Performance was assessed using task-specific evaluation metrics including precision (positive predictive value), recall (sensitivity), F1-score, accuracy, and correctness consistency across trials. Results: Across six benchmarks, LLaMA3-70B demonstrated the highest performance in four tasks: drug-formulation matching (F1, 54.0% [95 CI: 50.1-58]), drug-order generation (accuracy, 88.0%), drug-route identification (F1, 74.3% [95 CI: 71-78]), and drug-indication identification (accuracy, 97.6% [95 CI: 95.6-99.2]). In the drug–drug interaction task, GPT-4o-mini achieved the highest overall accuracy (70.4% [95 CI: 64.8-75.7]). For renal dose–adjustment identification, GPT-4o-mini demonstrated the highest F1 score (83.3% [95 CI: 77.6-88]). Correctness-consistency scores ranged from 8.0% to 97.6% across benchmarks, with no model exhibiting uniformly superior consistency. Conclusions: Model performance varied substantially across medication-related tasks. LLaMA3-70B demonstrated promising baseline performance in tasks involving formulation, ordering, route, and indication. GPT-4o-mini showed potential advantages in drug–drug interaction detection and renal dose adjustment. These findings underscore the need for task-specific evaluation when deploying models for medication-focused clinical decision support.

Abstract Background The purpose of this study was to develop a series of benchmarks specifically for large language model (LLM) performance in the identification of drug-drug interactions (DDIs). Methods We evaluated six LLMs using a clinician-designed dataset of 750 DDI scenarios. Tasks included: (1) a pointwise two-drug classification task (2) a pairwise three-drug discrimination task and (3) a listwise 4–6 drug selection task. Zero-shot prompting was applied and performance was assessed using precision, recall, F1 score, accuracy, and self-consistency. Results Model performance varied by task structure and interaction severity. GPT-5-Chat demonstrated the highest recall and F1 score in the pointwise task (80.7% and 80.2%), whereas DrugGPT achieved superior accuracy in the pairwise (93.7%) and listwise task (92.6%). All models had improved performance with more severe compared to less severe DDIs (accuracy 72.0-99.2% vs. 58.3-86.7%). All models tended to have higher self-consistency in the pairwise and listwise experiments compared to the pointwise experiment (68.8-93.6% vs. 31.0-74.2%). No model exhibited uniformly high reliability across all tasks. Conclusions Current LLMs show inconsistent capabilities in identifying DDIs. Performance degrades as reasoning space expands with limited stability. These findings emphasize the need for multi-format evaluation frameworks and reliability-aware assessment when considering LLMs for medication-safety applications.

ABSTRACT Pharmacists play a critical role in optimizing medication use, especially in vulnerable populations such as those with chronic kidney disease (CKD). Despite strong evidence and recent guideline endorsements, many pharmacists continue to rely on the Cockcroft–Gault (C‐G) equation for medication dosing. This opinion paper issues a call to action for the widespread adoption of a race‐free, body surface area–adjusted estimated glomerular filtration rate (eGFR BSAadj ) for medication‐related decision‐making, as endorsed by the 2024 Kidney Disease: Improving Global Outcomes CKD guidelines and the National Kidney Foundation and the American Society of Nephrology Taskforce. Persistent variability in pharmacy practice, institutional protocols, and pharmacy education underscores the need for coordinated change. We urge health systems to integrate eGFR BSAadj into electronic health records and clinical decision support to enhance medication decision‐making; institutions to align protocols with contemporary eGFR‐based prescribing standards; pharmacy educators to contemporize and standardize education on kidney function assessment and medication dosing; and regulatory bodies to update licensure and board examinations. The pharmacy profession must act decisively in order to align with evidence‐based, equitable kidney‐related dosing practices and ensure optimal medication management for all patients.

Abstract Purpose No consistent recommendations exist on how best to educate pharmacists to facilitate implementation of new evidence-based practices. This study aimed to determine whether spaced learning was more effective than traditional education when learning a new evidence-based practice (β-lactam therapeutic drug monitoring, or TDM). Summary β-lactam TDM was implemented in a phased fashion at Mayo Clinic in Rochester. Two go-lives took place, separated by 6 months, which allowed for a natural experiment to test education delivery methods (traditional education in the first go-live group of pharmacists vs spaced learning in the second go-live group of pharmacists). Each group received the same base educational package, with pharmacists in the spaced learning group also offered monthly in-person sessions for 6 months before go-live to discuss key considerations and cases with experts. There was no difference between the spaced learning group and the traditional education group when assessing appropriate use of TDM as defined by (1) appropriate patient selection; (2) correct ordering of laboratory testing; (3) appropriate clinical action; and (4) completed documentation. Assessments of secondary knowledge transfer also showed no difference between the groups. Pharmacists in the different education groups had similar satisfaction scores related to the effect and utility of the education. Both spaced learning and traditional education were part of a comprehensive implementation plan for β-lactam TDM, which might have influenced the outcomes. Spaced learning for future pharmacist education efforts needs further study. Conclusion As part of a greater β-lactam TDM implementation effort, altering the education delivery method did not impact the degree of knowledge transfer.

Due to extensive pathophysiological changes in critically ill patients, standard dosing of antibiotics may lead to inadequate drug exposure. This is concerning, as insufficient plasma drug concentrations may lead to treatment failure, whereas excessive drug exposure may increase the risk of toxic adverse events. The role of inflammation as a factor influencing the pharmacokinetics (PK) and pharmacodynamics (PD) of antibiotics remains largely unknown. PK/PD target attainment of antibiotics can be improved through therapeutic drug monitoring, i.e., measurement of drug concentrations in the blood with subsequent dosage adjustment to reach a certain target. Besides, population PK models may be used to predict drug exposure and tailor dosing in an individual patient (model‐informed precision dosing). Inflammatory biomarkers have been proposed to measure inflammation levels and guide antibiotic treatment. However, their potential to guide antibiotic dosing is unclear. This narrative review describes associations between inflammation and PK/PD of antibiotics in critically ill patients, and the role of biomarkers, therapeutic drug monitoring and model‐informed precision dosing in improving antibiotic dosing. A focus of future research should be on the interplay between inflammation and PK/PD of antibiotics by including inflammatory biomarkers in PK/PD models and using big data to predict antibiotic exposure in critically ill patients.

This study aimed to determine if extended infusion (EI; over > 3 hr) beta-lactam therapy increased IV access requirements compared with traditional dosing (TD; over 30 min). Eighty-six adult ICU patients treated with TD anti-pseudomonal beta-lactams who underwent therapeutic drug monitoring (TDM) were included. Patients who transitioned from TD to EI after TDM (EI group) were matched 1:1 to patients who remained on TD. In the primary analysis, the median (interquartile range) total number of lumens in the 24 hours before TDM compared with the 48 hours after TDM were similar between groups (pre: TD 3 [2–5] vs. EI 4 [3–5]; p = 0.22 and post: TD 3 [2–4] vs. EI 4 [3–5]; p = 0.05). Delivery of beta-lactams via EI was not associated with a need for more IV access. Practical challenges such as access should not limit use of EI beta-lactams when indicated.

Abstract Background Doxycycline is a frequently used alternative treatment for community-acquired pneumonia (CAP) in hospitalized patients; however, the evidence base is limited. We aimed to compare outcomes in CAP patients treated with doxycycline or azithromycin in combination with beta-lactams. Methods We evaluated patients with CAP admitted to Mayo Clinic hospitals from May 2018 to September 2022 and treated with either azithromycin or doxycycline in combination with a beta-lactam. Patients who received both antibiotics or were on either antibiotic as chronic therapy prior to admission were excluded. The primary outcome was mortality. Patients were 1-to-1 matched according to the presence of congestive heart failure, coronary artery disease, chronic obstructive pulmonary disease, human immunodeficiency virus infection, recent immunosuppressant use, and pneumonia severity index class. Results Among 8492 patients (5810 for azithromycin, 2682 for doxycycline), 2671 doxycycline-treated patients were matched with azithromycin-treated patients. In-hospital mortality was significantly lower in the azithromycin group than in the doxycycline group (odds ratio, 0.71; 95% confidence interval [CI], .56–.9), as was 90-day mortality (hazard ratio, 0.83; 95% CI, .73–.95). Patients on azithromycin also had significantly more hospital-free days (HFDs) than those on doxycycline (estimate, 1.37; 95% CI, .99–1.74). Conclusions Among hospitalized patients with CAP, azithromycin was associated with lower mortality and more HFDs compared with doxycycline in combination with beta-lactams. Randomized, controlled trials are needed to assess the effectiveness of each regimen and to compare their potential side effects.

OBJECTIVES: Estimated glomerular filtration rate is more accurate with combined creatinine and cystatin C equations (eGFRcr-cys) than creatinine alone. This study created and evaluated a cefepime dosing nomogram based on eGFRcr-cys for initial dosing in the critically ill. DESIGN: Pharmacokinetic modeling and simulation study. SETTING: Academic medical center. PATIENTS: Critically ill adults treated with cefepime. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: Data from 120 patients with baseline cystatin C and follow-up cefepime levels were used to develop a nomogram based on eGFRcr-cys and weight for initial cefepime dosing. The predicted proportion of patients who achieved a free cefepime concentration above the minimum inhibitory concentration of the organism for 100% of the dosing interval in the first 24 hours (100% ƒT > MIC at 24 hr) was compared between administered doses and those predicted by the nomogram doses. Overall drug exposure was estimated with the free area under the concentration time curve from 0 to 24 hours (ƒAUC0–24) and compared between administered and nomogram doses. Achievement of 100% ƒT > MIC at 24 hours was predicted to be significantly better with the nomogram compared with the administered dose (76% vs. 38%; p < 0.001). The median ƒAUC0–24 as predicted by the nomogram (666 mg·hr/L) was slightly higher than the actual ƒAUC0–24 with administered doses (612 mg·hr/L; p = 0.01), but the nomogram led to fewer ƒAUC0–24 values which were either too high (> 900) or too low (< 300) (7% vs. 20%; p = 0.004). CONCLUSIONS: Use of a cystatin C-inclusive dosing nomogram for cefepime could improve target attainment without increasing the risk of potentially toxic levels in the critically ill.