Zhou, Hongjian

Abstract Background The purpose of this study was to develop a series of benchmarks specifically for large language model (LLM) performance in the identification of drug-drug interactions (DDIs). Methods We evaluated six LLMs using a clinician-designed dataset of 750 DDI scenarios. Tasks included: (1) a pointwise two-drug classification task (2) a pairwise three-drug discrimination task and (3) a listwise 4–6 drug selection task. Zero-shot prompting was applied and performance was assessed using precision, recall, F1 score, accuracy, and self-consistency. Results Model performance varied by task structure and interaction severity. GPT-5-Chat demonstrated the highest recall and F1 score in the pointwise task (80.7% and 80.2%), whereas DrugGPT achieved superior accuracy in the pairwise (93.7%) and listwise task (92.6%). All models had improved performance with more severe compared to less severe DDIs (accuracy 72.0-99.2% vs. 58.3-86.7%). All models tended to have higher self-consistency in the pairwise and listwise experiments compared to the pointwise experiment (68.8-93.6% vs. 31.0-74.2%). No model exhibited uniformly high reliability across all tasks. Conclusions Current LLMs show inconsistent capabilities in identifying DDIs. Performance degrades as reasoning space expands with limited stability. These findings emphasize the need for multi-format evaluation frameworks and reliability-aware assessment when considering LLMs for medication-safety applications.