Belov, Artur

Outpatient clinical notes are a rich source of information regarding drug safety. However, data in these notes are currently underutilized for pharmacovigilance due to methodological limitations in text mining. Large language models (LLMs) like Bidirectional Encoder Representations from Transformers (BERT) have shown progress in a range of natural language processing tasks but have not yet been evaluated on adverse event (AE) detection. We adapted a new clinical LLM, University of California – San Francisco (UCSF)‐BERT, to identify serious AEs (SAEs) occurring after treatment with a non‐steroid immunosuppressant for inflammatory bowel disease (IBD). We compared this model to other language models that have previously been applied to AE detection. We annotated 928 outpatient IBD notes corresponding to 928 individual patients with IBD for all SAE‐associated hospitalizations occurring after treatment with a non‐steroid immunosuppressant. These notes contained 703 SAEs in total, the most common of which was failure of intended efficacy. Out of eight candidate models, UCSF‐BERT achieved the highest numerical performance on identifying drug‐SAE pairs from this corpus (accuracy 88–92%, macro F1 61–68%), with 5–10% greater accuracy than previously published models. UCSF‐BERT was significantly superior at identifying hospitalization events emergent to medication use ( P < 0.01). LLMs like UCSF‐BERT achieve numerically superior accuracy on the challenging task of SAE detection from clinical notes compared with prior methods. Future work is needed to adapt this methodology to improve model performance and evaluation using multicenter data and newer architectures like Generative pre‐trained transformer (GPT). Our findings support the potential value of using large language models to enhance pharmacovigilance.

AbstractThe efficacy of COVID‐19 convalescent plasma (CCP) as a treatment for hospitalized patients with COVID‐19 remains somewhat controversial; however, many studies have not evaluated CCP documented to have high neutralizing antibody titer by a highly accurate assay. To evaluate the correlation of the administration of CCP with titer determined by a live viral neutralization assay with 7‐ and 28‐day death rates during hospitalization, a total of 23 118 patients receiving a single unit of CCP were stratified into two groups: those receiving high titer CCP (>250 50% inhibitory dilution, ID50; n = 13 636) or low titer CCP (≤250 ID50; n = 9482). Multivariable Cox regression was performed to assess risk factors. Non‐intubated patients who were transfused with high titer CCP showed 1.1% and 1.7% absolute reductions in overall 7‐ and 28‐day death rates, respectively, compared to those non‐intubated patients receiving low titer CCP. No benefit of CCP was observed in intubated patients. The relative benefit of high titer CCP was confirmed in multivariable Cox regression. Administration of CCP with high titer antibody content determined by live viral neutralization assay to non‐intubated patients is associated with modest clinical efficacy. Although shown to be only of modest clinical benefit, CCP may play a role in the future should viral variants develop that are not neutralized by other available therapeutics.