Silverman, Anna L.

Abstract Introduction Despite regulatory mandates for advanced therapy (AT) washout periods of up to 5 drug half-lives before enrolling patients in inflammatory bowel disease (IBD) trials, the evidence base for these requirements is largely empirical, and whether shorter intervals independently increase the risk of serious infections in real-world practice remains unknown. Methods Using the OptumLabs Data Warehouse, we identified adult patients with IBD who initiated a new AT (tumor necrosis factor antagonists, vedolizumab, interleukin (IL)-12/23 or IL-23 antagonists, or Janus kinase inhibitors) between 2018 and 2023 using a class-specific new-user design. We examined whether exposure to AT within the preceding 12 months (vs no exposure to AT within that window) and timing of discontinuation (≤1 month, 1-<3 months, or 3-12 months) were associated with risk of serious infections using multivariable Cox proportional hazards regression. Results Among 21 564 AT initiators (47 ± 18 years; 52% female; 57% Crohn’s disease), 7490 (34.7%) had AT exposure within the preceding 12 months. Over a mean follow-up of 1.6 ± 0.9 y, 1955 (9.1%) experienced a serious infection. Prior AT exposure was associated with higher infection risk (hazard ratio [HR]: 1.19, 95% confidence interval [CI]: 1.07-1.33), consistent in inverse probability of treatment-weighted analyses (HR, 1.20 [1.06-1.37]). Among patients with prior exposure, timing of discontinuation was not associated with infection risk (HR, 1.17 [0.95-1.43] for ≤1 month vs 3-12 months; HR, 1.01 [0.83-1.22] for 1-<3 months), with no dose–response relationship on continuous modeling. Discussion AT exposure within the preceding 12 months is associated with modestly higher risk of serious infection after initiating a new AT in IBD, but timing of discontinuation does not independently influence this risk.

Objective Closed-source large language models (LLMs) like generative pre-trained transformer 4o (GPT-4o) have shown promise for clinical information extraction but are potentially limited by cost, data security concerns and inflexibility. Open-source models are an attractive alternative with various adaptation strategies with no consensus on best practices. This study aims to rigorously identify optimal adaptation strategies for open-source models and evaluate their performance relative to closed-source alternatives. Methods and Analysis We studied three LLM adaptation strategies: chain-of-thought prompting, few-shot prompting and fine-tuning. Our target for information extraction was the Mayo Endoscopic Subscore (MES). We applied those strategies in all combinations to six open-source models (8–70 billion parameters) using an annotated set of colonoscopy procedure reports from the University of California, San Francisco (N=608) and San Francisco General Hospital (N=217). We analysed the relationship of these strategies to several performance metrics with a mixed-effects model, accounting for the variability between centres and LLMs. GPT-4o served as a closed-source oracle and provided in-depth commentary on the cost-effectiveness of these options. Results Quantised low-rank adaptation (QLoRA) statistically improves ( p §amp;lt; 0.001 ) ) the performance of open-source LLMs by 9.1–15.7 percentage points across accuracy, precision recall and annotation eligibility accuracy. However, GPT-4o with prompt engineering outperforms the best open-source model by 4.9%–11.2%. A simple cost-effectiveness analysis suggests that GPT-4o is more affordable compared with open-source alternatives. Conclusion GPT-4o is currently the most efficient LLM for MES extraction. If unavailable, QLoRA-optimised open-source models are a competitive alternative. However, results also suggest that current instruction-following LLMs including GPT-4o do not fully follow user-provided instructions, leaving room for improvement. More work is needed to achieve consistent, near-perfect performance in clinical information extraction by LLMs.

Abstract Background and study aims There is limited consensus on the optimal method for measuring disease severity in familial adenomatous polyposis (FAP). We aimed to systematically review the operating properties of existing endoscopic severity indices for FAP. Methods We searched MEDLINE, EMBASE, and the Cochrane Library from inception to February 2023 to identify randomized controlled trials (RCTs) that utilized endoscopic outcomes or studies that evaluated the operating properties of endoscopic disease severity indices in FAP. Results A total of 134 studies were included. We evaluated scoring indices and component items of scoring indices, such as polyp count, polyp size, and histology. Partial validation was observed for polyp count and size. The most commonly reported scoring index was the Spigelman classification system, which was used for assessing the severity of duodenal involvement. A single study reported almost perfect interobserver and intra-observer agreement for this system. The InSIGHT polyposis staging system, which was used for assessing colorectal polyp burden, has been partially validated. It showed substantial interobserver reliability; however, the intra-observer reliability was not assessed. Novel criteria for high-risk gastric polyps have been developed and assessed for interobserver reliability. However, these criteria showed a poor level of agreement. Other scoring indices assessing the anal transition zone, duodenal, and colorectal polyps have not undergone validation. Conclusions There are no fully validated endoscopic disease severity indices for FAP. Development and validation of a reliable and responsive endoscopic disease severity instrument will be informative for clinical care and RCTs of pharmacological therapies for FAP.

Outpatient clinical notes are a rich source of information regarding drug safety. However, data in these notes are currently underutilized for pharmacovigilance due to methodological limitations in text mining. Large language models (LLMs) like Bidirectional Encoder Representations from Transformers (BERT) have shown progress in a range of natural language processing tasks but have not yet been evaluated on adverse event (AE) detection. We adapted a new clinical LLM, University of California – San Francisco (UCSF)‐BERT, to identify serious AEs (SAEs) occurring after treatment with a non‐steroid immunosuppressant for inflammatory bowel disease (IBD). We compared this model to other language models that have previously been applied to AE detection. We annotated 928 outpatient IBD notes corresponding to 928 individual patients with IBD for all SAE‐associated hospitalizations occurring after treatment with a non‐steroid immunosuppressant. These notes contained 703 SAEs in total, the most common of which was failure of intended efficacy. Out of eight candidate models, UCSF‐BERT achieved the highest numerical performance on identifying drug‐SAE pairs from this corpus (accuracy 88–92%, macro F1 61–68%), with 5–10% greater accuracy than previously published models. UCSF‐BERT was significantly superior at identifying hospitalization events emergent to medication use ( P < 0.01). LLMs like UCSF‐BERT achieve numerically superior accuracy on the challenging task of SAE detection from clinical notes compared with prior methods. Future work is needed to adapt this methodology to improve model performance and evaluation using multicenter data and newer architectures like Generative pre‐trained transformer (GPT). Our findings support the potential value of using large language models to enhance pharmacovigilance.

Abstract Background The Mayo endoscopic subscore (MES) is an important quantitative measure of disease activity in ulcerative colitis. Colonoscopy reports in routine clinical care usually characterize ulcerative colitis disease activity using free text description, limiting their utility for clinical research and quality improvement. We sought to develop algorithms to classify colonoscopy reports according to their MES. Methods We annotated 500 colonoscopy reports from 2 health systems. We trained and evaluated 4 classes of algorithms. Our primary outcome was accuracy in identifying scorable reports (binary) and assigning an MES (ordinal). Secondary outcomes included learning efficiency, generalizability, and fairness. Results Automated machine learning models achieved 98% and 97% accuracy on the binary and ordinal prediction tasks, outperforming other models. Binary models trained on the University of California, San Francisco data alone maintained accuracy (96%) on validation data from Zuckerberg San Francisco General. When using 80% of the training data, models remained accurate for the binary task (97% [n = 320]) but lost accuracy on the ordinal task (67% [n = 194]). We found no evidence of bias by gender (P = .65) or area deprivation index (P = .80). Conclusions We derived a highly accurate pair of models capable of classifying reports by their MES and recognizing when to abstain from prediction. Our models were generalizable on outside institution validation. There was no evidence of algorithmic bias. Our methods have the potential to enable retrospective studies of treatment effectiveness, prospective identification of patients meeting study criteria, and quality improvement efforts in inflammatory bowel diseases.