Silverman, Anna Lerman

Abstract Background and study aims There is limited consensus on the optimal method for measuring disease severity in familial adenomatous polyposis (FAP). We aimed to systematically review the operating properties of existing endoscopic severity indices for FAP. Methods We searched MEDLINE, EMBASE, and the Cochrane Library from inception to February 2023 to identify randomized controlled trials (RCTs) that utilized endoscopic outcomes or studies that evaluated the operating properties of endoscopic disease severity indices in FAP. Results A total of 134 studies were included. We evaluated scoring indices and component items of scoring indices, such as polyp count, polyp size, and histology. Partial validation was observed for polyp count and size. The most commonly reported scoring index was the Spigelman classification system, which was used for assessing the severity of duodenal involvement. A single study reported almost perfect interobserver and intra-observer agreement for this system. The InSIGHT polyposis staging system, which was used for assessing colorectal polyp burden, has been partially validated. It showed substantial interobserver reliability; however, the intra-observer reliability was not assessed. Novel criteria for high-risk gastric polyps have been developed and assessed for interobserver reliability. However, these criteria showed a poor level of agreement. Other scoring indices assessing the anal transition zone, duodenal, and colorectal polyps have not undergone validation. Conclusions There are no fully validated endoscopic disease severity indices for FAP. Development and validation of a reliable and responsive endoscopic disease severity instrument will be informative for clinical care and RCTs of pharmacological therapies for FAP.

Outpatient clinical notes are a rich source of information regarding drug safety. However, data in these notes are currently underutilized for pharmacovigilance due to methodological limitations in text mining. Large language models (LLMs) like Bidirectional Encoder Representations from Transformers (BERT) have shown progress in a range of natural language processing tasks but have not yet been evaluated on adverse event (AE) detection. We adapted a new clinical LLM, University of California – San Francisco (UCSF)‐BERT, to identify serious AEs (SAEs) occurring after treatment with a non‐steroid immunosuppressant for inflammatory bowel disease (IBD). We compared this model to other language models that have previously been applied to AE detection. We annotated 928 outpatient IBD notes corresponding to 928 individual patients with IBD for all SAE‐associated hospitalizations occurring after treatment with a non‐steroid immunosuppressant. These notes contained 703 SAEs in total, the most common of which was failure of intended efficacy. Out of eight candidate models, UCSF‐BERT achieved the highest numerical performance on identifying drug‐SAE pairs from this corpus (accuracy 88–92%, macro F1 61–68%), with 5–10% greater accuracy than previously published models. UCSF‐BERT was significantly superior at identifying hospitalization events emergent to medication use ( P < 0.01). LLMs like UCSF‐BERT achieve numerically superior accuracy on the challenging task of SAE detection from clinical notes compared with prior methods. Future work is needed to adapt this methodology to improve model performance and evaluation using multicenter data and newer architectures like Generative pre‐trained transformer (GPT). Our findings support the potential value of using large language models to enhance pharmacovigilance.

Abstract Background The Mayo endoscopic subscore (MES) is an important quantitative measure of disease activity in ulcerative colitis. Colonoscopy reports in routine clinical care usually characterize ulcerative colitis disease activity using free text description, limiting their utility for clinical research and quality improvement. We sought to develop algorithms to classify colonoscopy reports according to their MES. Methods We annotated 500 colonoscopy reports from 2 health systems. We trained and evaluated 4 classes of algorithms. Our primary outcome was accuracy in identifying scorable reports (binary) and assigning an MES (ordinal). Secondary outcomes included learning efficiency, generalizability, and fairness. Results Automated machine learning models achieved 98% and 97% accuracy on the binary and ordinal prediction tasks, outperforming other models. Binary models trained on the University of California, San Francisco data alone maintained accuracy (96%) on validation data from Zuckerberg San Francisco General. When using 80% of the training data, models remained accurate for the binary task (97% [n = 320]) but lost accuracy on the ordinal task (67% [n = 194]). We found no evidence of bias by gender (P = .65) or area deprivation index (P = .80). Conclusions We derived a highly accurate pair of models capable of classifying reports by their MES and recognizing when to abstain from prediction. Our models were generalizable on outside institution validation. There was no evidence of algorithmic bias. Our methods have the potential to enable retrospective studies of treatment effectiveness, prospective identification of patients meeting study criteria, and quality improvement efforts in inflammatory bowel diseases.