Lin, Lifeng

Objective To examine the potential errors of a general large language model (LLM) (ie, Claude 3.5 Sonnet) on data extraction from randomised controlled trials (RCTs). Design and setting An empirical study comparing Claude 3.5 Sonnet extractions against a human-performed verification dataset. The extraction tasks for Claude 3.5 Sonnet were based solely on original RCT portable document format (PDF) files. For PDFs that could not be directly extracted by Claude 3.5 Sonnet, optical character recognition was employed to convert them into text format before extraction. Participants A random sample of 664 trials was selected from a well-established trial bank and a final data pool was established based on rigorous manual cross-checking as a reference standard. Data sources PubMed, EMBASE, Scopus, Web of Science (all databases) and the Cochrane Central Register of Controlled Trials (CENTRAL) up to February 2023. Eligibility criteria for selecting studies RCTs on children involving medication and adverse events. Main outcome measures Claude 3.5 Sonnet was applied to extract the basic information (eg, trial design, population information and source of funding) and adverse outcomes (ie, name of adverse events, number of events). Claude 3.5 Sonnet outputs were compared against the final data pool and all errors were recorded. Results are presented as error rates and with 95% CI, estimated using a generalised linear mixed model. Results For the 664 trials, a total of 23 069 data cells were extracted via Claude 3.5 Sonnet, with 10 624 for basic information and 12 445 for adverse outcomes. The overall error rate for data extraction was 6.6% (95% CI 5.4% to 8.2%), with 5.7% (95% CI 5.2% to 6.1%) in basic information and 7.6% (95% CI 4.9% to 11.8%) in adverse outcomes. When stratified the 1542 total errors by error types, misallocation (assigning data to incorrect fields; 57.1%, 881/1542) and missed or omitted data (incomplete extraction of available data; 23.2%, 357/1542) accounted for the two most frequent errors, with misallocation occurring more in basic information (53.3%, 470/881), while missed or omitted data occurred more in adverse outcomes (96.1%, 343/357). Post hoc analysis examining the association between trial reporting quality (assessed using Consolidated Standards of Reporting Trials (CONSORT) 2025 and LLM data extraction error rates indicated that higher CONSORT adherence was associated with lower extraction error rates. Conclusions The data extraction error of Claude was relatively low, but it alerts LLM applications in evidence synthesis. Detailed checking for LLM outputs should be the primary consideration for evidence synthesisers.

Introduction Increasing publication of fraudulent clinical trials poses a serious threat to evidence-based medicine. In the VITALITY Study I, we demonstrated that contamination by retracted trials significantly distorts evidence synthesis. The upcoming VITALITY Study II will take a step further and investigate how such problematic evidence undermines the credibility of guideline recommendations. Methods and analyses The VITALITY Study II will adopt a retrospective cohort design and will be structured as three work packages (WPs). To start with, a cohort of clinical practice guidelines (CPGs) that were contaminated by retracted trials and/or meta-analyses that included retracted trials will be established based on forward citation searching (WP1). Then, recommendations from these CPGs that used evidence from retracted trials and/or meta-analyses that synthesised these retracted trials will be re-evaluated after removing such problematic evidence, in terms of both the direction and strength of effect sizes (WP2). Finally, the subsequent impact on patient outcomes attributable to distorted recommendations will be estimated on a hypothetical population, measured by the number of expected benefit loss and the number of expected harm increment per 100 000 patients (WP3). Ethics and dissemination Formal ethical approval is not necessary for this study as it does not involve human or animal participants, nor does it involve the collection of primary data. We will disseminate the findings through publication in peer-reviewed journals and, whenever possible, presentations at academic conferences.

ABSTRACT Introduction Publication bias (PB) threatens the validity of meta‐analyses by distorting effect size estimates, potentially leading to misleading conclusions. With advanced pattern recognition and multimodal capabilities, large language models (LLMs) may be able to evaluate PB and make the systematic review process more efficient. Methods We evaluated the ability of two state‐of‐the‐art multimodal LLMs, GPT‐4o and Llama 3.2 Vision, to detect PB using funnel plots alone and in combination with quantitative inputs. We simulated meta‐analyses under varying conditions, including the absence of PB, different levels of presence of PB, varying total number of studies within a meta‐analysis, and differing degrees of between‐study heterogeneity. Results Neither GPT‐4o nor Llama 3.2 Vision consistently detected the presence of PB across various settings. Under no‐publication‐bias conditions, GPT‐4o achieved a higher specificity outperforming Llama 3.2 Vision, with the difference most shown in the meta‐analyses with 20 or more studies. The inclusion of quantitative inputs alongside funnel plots did not significantly improve performance. Additionally, between‐study heterogeneity and patterns of non‐reported studies had minimal impact on the models’ assessments. Conclusions The ability of LLMs to detect PB without fine‐tuning is limited at the present time. This study highlights the need for specialized model adaptation before LLMs can be effectively integrated into meta‐analysis workflows. Future research can focus on targeted refinements to enhance LLM performance and utility in evidence synthesis.

ABSTRACT Background and Objectives The relevance of Cochrane systematic reviews depends on timely completion and updates. This study aimed to empirically assess the lifecycles of Cochrane reviews published from 2003 to 2024, including transitions from protocol to review, update patterns, and withdrawals. Methods We extracted data from Cochrane Library publications between 2003 and 2024. Each review topic was identified using a unique six‐digit DOI‐based ID. We recorded protocol publication, review publication, updates, and withdrawals (i.e., removed from the Cochrane Library for editorial or procedural reasons), calculating time intervals between stages and conducting subgroup analyses by review type. Results Of 8137 protocols, 71.9% progressed to reviews (median 25.7 months), 2.4% were updated during the protocol stage, and 10.0% were withdrawn. Among 8477 reviews, 64.3% were never updated by the time of our analysis; for those updated at least once, the median interval between updates was 57.2 months. Withdrawal occurred in 2.5% of reviews (median 67.6 months post‐publication). Subgroup analyses showed variation across review types; diagnostic and qualitative reviews tended to have longer protocol‐to‐review times than other types of reviews. Conclusions Cochrane reviews show long development and update intervals, with variation by review type. Greater use of automation and targeted support may improve review efficiency and timeliness.

Abstract The common practice in meta‐analysis and clinical practice guidelines is to derive the absolute treatment effect (also called risk difference, RD) from a combination of a pooled relative risk (RR) that resulted from a meta‐analysis, and a user‐provided baseline risk (BR). However, this method does not address the uncertainty in BR. We developed a web‐based R Shiny tool to perform simple microsimulation and incorporate uncertainty in BR into the precision of RD. We empirically evaluated this approach by estimating the impact of incorporating this uncertainty when BR is derived from the control group rates in 3,128 meta‐analyses curated from the Cochrane Library (26,964 individual studies). When BR was derived from the largest study in each meta‐analysis, the median width of the CI of BR was 11.6% (interquartile range (IQR), 6.30%–18.5%). Incorporating this uncertainty in BR led to expansion of the RD CI by a median of 8 per 1,000 persons (IQR 2–24). This expansion increased in a linear fashion with BR imprecision and was more prominent in meta‐analyses with low BR. This study provides a web‐based tool to perform simple microsimulation and incorporate uncertainty in BR into the CI of RD.

Publication bias occurs when studies with statistically significant results have increased likelihood of being published. Publication bias is commonly associated with inflated treatment effect which lowers the certainty of decision makers about the evidence. In this guide we propose that systematic reviewers and decision makers consider the direction and magnitude of publication bias, as opposed to just the binary determination of the presence of this bias, before lowering their certainty in the evidence. Direction of bias may not always exaggerate the treatment effect. The presence of bias with a trivial magnitude may not affect the decision at hand. Various statistical approaches are available to determine the direction and magnitude of publication bias.