Citations

Objective To examine the potential errors of a general large language model (LLM) (ie, Claude 3.5 Sonnet) on data extraction from randomised controlled trials (RCTs). Design and setting An empirical study comparing Claude 3.5 Sonnet extractions against a human-performed verification dataset. The extraction tasks for Claude 3.5 Sonnet were based solely on original RCT portable document format (PDF) files. For PDFs that could not be directly extracted by Claude 3.5 Sonnet, optical character recognition was employed to convert them into text format before extraction. Participants A random sample of 664 trials was selected from a well-established trial bank and a final data pool was established based on rigorous manual cross-checking as a reference standard. Data sources PubMed, EMBASE, Scopus, Web of Science (all databases) and the Cochrane Central Register of Controlled Trials (CENTRAL) up to February 2023. Eligibility criteria for selecting studies RCTs on children involving medication and adverse events. Main outcome measures Claude 3.5 Sonnet was applied to extract the basic information (eg, trial design, population information and source of funding) and adverse outcomes (ie, name of adverse events, number of events). Claude 3.5 Sonnet outputs were compared against the final data pool and all errors were recorded. Results are presented as error rates and with 95% CI, estimated using a generalised linear mixed model. Results For the 664 trials, a total of 23 069 data cells were extracted via Claude 3.5 Sonnet, with 10 624 for basic information and 12 445 for adverse outcomes. The overall error rate for data extraction was 6.6% (95% CI 5.4% to 8.2%), with 5.7% (95% CI 5.2% to 6.1%) in basic information and 7.6% (95% CI 4.9% to 11.8%) in adverse outcomes. When stratified the 1542 total errors by error types, misallocation (assigning data to incorrect fields; 57.1%, 881/1542) and missed or omitted data (incomplete extraction of available data; 23.2%, 357/1542) accounted for the two most frequent errors, with misallocation occurring more in basic information (53.3%, 470/881), while missed or omitted data occurred more in adverse outcomes (96.1%, 343/357). Post hoc analysis examining the association between trial reporting quality (assessed using Consolidated Standards of Reporting Trials (CONSORT) 2025 and LLM data extraction error rates indicated that higher CONSORT adherence was associated with lower extraction error rates. Conclusions The data extraction error of Claude was relatively low, but it alerts LLM applications in evidence synthesis. Detailed checking for LLM outputs should be the primary consideration for evidence synthesisers.

Introduction Increasing publication of fraudulent clinical trials poses a serious threat to evidence-based medicine. In the VITALITY Study I, we demonstrated that contamination by retracted trials significantly distorts evidence synthesis. The upcoming VITALITY Study II will take a step further and investigate how such problematic evidence undermines the credibility of guideline recommendations. Methods and analyses The VITALITY Study II will adopt a retrospective cohort design and will be structured as three work packages (WPs). To start with, a cohort of clinical practice guidelines (CPGs) that were contaminated by retracted trials and/or meta-analyses that included retracted trials will be established based on forward citation searching (WP1). Then, recommendations from these CPGs that used evidence from retracted trials and/or meta-analyses that synthesised these retracted trials will be re-evaluated after removing such problematic evidence, in terms of both the direction and strength of effect sizes (WP2). Finally, the subsequent impact on patient outcomes attributable to distorted recommendations will be estimated on a hypothetical population, measured by the number of expected benefit loss and the number of expected harm increment per 100 000 patients (WP3). Ethics and dissemination Formal ethical approval is not necessary for this study as it does not involve human or animal participants, nor does it involve the collection of primary data. We will disseminate the findings through publication in peer-reviewed journals and, whenever possible, presentations at academic conferences.

Abstract The dic(9;12)(p12;p11.2) has been described as a rare cytogenetic abnormality in pediatric precursor B-cell ALL. Initial studies suggested that the rearrangement is associated with a favorable outcome, and recent studies demonstrated the presence of a PAX5-ETV6 fusion gene was associated with this cytogenetic abnormality. Twenty cases with a cytogenetic dic(9;12) were identified in the Children’s Oncology Group (COG) cytogenetics databases. FISH analysis with the ETV6-RUNX1 (TEL-AML1) probes was done on 12 of these samples. Five cases were positive for fusion, indicating a cryptic t(12;21)(p13;q22), and also had loss of the ETV6 probe from the chromosome 12 not involved in the t(12;21). Seven cases were negative for fusion and had loss of an ETV6 signal, although one of the latter had a diminished ETV6 signal identified. To determine whether both PAX5-ETV6 and ETV6-RUNX1 rearrangements occurred in some patients, a diagnostic sample from each patient was analyzed by RT-PCR for the PAX5-ETV6 and ETV6-RUNX1 fusion genes. Primers from exon 3 of PAX5 and exon 3 of ETV6 were used for the PAX5-ETV6 analysis and from exon 5 of ETV6 and exon 4 of RUNX1 for the ETV6-RUNX1 analysis. Of the 20 cases, only 8 were RT-PCR positive for the PAX5-ETV6 fusion with the above primers; however, an additional 2 were RT-PCR positive with alternate primers, and all 10 of these were negative for the ETV6-RUNX1 fusion by RT-PCR. Of the remaining 10 patients, 9 were RT-PCR positive for the ETV6-RUNX1 fusion, including all of the ETV6-RUNX1 cases positive by FISH. The gene rearrangement associated with the dic(9;12) in these cases is not known. One patient was negative for both fusions by RT-PCR, negative by FISH for ETV6-RUNX1 rearrangement, yet had loss of an ETV6 signal. No cytogenetic differences could be seen between the 2 groups, either in the appearance of the dic(9;12) or in the other abnormalities identified. These results demonstrate the presence of two mutually exclusive dic(9;12) rearrangements in pediatric ALL; one associated with ETV6-RUNX1 rearrangement and one resulting in PAX5-ETV6 fusion. Both PAX5-ETV6 and ETV6-RUNX1 rearrangements are associated with a favorable prognosis. However, molecular analysis of the dic(9;12) patients must be performed to determine whether the dicentric chromosome results in PAX5-ETV6 fusion or whether the case has ETV6-RUNX1 fusion.