Shamsi Basha, Ayla

Objectives This study aims to compare the reliability and accuracy of three large language models (LLMs) (Claude, Gemini and GPT) in assessing the risk of bias of nonrandomised studies using the ROBINS-I tool. Methods and analysis We conducted a secondary analysis of 171 nonrandomised studies previously assessed with Risk Of Bias In Non-randomized Studies of Interventions (ROBINS-I) tool by two independent human review teams. Only studies with concordant human domain-level ratings were included. Each study was independently assessed twice by Claude, Gemini and Generative Pre-trained Transformer (GPT) using agent-based structured implementations of the ROBINS-I tool. Reliability (agreement between two runs of the same LLM) was evaluated using percent agreement and Gwet’s AC1. Accuracy (agreement with human reviewers) was assessed only for studies with consistent LLM ratings, using the same metrics. Results Claude demonstrated high reliability across all domains (79.5–98.0% agreement, AC1=0.729–0.975). Gemini showed moderate-to-high reliability (agreement 76.7–100%, AC1=0.680–1.0). GPT exhibited lower reliability overall, though domain-level agreement ranged from 70.9–95.6% (AC1=0.596–0.944). In terms of accuracy, Claude showed overall poor agreement with human reviewers (14.4–68.5% agreement; low AC1 values). Gemini demonstrated moderate-to-high accuracy in several domains, including deviations from intended interventions (79.6%, AC1=0.848) and measurement of outcomes (73.9%, AC1=0.702), with the highest overall agreement (40.0%, AC1=0.672). GPT showed variable accuracy, with the highest in measurement of outcomes (62.8%, AC1=0.571) and classification of interventions (57.8%, AC1=0.498), but poor performance in selection (14.3%, AC1 = −0.041) and overall agreement (23.0%, AC1=0.267). Conclusions Claude was internally consistent but poorly aligned with human reviewers. Gemini achieved both high reliability and moderate-to-high accuracy, whereas GPT had lower reliability and mixed accuracy. Current off-the-shelf LLMs cannot reliably perform ROBINS-I risk of bias assessments.

Objective: The objective of this investigation was to evaluate the agreement rate on judgments made using the Murad tool by different systematic review teams. Introduction: Evaluating the methodological quality of case reports and case series is challenging but some tools do exist for this purpose. We leveraged the presence of studies that have been evaluated by different systematic review teams to assess the inter-consensus agreement among different teams on Murad tool domains. Methods: Using a back-citation method, we identified systematic reviews that used the Murad tool and retrieved all the included primary studies. We selected studies that were assessed by more than 1 systematic review team. We calculated observed agreement and Gwet’s agreement coefficient on judgments made about each signaling question. Results: We identified 982 systematic reviews that cited the Murad tool and collectively cited 59,080 references. The final data comprised of 81 duplicated case reports and series assessed by more than 1 systematic review team. Overall, the signaling questions had very high observed agreement with 5 of the 8 questions having agreement over 75%. The signaling questions with the highest agreement addressed the adequacy of the ascertainment of the exposure (coefficient 0.959), ascertainment of the outcome (coefficient 0.829), presence of a dose-response gradient (perfect agreement), and clarity of reporting (coefficient 0.755). Conclusion: The current study demonstrates overall high agreement among different systematic review teams that used the Murad tool for appraisal of the same case series and case reports. Leveraging duplicated studies across systematic reviews is a feasible way to retrospectively assess the reliability of the tool.

Abstract Background Risk of bias (RoB) tools are critical in systematic reviews and affect subsequent decision‐making. RoB tools should have adequate interrater reliability and interconsensus agreement. We present an approach of post hoc evaluation of RoB tools using duplicated studies that overlap systematic reviews. Methods Using a back‐citation approach, we identified systematic reviews that used the Risk Of Bias In Nonrandomized Studies‐of Interventions (ROBINS‐I) tool and retrieved all the included primary studies. We selected studies that were appraised by more than one systematic review and calculated observed agreement and unweighted kappa comparing the different systematic reviews' assessments. Results We identified 903 systematic reviews that used the tool with 51,676 cited references, from which we eventually analyzed 171 duplicated studies assessed using ROBINS‐I by different systematic reviewers. The observed agreement on ROBINS‐I domains ranged from 54.9% (missing data domain) to 70.3% (deviations from intended interventions domain), and was 63.0% for overall RoB assessment of the study. Kappa coefficient ranged from 0.131 (measurement of outcome domain) to 0.396 (domains of confounding and deviations from intended interventions), and was 0.404 for overall RoB assessment of the study. Conclusion A post hoc evaluation of RoB tools is feasible by focusing on duplicated studies that overlap systematic review. ROBINS‐I assessments demonstrated considerable variation in interconsensus agreement among various systematic reviewes that assessed the same study and outcome, suggesting the need for more intensive upfront work to calibrate systematic reviewers on how to identify context‐specific information and agree on how to judge it.