Basha, Ayla Shamsi

Objectives This study aims to compare the reliability and accuracy of three large language models (LLMs) (Claude, Gemini and GPT) in assessing the risk of bias of nonrandomised studies using the ROBINS-I tool. Methods and analysis We conducted a secondary analysis of 171 nonrandomised studies previously assessed with Risk Of Bias In Non-randomized Studies of Interventions (ROBINS-I) tool by two independent human review teams. Only studies with concordant human domain-level ratings were included. Each study was independently assessed twice by Claude, Gemini and Generative Pre-trained Transformer (GPT) using agent-based structured implementations of the ROBINS-I tool. Reliability (agreement between two runs of the same LLM) was evaluated using percent agreement and Gwet’s AC1. Accuracy (agreement with human reviewers) was assessed only for studies with consistent LLM ratings, using the same metrics. Results Claude demonstrated high reliability across all domains (79.5–98.0% agreement, AC1=0.729–0.975). Gemini showed moderate-to-high reliability (agreement 76.7–100%, AC1=0.680–1.0). GPT exhibited lower reliability overall, though domain-level agreement ranged from 70.9–95.6% (AC1=0.596–0.944). In terms of accuracy, Claude showed overall poor agreement with human reviewers (14.4–68.5% agreement; low AC1 values). Gemini demonstrated moderate-to-high accuracy in several domains, including deviations from intended interventions (79.6%, AC1=0.848) and measurement of outcomes (73.9%, AC1=0.702), with the highest overall agreement (40.0%, AC1=0.672). GPT showed variable accuracy, with the highest in measurement of outcomes (62.8%, AC1=0.571) and classification of interventions (57.8%, AC1=0.498), but poor performance in selection (14.3%, AC1 = −0.041) and overall agreement (23.0%, AC1=0.267). Conclusions Claude was internally consistent but poorly aligned with human reviewers. Gemini achieved both high reliability and moderate-to-high accuracy, whereas GPT had lower reliability and mixed accuracy. Current off-the-shelf LLMs cannot reliably perform ROBINS-I risk of bias assessments.

Abstract Context Central precocious puberty (CPP). Objective To summarize the available supporting evidence for the Endocrine Society guidelines about the management of CPP. Methods Multiple databases (MEDLINE, EMBASE, Scopus) were searched to identify studies that addressed the Endocrine Society Guideline Development Panel's 10 clinical questions. Identified studies were selected and appraised, and data were extracted by pairs of independent trained reviewers. Results The systematic review yielded 3796 citations, of which 32 citations were included. The systematic review and meta-analysis demonstrate that in children with CPP, with no central nervous system symptoms, the rate of magnetic resonance imaging identification of pathogenic lesions (ie, hamartomas and brain tumors) was 6% (35 noncomparative studies with 5541 children with CPP). For girls with idiopathic CPP, gonadotropin-releasing hormone agonist (GnRHa) treatment was associated with +2.7 cm adult height gain compared to those who did not receive treatment (21 comparative observational studies with 1835 girls [mean age 8.20 ± 1.08 years]). The review did not identify any studies that assessed the clinical questions on: benefits of additional evaluation in cases of early thelarche, differentiation of slowly vs rapidly progressing CPP, order of biochemical testing to establish and monitor the diagnosis of CPP, genetic testing for individuals diagnosed with CPP, and chronological age and/or bone age for discontinuation of GnRHa treatment. Conclusion This systematic review addresses various aspects of CPP evaluation and treatment of CPP and will support the development of the Endocrine Society guidelines.

Objective: The objective of this investigation was to evaluate the agreement rate on judgments made using the Murad tool by different systematic review teams. Introduction: Evaluating the methodological quality of case reports and case series is challenging but some tools do exist for this purpose. We leveraged the presence of studies that have been evaluated by different systematic review teams to assess the inter-consensus agreement among different teams on Murad tool domains. Methods: Using a back-citation method, we identified systematic reviews that used the Murad tool and retrieved all the included primary studies. We selected studies that were assessed by more than 1 systematic review team. We calculated observed agreement and Gwet’s agreement coefficient on judgments made about each signaling question. Results: We identified 982 systematic reviews that cited the Murad tool and collectively cited 59,080 references. The final data comprised of 81 duplicated case reports and series assessed by more than 1 systematic review team. Overall, the signaling questions had very high observed agreement with 5 of the 8 questions having agreement over 75%. The signaling questions with the highest agreement addressed the adequacy of the ascertainment of the exposure (coefficient 0.959), ascertainment of the outcome (coefficient 0.829), presence of a dose-response gradient (perfect agreement), and clarity of reporting (coefficient 0.755). Conclusion: The current study demonstrates overall high agreement among different systematic review teams that used the Murad tool for appraisal of the same case series and case reports. Leveraging duplicated studies across systematic reviews is a feasible way to retrospectively assess the reliability of the tool.

Abstract Background Risk of bias (RoB) tools are critical in systematic reviews and affect subsequent decision‐making. RoB tools should have adequate interrater reliability and interconsensus agreement. We present an approach of post hoc evaluation of RoB tools using duplicated studies that overlap systematic reviews. Methods Using a back‐citation approach, we identified systematic reviews that used the Risk Of Bias In Nonrandomized Studies‐of Interventions (ROBINS‐I) tool and retrieved all the included primary studies. We selected studies that were appraised by more than one systematic review and calculated observed agreement and unweighted kappa comparing the different systematic reviews' assessments. Results We identified 903 systematic reviews that used the tool with 51,676 cited references, from which we eventually analyzed 171 duplicated studies assessed using ROBINS‐I by different systematic reviewers. The observed agreement on ROBINS‐I domains ranged from 54.9% (missing data domain) to 70.3% (deviations from intended interventions domain), and was 63.0% for overall RoB assessment of the study. Kappa coefficient ranged from 0.131 (measurement of outcome domain) to 0.396 (domains of confounding and deviations from intended interventions), and was 0.404 for overall RoB assessment of the study. Conclusion A post hoc evaluation of RoB tools is feasible by focusing on duplicated studies that overlap systematic review. ROBINS‐I assessments demonstrated considerable variation in interconsensus agreement among various systematic reviewes that assessed the same study and outcome, suggesting the need for more intensive upfront work to calibrate systematic reviewers on how to identify context‐specific information and agree on how to judge it.