Firwana, Mohammed

Objectives This study aims to compare the reliability and accuracy of three large language models (LLMs) (Claude, Gemini and GPT) in assessing the risk of bias of nonrandomised studies using the ROBINS-I tool. Methods and analysis We conducted a secondary analysis of 171 nonrandomised studies previously assessed with Risk Of Bias In Non-randomized Studies of Interventions (ROBINS-I) tool by two independent human review teams. Only studies with concordant human domain-level ratings were included. Each study was independently assessed twice by Claude, Gemini and Generative Pre-trained Transformer (GPT) using agent-based structured implementations of the ROBINS-I tool. Reliability (agreement between two runs of the same LLM) was evaluated using percent agreement and Gwet’s AC1. Accuracy (agreement with human reviewers) was assessed only for studies with consistent LLM ratings, using the same metrics. Results Claude demonstrated high reliability across all domains (79.5–98.0% agreement, AC1=0.729–0.975). Gemini showed moderate-to-high reliability (agreement 76.7–100%, AC1=0.680–1.0). GPT exhibited lower reliability overall, though domain-level agreement ranged from 70.9–95.6% (AC1=0.596–0.944). In terms of accuracy, Claude showed overall poor agreement with human reviewers (14.4–68.5% agreement; low AC1 values). Gemini demonstrated moderate-to-high accuracy in several domains, including deviations from intended interventions (79.6%, AC1=0.848) and measurement of outcomes (73.9%, AC1=0.702), with the highest overall agreement (40.0%, AC1=0.672). GPT showed variable accuracy, with the highest in measurement of outcomes (62.8%, AC1=0.571) and classification of interventions (57.8%, AC1=0.498), but poor performance in selection (14.3%, AC1 = −0.041) and overall agreement (23.0%, AC1=0.267). Conclusions Claude was internally consistent but poorly aligned with human reviewers. Gemini achieved both high reliability and moderate-to-high accuracy, whereas GPT had lower reliability and mixed accuracy. Current off-the-shelf LLMs cannot reliably perform ROBINS-I risk of bias assessments.

Abstract Context Central precocious puberty (CPP). Objective To summarize the available supporting evidence for the Endocrine Society guidelines about the management of CPP. Methods Multiple databases (MEDLINE, EMBASE, Scopus) were searched to identify studies that addressed the Endocrine Society Guideline Development Panel's 10 clinical questions. Identified studies were selected and appraised, and data were extracted by pairs of independent trained reviewers. Results The systematic review yielded 3796 citations, of which 32 citations were included. The systematic review and meta-analysis demonstrate that in children with CPP, with no central nervous system symptoms, the rate of magnetic resonance imaging identification of pathogenic lesions (ie, hamartomas and brain tumors) was 6% (35 noncomparative studies with 5541 children with CPP). For girls with idiopathic CPP, gonadotropin-releasing hormone agonist (GnRHa) treatment was associated with +2.7 cm adult height gain compared to those who did not receive treatment (21 comparative observational studies with 1835 girls [mean age 8.20 ± 1.08 years]). The review did not identify any studies that assessed the clinical questions on: benefits of additional evaluation in cases of early thelarche, differentiation of slowly vs rapidly progressing CPP, order of biochemical testing to establish and monitor the diagnosis of CPP, genetic testing for individuals diagnosed with CPP, and chronological age and/or bone age for discontinuation of GnRHa treatment. Conclusion This systematic review addresses various aspects of CPP evaluation and treatment of CPP and will support the development of the Endocrine Society guidelines.

Objectives To describe the incidence, presentation and long-term health outcomes of suicidal thoughts and behaviours (STBs) in children aged 12 years or under. Methods This population-based study included children identified through the Rochester Epidemiology Project who presented between 2005 and 2023 with STBs across primary, secondary and tertiary care centres in Olmsted County, Minnesota, USA. Information related to the patient and family characteristics, presentation, prior history and outcomes was manually extracted by two independent researchers. Patients were excluded if the index visit note could not be located, the patient had no suicidal ideation, attempts, intent or plan, was older than 12 years 11 months at the index date, less than 2 years old or was a duplicate entry. Results The average annual population was 28,035 children, of which 637 presented with STBs (mean (SD) age, 10.6 (1.7) years; 51.2% girls, 76.3% White, average follow-up 7 years). The majority of the cases presented to the emergency department (ED) 491 (77.1%). The annual incidence per 100,000 person-years tripled from 68.8 in 2005 to 208.6 in 2023. Overall, 105 patients (16.5%) presented with a suicidal attempt. There were no cases of death by suicide. A prior psychiatric diagnosis was present in 454 (71.3%). STB events were preceded by a precipitating event in 471 (73.9%), the most common of which was an argument with a parent, followed by a bullying event and family distress. A specific suicide plan was reported by 328 (51.5%) with laceration reported most frequently, followed by hanging and overdose. Significant predictors of a suicide attempt were previous use of mood stabilisers (OR 3.21; 95% CI 1.24 to 7.97) and having a specific plan (OR 2.73; 95% CI 1.72 to 4.41). Children who had suicidal attempts had more subsequent ED visits (3.50 vs 2.58; p=0.009) and hospitalisations for psychiatric reasons (1.90 vs 1.30; p=0.003) and psychiatric hospitalisation days (12.70 vs 9.04; p=0.048). Subsequent suicide attempts occurred in 31% of the cohort during follow-up. Conclusion STBs in preadolescent children are rare but are increasing in incidence. Such children have significant psychological diagnoses, use of mental health services, and subsequent suicide attempts. Novel age-appropriate interventions are needed.

Abstract Background Partial breast irradiation (PBI) is an effective treatment for early-stage breast cancer. However, evidence comparing different PBI modalities is limited. Methods This trial emulation uses electronic health records from a multi-state large tertiary health system. Three PBI modalities were compared: proton radiation therapy, photon radiation therapy, and applicator-based brachytherapy. Inverse probability weighting (IPW) was used to balance confounders. The primary outcome was ipsilateral breast recurrence (IBR). Results Between 2013 and 2023, 1041 women with early-stage breast cancer were treated with PBI. Of these, 1026 patients were included in the analyses. The median age was 66 years; 22.51% had ductal carcinoma in situ (DCIS), and 97.56% were estrogen receptor positive. With median follow-up of 38.2 months, the rate of IBR in the overall group was 3.05% (95% confidence interval [CI] = 1.99% to 4.65%) at 3 years. After IPW, compared with photon PBI, the hazard ratio for IBR for proton PBI was 2.11 (95% CI = 0.70 to 6.33) and for brachytherapy, 3.86 (95% CI = 1.39 to 10.69). Compared with proton PBI, the hazard ratio for IBR for brachytherapy was 1.83 (95% CI = 0.88 to 3.81). IBR risk was similar across PBI modalities for patients with tumor size ≤ 10 mm and in patients treated for DCIS. Among patients with tumor size >10 mm, the hazard ratio for IBR for brachytherapy was 7.64 (95% CI = 1.64 to 35.58) and 3.59 (95% CI = 1.22 to 10.56) relative to photons and protons, respectively. Conclusion PBI with applicator-based brachytherapy was associated with higher IBR than photon or proton PBI in patients with tumor size > 10 mm, suggesting the need for more careful PBI patient selection for this modality.

Importance Suicide is a leading cause of death among young people and an escalating public health crisis. Objective To assess the effectiveness and harms of available treatments for suicidal thoughts and behaviors among youths at heightened risk for suicide. Evidence Review A systematic review was conducted searching databases including MEDLINE, Embase, Cochrane Central Register of Controlled Trials, Cochrane Database of Systematic Reviews, Scopus, and various gray literature sources from January 1, 2000, to September 26, 2024. We included randomized clinical trials, comparative observational studies, and before-after studies of psychosocial interventions, pharmacologic interventions, neurotherapeutics, emerging therapies, and combination therapies. Eligible patients were youths (aged 5-24 years) with heightened risk for suicide, adolescents from racial and ethnic minority groups known to be at increased risk of suicide, or those exposed to crime or violence. Pairs of independent reviewers selected and appraised studies. Findings This review included 65 studies (33 randomized clinical trials, 13 comparative observational studies, and 19 before-after studies) reporting on 14 534 patients (median age, 15.1 years; 75.1% female patients). Psychosocial interventions comprised psychotherapy interventions (33 studies; cognitive behavioral therapy, dialectical behavioral therapy, collaborative assessment and management of suicidality, dynamic deconstructive psychotherapy, attachment-based family therapy, and family-focused therapy), acute (ie, 1-4 sessions or contacts) psychosocial interventions (19 studies; safety planning, family-based crisis management, motivational interviewing crisis interventions, continuity of care after crisis, and brief adjunctive treatments), and school- or community-based psychosocial interventions (13 studies; social network interventions, school-based skills interventions, suicide awareness or gatekeeper programs, and community-based, culturally tailored adjunct programs). Dialectical behavior therapy showed moderate strength of evidence for reducing suicidal ideation. Other categories of psychosocial treatments showed insufficient to low strength of evidence for reducing suicidal outcomes. None of the studies evaluated adverse events. The evidence base on pharmacologic treatment for youths at risk of suicide was largely nonexistent. Conclusions and Relevance This systematic review found that the current evidence on available interventions targeting youths at heightened risk of suicide is uncertain. Medication, neurotherapeutics, and emerging therapies remain unstudied among this population. The limited evidence base calls for the development of novel, developmentally and trauma-informed treatments, as well as multilevel interventions to target the increasing suicide risk among youths.

Abstract Context Primary aldosteronism (PA) is a leading endocrine cause of secondary hypertension. Objective To support the development of the Endocrine Society Clinical Practice Guideline on managing PA. Data Source MEDLINE, Embase, Scopus, and others were searched on October 4, 2024. Study Selection Studies were selected by pairs of independent reviewers. Data Extraction Data were extracted and appraised by pairs of independent reviewers. Data Synthesis We included 95 studies (7 randomized trials and 88 observational studies). We did not identify trials that evaluated the outcomes of PA screening. One observational study suggested that screening was associated with higher rates of using PA-specific medical therapies and better blood pressure control. Patients managed with adrenal venous sampling (vs computed tomography alone) may have a better post-adrenalectomy biochemical cure rate, but with an increased risk of adrenal hemorrhage. Two small observational studies suggested that PA-specific medical or surgical therapy was likely associated with better blood pressure control than nonspecific therapy. Small randomized trials suggested that surgical therapy may be associated with better blood pressure control than medical therapy, with a lower number and dosage of antihypertensive medications. Compared to eplerenone, spironolactone may be associated with better control of hypokalemia and a lower number and dosage of antihypertensive agents. Unsuppressed plasma renin activity was associated with better control of hypokalemia, while suppression was associated with higher risk of mortality, atrial fibrillation, and stroke (very low certainty). Conclusion This systematic review addresses various aspects of managing PA and will support the development of the Endocrine Society guidelines.

Abstract Introduction Case reports and series are critical to guide initial decision-making in a pandemic, but may have lower rigor because of the need to publish them quickly. This meta-epidemiologic study compares the methodological quality of case series that described the acute coronavirus disease 2019 (COVID-19) pandemic in 2020 versus those that described long-haul cases. Methods We conducted a systematic review in multiple databases for long-haul case series and reports. We identified early cases of acute COVID-19 synthesized in published systematic reviews. We evaluated the methodological quality by pairs of independent reviewers using a tool dedicated for appraising case series. Results We included 239 original case series (81 published in the first year of the pandemic and 158 published later describing long-haul COVID). The methodological quality of both groups of case series was very good (80–100% of series satisfying quality items) except for two items, the selection approach of cases included in the series and ruling out other causes that can explain the main finding described in the series. The appraisal tool demonstrated high agreement and reliability between reviewers. Conclusion The methodological quality of modern case series is high, except for two quality items that represent an area for potential for improvement for authors of case series describing future pandemics.

Abstract Background Risk of bias (RoB) tools are critical in systematic reviews and affect subsequent decision‐making. RoB tools should have adequate interrater reliability and interconsensus agreement. We present an approach of post hoc evaluation of RoB tools using duplicated studies that overlap systematic reviews. Methods Using a back‐citation approach, we identified systematic reviews that used the Risk Of Bias In Nonrandomized Studies‐of Interventions (ROBINS‐I) tool and retrieved all the included primary studies. We selected studies that were appraised by more than one systematic review and calculated observed agreement and unweighted kappa comparing the different systematic reviews' assessments. Results We identified 903 systematic reviews that used the tool with 51,676 cited references, from which we eventually analyzed 171 duplicated studies assessed using ROBINS‐I by different systematic reviewers. The observed agreement on ROBINS‐I domains ranged from 54.9% (missing data domain) to 70.3% (deviations from intended interventions domain), and was 63.0% for overall RoB assessment of the study. Kappa coefficient ranged from 0.131 (measurement of outcome domain) to 0.396 (domains of confounding and deviations from intended interventions), and was 0.404 for overall RoB assessment of the study. Conclusion A post hoc evaluation of RoB tools is feasible by focusing on duplicated studies that overlap systematic review. ROBINS‐I assessments demonstrated considerable variation in interconsensus agreement among various systematic reviewes that assessed the same study and outcome, suggesting the need for more intensive upfront work to calibrate systematic reviewers on how to identify context‐specific information and agree on how to judge it.

Abstract Context Low vitamin D status is common and is associated with various common medical conditions. Objective To support the development of the Endocrine Society's Clinical Practice Guideline on Vitamin D for the Prevention of Disease. Methods We searched multiple databases for studies that addressed 14 clinical questions prioritized by the guideline panel. Of the 14 questions, 10 clinical questions assessed the effect of vitamin D vs no vitamin D in the general population throughout the lifespan, during pregnancy, and in adults with prediabetes; 1 question assessed dosing; and 3 questions addressed screening with serum 25-hydroxyvitamin D (25[OH]D). The Grading of Recommendations Assessment, Development and Evaluation approach was used to assess certainty of evidence. Results Electronic searches yielded 37 007 citations, from which we included 151 studies. In children and adolescents, low-certainty evidence suggested reduction in respiratory tract infections with empiric vitamin D. There was no significant effect on select outcomes in healthy adults aged 19 to 74 years with variable certainty of evidence. There was a very small reduction in mortality among adults older than 75 years with high certainty of evidence. In pregnant women, low-certainty evidence suggested possible benefit on various maternal, fetal, and neonatal outcomes. In adults with prediabetes, moderate certainty of evidence suggested reduction in the rate of progression to diabetes. Administration of high-dose intermittent vitamin D may increase falls, compared to lower-dose daily dosing. We did not identify trials on the benefits and harms of screening with serum 25(OH)D. Conclusion The evidence summarized in this systematic review addresses the benefits and harms of vitamin D for the prevention of disease. The guideline panel considered additional information about individuals’ and providers’ values and preferences and other important decisional and contextual factors to develop clinical recommendations.

AbstractContextHypercalcemia is a common complication of malignancy that is associated with high morbidity and mortality.ObjectiveTo support development of the Endocrine Society Clinical Practice Guideline for the treatment of hypercalcemia of malignancy in adults.MethodsWe searched multiple databases for studies that addressed 8 clinical questions prioritized by a guideline panel from the Endocrine Society. Quantitative and qualitative synthesis was performed. The Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach was used to assess certainty of evidence.ResultsWe reviewed 1949 citations, from which we included 21 studies. The risk of bias for most of the included studies was moderate. A higher proportion of patients who received bisphosphonate achieved resolution of hypercalcemia when compared to placebo. The incidence rate of adverse events was significantly higher in the bisphosphonate group. Comparing denosumab to bisphosphonate, there was no significant difference in the rate of patients who achieved resolution of hypercalcemia. Two-thirds of patients with refractory/recurrent hypercalcemia of malignancy who received denosumab following bisphosphonate therapy achieved resolution of hypercalcemia. Addition of calcitonin to bisphosphonate therapy did not affect the resolution of hypercalcemia, time to normocalcemia, or hypocalcemia. Only indirect evidence was available to address questions on the management of hypercalcemia in tumors associated with high calcitriol levels, refractory/recurrent hypercalcemia of malignancy following the use of bisphosphonates, and the use of calcimimetics in the treatment of hypercalcemia associated with parathyroid carcinoma. The certainty of the evidence to address all 8 clinical questions was low to very low.ConclusionThe evidence summarized in this systematic review addresses the benefits and harms of treatments of hypercalcemia of malignancy. Additional information about patients’ values and preferences, and other important decisional and contextual factors is needed to facilitate the development of clinical recommendations.