Dahm, Philipp

Abstract Introduction While certainty of evidence assessment is key to a rigorous and transparent systematic review, it is unknown how – and how frequently – it is assessed in systematic reviews. The objective of this study was to examine the prevalence and approaches used for certainty of evidence assessment in systematic reviews published in high‐impact medicine journals over the past 11 years. Methods A PubMed search and hand‐searching of relevant journal websites identified systematic reviews published between 24 January 2013 and 23 January 2024 in any of the ten highest‐impact journals in the General and Internal Medicine category of the Journal Citation Report. Two reviewers independently selected any systematic review related to health outcomes assessing certainty of evidence using any method. We extracted data related to review characteristics, certainty of evidence and risk of bias/methodological quality assessment frameworks, and reported consideration of certainty of evidence domains. Logistic regression examined year of publication to determine whether the prevalence of certainty of evidence assessment changed over time. Results Of 1,023 included reviews, 346 (33.8%) assessed certainty of evidence. Prevalence of certainty of evidence assessment increased over time (0.16 ± 0.2; p < .001). Most (89.3%) of reviews used the Grading of Recommendations Assessment, Development and Evaluation (GRADE) framework to assess certainty of evidence. Conclusion Only one in three systematic reviews published in the highest‐impact medical journals over the past 11 years assessed certainty of evidence, though prevalence increased over time. The use of specific domains within each certainty of evidence framework was not clearly described in all reviews.

When one initially targets the null effect and the point estimate falls close to the null, two challenges exist in rating certainty of evidence. First, when the point estimate is near the null and the data, therefore, suggests little or no effect, rating certainty in a benefit or harm is misleading. Second, since in general the narrower the confidence interval (CI) the more precise the estimate, if the CI is narrow, rating down for imprecision due simply to crossing the null is inappropriate. This paper addresses these issues and provides a solution: to revise the target of certainty rating from a non-zero effect to a little or no effect. This solution requires estimating a range in which the minimal important difference (MID) for benefit and an MID for harm might lie, and thus establishing a range that represents little or no effect. If GRADE (Grading of Recommendations, Assessment, Development, and Evaluations) users are confident that the point estimate represents an effect less than the smallest plausible MID, they will revise their target and rate certainty to a little or no effect. If the entire CI falls within the range of little or no effect, they will not rate down for imprecision. Otherwise (if the CI includes an important effect), they will rate down. Using the solution provided in this paper GRADE users can make an optimal choice of the target of certainty rating.

Objectives To assess the effects of percutaneous nephrolithotomy (PCNL) vs retrograde intrarenal surgery (RIRS) for the treatment of renal stones in adults. Methods We performed a comprehensive search of the Cochrane Library, MEDLINE, Embase, three other databases, trials registries, other sources of the grey literature, and conference proceedings up to 23 March 2023. We applied no restrictions on publication language or status. Screening, data extraction, risk‐of‐bias assessment, and certainty of evidence (CoE) rating using the Grading of Recommendations Assessment, Development and Evaluations (GRADE) approach were done in duplicate by two independent reviewers. This co‐publication focuses on the primary outcomes of this review only. Results We included 42 trials that met the inclusion criteria. Stone‐free rate (SFR): PCNL may improve SFRs (risk ratio [RR] 1.13, 95% confidence interval [CI] 1.08–1.18; I 2 = 71%; 39 studies, 4088 participants; low CoE). Major complications: PCNL probably has little to no effect on major complications (RR 0.86, 95% CI 0.59–1.25; I 2 = 15%; 34 studies, 3649; participants; moderate CoE) compared to RIRS. Need for secondary interventions: PCNL may reduce the need for secondary interventions (RR 0.31, 95% CI 0.17–0.55; I 2 = 61%; 21 studies, 2005 participants; low CoE) compared to RIRS. Conclusion Despite shortcomings in most studies that lowered our certainty in the estimates of effect to mostly very low or low, we found that PCNL may improve SFRs and reduce the need for secondary interventions while not impacting major complications. Ureteric stricture rates may be similar compared to RIRS. We expect the findings of this review to be helpful for shared decision‐making about management choices for individuals with renal stones.