Application of Thresholds for Rating Certainty of Evidence Using the GRADE Approach: A Systematic Survey of Systematic Reviews

Abstract

Background: Systematic review authors using the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach must establish thresholds to assess certainty of evidence.

Objectives: To describe how systematic review authors applied thresholds and thus inform the perceived usefulness of the available thresholds for authors using GRADE.

Methods: We conducted a systematic survey sampling the 200 most recently published Cochrane and 200 non-Cochrane reviews that used GRADE. We documented how review authors used thresholds and the thresholds they chose (the null, minimally important difference [MID], or MID, moderate and large effect thresholds).

Results: Among the sampled reviews, 118 of 200 (59%) Cochrane and 63 of 200 (31.5%) non-Cochrane reviews used thresholds. Of these, we identified the threshold type in 112 of 118 (94.9%) and 55 of 63 (87.3%), in turn. Among these, 55 of 112 (49.1%) Cochrane and 26 of 55 (47.3%) non-Cochrane reviews used the null, while 54 of 112 (48.2%) and 25 of 55 (45.5%), respectively, used the MID. Few used the MID, moderate, and large effects thresholds (2 of 112 [1.8%] Cochrane; 3 of 55 [5.5%] non-Cochrane) or applied the null for some outcomes and the MID for others (1 of 112 [0.9%] Cochrane; 2 of 55 [3.6%] non-Cochrane). Among reviews with identifiable thresholds, 15 of 112 (13.4%) Cochrane and 17 of 55 (30.9%) non-Cochrane reviews mentioned or referred to the thresholds in their methods sections.

Conclusions: Reporting and application of thresholds in systematic reviews remain limited. Over 40% of Cochrane and more than two-thirds of non-Cochrane reviews showed no apparent threshold use, reflecting either a lack of transparency in reporting or, more likely, the absence of threshold usage. When thresholds were identifiable, the null and MID were most commonly and equally used, suggesting that review authors consider both valuable for rating certainty of evidence.