The Role of Pragmatism in Explaining Heterogeneity in Meta-Analyses of Randomized Trials: A Methodological Review

Abstract

Introduction: There has been increasing interest in evidence from pragmatic trials as healthcare providers and decision makers must determine if available evidence can be translated and used in real world practice. As a result, a number of tools have been developed to help researchers design and appraise randomized controlled trials (RCTs) within the pragmatic-explanatory continuum. It is unclear what role pragmatism plays in heterogeneity and if pragmatic and explanatory trials should be pooled in meta-analyses of systematic reviews.

Objectives: Our primary objective was to explore the role of pragmatism (based on the Pragmatic-Explanatory Continuum Indicator Summary-2 [PRECIS-2] score) as a source of heterogeneity in Cochrane systematic reviews with at least substantial heterogeneity (I2≥ 50%). Our secondary objective was to compare and contrast the application of the established PRECIS-2 tool to the newly developed Rating of Included Trials on the Efficacy-Effectiveness Spectrum (RITES) tool.

Methods: We conducted a cross-sectional methodological review on systematic reviews of RCTs published in the Cochrane Library from January 1, 2014 to January 1, 2017. Included systematic reviews had a minimum of 10 RCTs in the meta-analysis of the primary outcome and at least moderate heterogeneity (I2≥ 50%). Of the eligible systematic reviews, a random selection of 10 were included for quantitative evaluation. In each systematic review, RCTs were scored using the PRECIS-2 and RITES tools, in duplicate, to determine the amount of pragmatism. Meta-regression modelling was performed to evaluate how much variability in heterogeneity (quantified by I2) was due to pragmatism. Inter-rater reliability of both PRECIS-2 and RITES was measured using the intraclass correlation coefficient and Spearman’s correlation coefficient was used to determine the strength of the relation between PRECIS-2 and RITES.

Results: Ten systematic reviews from nine Cochrane Review Groups were included in the quantitative analysis. The reviews included an average of 13 RCTs (standard deviation=2.6) for a total of 132 RCTs of which 128 could be obtained. When the PRECIS-2 summary score was entered as a covariate in random effects meta-regression models for each systematic review, there were minimal changes in heterogeneity. The changes in I2 ranged from 0.2% to 13.3%.

Conclusion: Based on these findings it appears pragmatism as measured by PRECIS-2 does not explain heterogeneity in systematic reviews, therefore pooling of pragmatic and explanatory RCTs is unlikely to be detrimental to meta-analyses.