EVALUATING THE CREDIBILITY OF EFFECT MODIFICATION CLAIMS IN RANDOMIZED CONTROLLED TRIALS AND META-ANALYSES

Abstract

Background: Many randomized controlled trials (RCTs) and meta-analyses include analyses of effect modification (also known as subgroup, interaction, or moderation analyses). Methodologists have widely acknowledged the challenges in deciding whether an apparent effect modification is credible or likely the result of chance or bias. Various sets of credibility criteria are available (Chapter 2 provides an example) but are inconsistent, vague in wording, lack guidance for deciding on overall credibility, and have not been systematically tested.

Objective: To systematically develop a formal instrument to assess the credibility of effect modification analyses (ICEMAN) in RCTs and meta-analyses of RCTs.

Methods: Key steps in the development process included 1) a systematic survey of the literature to identify available criteria, rationales, and previous instruments, 2) a formal consensus study among 10 leading experts, and 3) a formal user-testing study to refine the instrument based on interviews with trial investigators, systematic reviewer authors, and journal editors who applied drafts of the instrument to published claims of effect modification.

Results: The systematic survey identified 150 relevant publications, 36 candidate credibility criteria with associated rationales, and 30 existing checklists (Chapter 3). The consensus study consisted of two main video conferences and multiple rounds of written discussion. The user-testing involved 17 users (including systematic review authors, trial investigators, and journal editors) who suggested substantial improvements based on detailed interviews. The final instrument provides separate versions for RCTs (five core questions) and meta-analyses (eight core questions) with explicit response options, and an overall credibility rating ranging from very low to high credibility. A detailed manual provides rationales, supporting references, examples from the literature, and suggestions for use in combination with other quality appraisal tools and reporting (Chapter 4).

Discussion: ICEMAN is a rigorously developed instrument to evaluate claims of effect modification and addresses the main limitations of previous approaches.