Assessing the Influence of Contamination on Fixed-Effect Meta-Analysis for a Continuous Outcome: A Simulation Study

Abstract

Important research questions are typically studied and analyzed more than once, often by different research teams in different locations. However, in many instances, the results of these multiple small studies are diverse and conflicting, which makes decision-making difficult. The need to arrive at decisions fostered the momentum towards synthesizing the results of these multiple studies. Therefore, meta-analysis, also referred to as the standard or traditional meta-analysis, is a statistical technique for combining the results or findings from multiple independent studies to address a specific research question. The applications of meta-analysis have been extended to many fields of research including medicine, psychology, ecology, education, business and many others.

Prior to carrying out a meta-analysis or statistically synthesizing data, a researcher must undertake a systematic review. Systematic review attempts to collate empirical evidence that fits pre-specified eligibility criteria to answer a specific research question. That is to determine which studies will be included or excluded from the analysis. Standard meta-analysis methods are used to obtain the relative efficacy (or safety) of a particular intervention versus a competing intervention in the presence of a direct or head-to-head comparison. Thus only a pair-wise comparison can be made. The outcome of these interventions could be continuous, binary or count data.

A number of methodologies related to meta-analysis, assessments of underlying assumptions and strategies for the presentation of results have been proposed by several researchers. A commonly used model for estimating effect sizes in meta-analysis is the fixed-effect model. However, various factors can determine the performance of the model which needs to be considered before using the results for decision making.

This project aimed to investigate the performance of hypothesis properties and estimation properties on selecting data points from an underlying contaminated distribution under different scenarios for modeling a continuous outcome. Different levels of contamination, levels of significance, number of studies, number of individual study sample sizes, standard deviations and effect sizes were investigated in our simulation study for a continuous outcome.

The results of our simulation study shows that, the fixed-effect meta-analytic model does not perform well in the presence of contamination. As the level of contamination in the treatment group increases, the properties of estimators and hypothesis are greatly influenced. The method performs well as expected in the absence of contamination but performs poorly as we observe 50% contamination in the treatment group regardless of the individual sample size, the number of studies, the standard deviation and the effect size.