Elsevier

Contemporary Clinical Trials

Volume 45, Part A, November 2015, Pages 130-138
Contemporary Clinical Trials

Advances in the meta-analysis of heterogeneous clinical trials I: The inverse variance heterogeneity model

https://doi.org/10.1016/j.cct.2015.05.009Get rights and content

Abstract

This article examines an improved alternative to the random effects (RE) model for meta-analysis of heterogeneous studies. It is shown that the known issues of underestimation of the statistical error and spuriously overconfident estimates with the RE model can be resolved by the use of an estimator under the fixed effect model assumption with a quasi-likelihood based variance structure — the IVhet model. Extensive simulations confirm that this estimator retains a correct coverage probability and a lower observed variance than the RE model estimator, regardless of heterogeneity. When the proposed IVhet method is applied to the controversial meta-analysis of intravenous magnesium for the prevention of mortality after myocardial infarction, the pooled OR is 1.01 (95% CI 0.71–1.46) which not only favors the larger studies but also indicates more uncertainty around the point estimate. In comparison, under the RE model the pooled OR is 0.71 (95% CI 0.57–0.89) which, given the simulation results, reflects underestimation of the statistical error. Given the compelling evidence generated, we recommend that the IVhet model replace both the FE and RE models. To facilitate this, it has been implemented into free meta-analysis software called MetaXL which can be downloaded from www.epigear.com.

Introduction

In the era of evidence based medicine, meta-analyses of well-designed and executed randomized controlled trials have the potential to provide high levels of evidence to support therapeutic interventions in all areas of clinical practice. Despite the potential of the outcome of such trials to guide decision making, they may sometimes fail to produce credible conclusive results or may disagree if there were multiple independent trials that investigate the same clinical question. In this situation, a meta-analysis of the trial results has the potential to combine conceptually similar and independent studies with the purpose of deriving more reliable statistical conclusions (based on a much larger sample data) than any of the individual studies [1], [2]. Today, clinical decision making relies heavily on this methodology as is evident by the work of the Cochrane collaboration and the high volume of publications for meta-analyses outside the collaboration [3].

Meta-analyses, customarily, are performed using either the fixed effect (FE) or the random effects (RE) models [4], [5]. The FE model of meta-analysis is underpinned by the assumption that one identical true treatment effect is common to every study included in the meta-analysis, from which they depart under the influence of the random error only [4]. As such, only within-study variation is considered to be present. In practice, this model ensures that the larger studies (with the lowest probability of random error) have the greatest influence on the pooled estimate. The drawback however is that this model also demonstrates increasing overdispersion as heterogeneity increases. Overdispersion here refers to an estimator that has a greater observed variance (true variance often assessed through simulation) than that theoretically expected which is based on the statistical model (used in the confidence interval computation).

In an attempt to tackle the issue of overdispersion, the RE approach was suggested which attempts to create a more fully specified model [6]. This model makes the additional assumption that the true treatment effects in the individual studies are different from each other and these differences follow a normal distribution with a common variance. The assumption of normally distributed random effects is not justified [7] because the underlying effects included in the meta-analysis do not constitute a random sample from the population. This model nevertheless ignores the need for randomization in statistical inference [8] and the variance of these underlying effects is usually approximated by a moment-based estimator [9]. The application of this common variance to the model has the unintended effect of redistributing the study weights in only one direction: from larger to smaller studies [10]. Thus the studies with the lowest probability of random error are penalized and do not influence the combined estimates as strongly. The inclusion of this estimated common between-studies variance also seems to be the mechanism that attempts to address overdispersion with increasing heterogeneity, yielding wider confidence intervals and lesser statistical significance than would be attained through the conventional fixed effect model. Yet, given the faulty assumptions, it does not work as expected and as heterogeneity increases, the coverage of the confidence interval drops well below the nominal level [7], [11]. Though corrections have been suggested [12], [13] they have not been easy to implement and this estimator substantially underestimates the statistical error and remains potentially overconfident in its conclusions [13], [14].

A careful look at the inputs to these models demonstrates that both use inverse variance weighting to decrease observed variance of the estimator. However, the approach taken with the RE estimator disadvantages it because as heterogeneity increases, the inverse variance weights are moved towards equality thus increasing estimator variance. This also leads to a failure to specify the theoretical variance correctly so that it now falls short of the observed variance and nominal coverage is not achieved.

While alternative frequentist RE models that attempt to improve on the conventional theoretical model in various ways have been described in the literature [11], they all continue to be based on the assumption of normally distributed random effects which, as mentioned above, leads to several problems. There is therefore the need for a better method and this paper argues that the random effects model should be replaced by a distributional assumption free model. Such a model has been proposed by us as a variant of the quality effects model that sets quality to equal (called the IVhet model) [15]. This paper reviews the model's theoretical construct and presents an evaluation of its performance using standard performance measures [16].

Section snippets

Difference between empirically weighted means and the arithmetic mean

Consider a collection of k independent studies, the jth of which has estimated effect size δ^j which varies from its true effect size, δj through random error. Also consider that the true effects, δjs, also vary from an underlying common effect, θ, through bias. There is the possibility of some diversity of true effects (which remain similar) across studies (in which case θ would simply be the mean of the true (unbiased) effects). A greater diversity that leads to dissimilarity of effects would

Variance of the estimator under different models

It is clear from the previous discussion that overdispersion is a problem with both (RE and FE) estimators, more so with the FE estimator. The variance of any weighted estimator [varθ^w] in general is given by:varθ^w=j=1kωj2varδ^j,where ωjs are the weights that sum to 1. When there is heterogeneity, the observed variance (or true variance) of the FE model and arithmetic mean (AM) estimator are larger than that computed through the theoretical model, consequently the coverage probability is

Examining estimator performance using simulation

We now proceed to examine the performance of the three estimators under varying degrees of heterogeneity. These estimators are what we now call the inverse variance (fixed effect) heterogeneity (IVhet) estimator, the arithmetic mean heterogeneity (AMhet) estimator and the RE estimator (see Table 1 for the mathematical form of the three estimators and their variances). The log odds ratio is used as the effect size (but the models can work with any of the normally distributed effect sizes) and

Real data examples from the literature

We also looked at the controversial meta-analysis of intravenous magnesium for prevention of early mortality after myocardial infarction mentioned earlier which consisted of 19 English language randomized trials (published prior to June 2006) [10]. Early mortality was defined as occurring in hospital during the acute admission phase or within 35 days of onset of myocardial infarction. When the meta-analytic estimates were computed using the three methods, they were most extreme with the AMhet

Discussion

The IVhet model estimate differs from the RE model estimate in three perspectives: Pooled IVhet estimates favor larger trials (as opposed to penalizing larger trials in the RE model), have a more conservative confidence interval with correct coverage probability and exhibit a lesser observed (true) variance irrespective of the degree of heterogeneity. While the RE model represents the conventional method of fitting the overdispersed study data, it is clear from the simulated results that using

Conflict of interest

JJB owns Epigear International Pty Ltd. which sells the Ersatz Monte-Carlo simulation software used in this study.

References (35)

  • S.E. Brockwell et al.

    A comparison of statistical methods for meta-analysis

    Stat. Med.

    (2001)
  • R.C. Overton

    A comparison of fixed-effects and mixed (random-effects) models for meta-analysis tests of moderator variable effects

    Psychol. Methods

    (1998)
  • J.E. Cornell et al.

    Random-effects meta-analysis of inconsistent effects: a time for change

    Ann. Intern. Med.

    (2014)
  • S.E. Brockwell et al.

    A simple method for inference on an overall effect in meta-analysis

    Stat. Med.

    (2007)
  • H. Noma

    Confidence intervals for a random-effects meta-analysis based on Bartlett-type corrections

    Stat. Med.

    (2011)
  • C. Poole et al.

    Random-effects meta-analyses are not always conservative

    Am. J. Epidemiol.

    (1999)
  • S.A. Doi et al.

    Simulation comparison of the quality effects and random effects methods of meta-analysis

    Epidemiology

    (2015)
  • Cited by (421)

    View all citing articles on Scopus
    View full text