Advances in the meta-analysis of heterogeneous clinical trials I: The inverse variance heterogeneity model
Introduction
In the era of evidence based medicine, meta-analyses of well-designed and executed randomized controlled trials have the potential to provide high levels of evidence to support therapeutic interventions in all areas of clinical practice. Despite the potential of the outcome of such trials to guide decision making, they may sometimes fail to produce credible conclusive results or may disagree if there were multiple independent trials that investigate the same clinical question. In this situation, a meta-analysis of the trial results has the potential to combine conceptually similar and independent studies with the purpose of deriving more reliable statistical conclusions (based on a much larger sample data) than any of the individual studies [1], [2]. Today, clinical decision making relies heavily on this methodology as is evident by the work of the Cochrane collaboration and the high volume of publications for meta-analyses outside the collaboration [3].
Meta-analyses, customarily, are performed using either the fixed effect (FE) or the random effects (RE) models [4], [5]. The FE model of meta-analysis is underpinned by the assumption that one identical true treatment effect is common to every study included in the meta-analysis, from which they depart under the influence of the random error only [4]. As such, only within-study variation is considered to be present. In practice, this model ensures that the larger studies (with the lowest probability of random error) have the greatest influence on the pooled estimate. The drawback however is that this model also demonstrates increasing overdispersion as heterogeneity increases. Overdispersion here refers to an estimator that has a greater observed variance (true variance often assessed through simulation) than that theoretically expected which is based on the statistical model (used in the confidence interval computation).
In an attempt to tackle the issue of overdispersion, the RE approach was suggested which attempts to create a more fully specified model [6]. This model makes the additional assumption that the true treatment effects in the individual studies are different from each other and these differences follow a normal distribution with a common variance. The assumption of normally distributed random effects is not justified [7] because the underlying effects included in the meta-analysis do not constitute a random sample from the population. This model nevertheless ignores the need for randomization in statistical inference [8] and the variance of these underlying effects is usually approximated by a moment-based estimator [9]. The application of this common variance to the model has the unintended effect of redistributing the study weights in only one direction: from larger to smaller studies [10]. Thus the studies with the lowest probability of random error are penalized and do not influence the combined estimates as strongly. The inclusion of this estimated common between-studies variance also seems to be the mechanism that attempts to address overdispersion with increasing heterogeneity, yielding wider confidence intervals and lesser statistical significance than would be attained through the conventional fixed effect model. Yet, given the faulty assumptions, it does not work as expected and as heterogeneity increases, the coverage of the confidence interval drops well below the nominal level [7], [11]. Though corrections have been suggested [12], [13] they have not been easy to implement and this estimator substantially underestimates the statistical error and remains potentially overconfident in its conclusions [13], [14].
A careful look at the inputs to these models demonstrates that both use inverse variance weighting to decrease observed variance of the estimator. However, the approach taken with the RE estimator disadvantages it because as heterogeneity increases, the inverse variance weights are moved towards equality thus increasing estimator variance. This also leads to a failure to specify the theoretical variance correctly so that it now falls short of the observed variance and nominal coverage is not achieved.
While alternative frequentist RE models that attempt to improve on the conventional theoretical model in various ways have been described in the literature [11], they all continue to be based on the assumption of normally distributed random effects which, as mentioned above, leads to several problems. There is therefore the need for a better method and this paper argues that the random effects model should be replaced by a distributional assumption free model. Such a model has been proposed by us as a variant of the quality effects model that sets quality to equal (called the IVhet model) [15]. This paper reviews the model's theoretical construct and presents an evaluation of its performance using standard performance measures [16].
Section snippets
Difference between empirically weighted means and the arithmetic mean
Consider a collection of k independent studies, the jth of which has estimated effect size which varies from its true effect size, δj through random error. Also consider that the true effects, δjs, also vary from an underlying common effect, θ, through bias. There is the possibility of some diversity of true effects (which remain similar) across studies (in which case θ would simply be the mean of the true (unbiased) effects). A greater diversity that leads to dissimilarity of effects would
Variance of the estimator under different models
It is clear from the previous discussion that overdispersion is a problem with both (RE and FE) estimators, more so with the FE estimator. The variance of any weighted estimator [] in general is given by:where ωjs are the weights that sum to 1. When there is heterogeneity, the observed variance (or true variance) of the FE model and arithmetic mean (AM) estimator are larger than that computed through the theoretical model, consequently the coverage probability is
Examining estimator performance using simulation
We now proceed to examine the performance of the three estimators under varying degrees of heterogeneity. These estimators are what we now call the inverse variance (fixed effect) heterogeneity (IVhet) estimator, the arithmetic mean heterogeneity (AMhet) estimator and the RE estimator (see Table 1 for the mathematical form of the three estimators and their variances). The log odds ratio is used as the effect size (but the models can work with any of the normally distributed effect sizes) and
Real data examples from the literature
We also looked at the controversial meta-analysis of intravenous magnesium for prevention of early mortality after myocardial infarction mentioned earlier which consisted of 19 English language randomized trials (published prior to June 2006) [10]. Early mortality was defined as occurring in hospital during the acute admission phase or within 35 days of onset of myocardial infarction. When the meta-analytic estimates were computed using the three methods, they were most extreme with the AMhet
Discussion
The IVhet model estimate differs from the RE model estimate in three perspectives: Pooled IVhet estimates favor larger trials (as opposed to penalizing larger trials in the RE model), have a more conservative confidence interval with correct coverage probability and exhibit a lesser observed (true) variance irrespective of the degree of heterogeneity. While the RE model represents the conventional method of fitting the overdispersed study data, it is clear from the simulated results that using
Conflict of interest
JJB owns Epigear International Pty Ltd. which sells the Ersatz Monte-Carlo simulation software used in this study.
References (35)
Meta-analyses and overviews of randomised trials
Blood Rev.
(1995)- et al.
Meta-analysis in clinical trials
Control. Clin. Trials
(1986) - et al.
Combining heterogenous studies using the random-effects model is a mistake and leads to inconclusive meta-analyses
J. Clin. Epidemiol.
(2011) - et al.
Incorporating variations in the quality of individual randomized trials into meta-analysis
J. Clin. Epidemiol.
(1992) - et al.
Advances in the meta-analysis of heterogenous clinical trials II: the quality effects model
Contemp. Clin. Trials
(2015) - et al.
Primer: strengths and weaknesses of meta-analysis
Nat. Clin. Pract. Rheumatol.
(2008) Formalizing syntheses of medical knowledge: the rise of meta-analyses and systematic reviews
Perspect. Sci.
(2012)- et al.
Fixed- versus random-effects models in meta-analysis: model properties and an empirical comparison of differences in results
Br. J. Math. Stat. Psychol.
(2009) - et al.
Fixed- and random-effects models in meta-analysis
Psychol. Methods
(1998) - et al.
An overdispersion model in meta-analysis
Stat. Model.
(2014)
A comparison of statistical methods for meta-analysis
Stat. Med.
A comparison of fixed-effects and mixed (random-effects) models for meta-analysis tests of moderator variable effects
Psychol. Methods
Random-effects meta-analysis of inconsistent effects: a time for change
Ann. Intern. Med.
A simple method for inference on an overall effect in meta-analysis
Stat. Med.
Confidence intervals for a random-effects meta-analysis based on Bartlett-type corrections
Stat. Med.
Random-effects meta-analyses are not always conservative
Am. J. Epidemiol.
Simulation comparison of the quality effects and random effects methods of meta-analysis
Epidemiology
Cited by (421)
Associations of long-term particulate matter exposure with cardiometabolic diseases: A systematic review and meta-analysis
2023, Science of the Total Environment