Elsevier

Journal of Clinical Epidemiology

Volume 118, February 2020, Pages 42-54
Journal of Clinical Epidemiology

Original Article
Excluding non-English publications from evidence-syntheses did not change conclusions: a meta-epidemiological study

https://doi.org/10.1016/j.jclinepi.2019.10.011Get rights and content

Abstract

Objectives

We aimed to assess whether limiting the inclusion criteria solely to English-language publications affected the overall conclusions of evidence syntheses.

Study Design and Setting

Our analyses used a dataset of a previous methods study that included 59 randomly selected Cochrane intervention reviews with no language restrictions. First, we ascertained the publication language of all 2,026 included publications. Next, we excluded studies based on the following criteria: (1) publication solely in non-English language, or (2) main publication (in case of multiple publications of the same study) in non-English language. We then re-calculated meta-analyses for outcomes that were presented in the main summary of findings tables of the Cochrane reports. If the direction of the effect estimate or the statistical significance changed, authors of the respective Cochrane reviews were consulted to assess whether the new evidence base would have changed their conclusions. The primary outcome of our analyses examined the proportion of conclusions that would change with the exclusion of non-English publications. We set the threshold for the approach as noninferior if the upper limit of the 95% confidence interval of the proportion of changed conclusions did not cross a margin of 10%.

Results

Across all 59 Cochrane reviews, 29 (49%) included 80 non-English publications. For 16 (27%) of these Cochrane reviews, the exclusion of non-English publications resulted in the exclusion of at least one study. In the remaining 13 Cochrane reviews, the non-English publications were not the only or main publication of the study or they did not contribute to the main summary of the findings table, so their exclusion did not result in an exclusion of the study.

Overall, the exclusion of non-English publications led to the exclusion of 31 studies contributing to 40 outcomes. For 38 of the 40 outcomes, the exclusion of non-English studies did not markedly alter the size or direction of effect estimates or statistical significance. In two outcomes, the statistical significance changed, but authors would have still drawn the same conclusion, albeit with less certainty. Thus, the proportion of changed conclusions in our sample was 0.0% (95% CI 0.0–0.6), which indicated the noninferiority of the approach. However, the majority of excluded studies were small.

Conclusion

Exclusion of non-English publications from systematic reviews on clinical interventions had a minimal effect on overall conclusions and could be a viable methodological shortcut, especially for rapid reviews.

Introduction

Systematic reviews that adhere to high-level methodological standards are considered the most reliable and valid support for decision-making in health care. Systematic reviews, however, require a considerable amount of time (up to 2 years) and resources [1,2]. By accelerating the production of evidence syntheses, researchers can better meet the time-sensitive needs of decision-makers. Because rapid reviews often can be completed within a few weeks or months, they have emerged as an alternative evidence synthesis tool to systematic reviews, which tend to consume more time, and thus, resources. A potential risk of speeding up the review process by streamlining the methodological approach is that the generated findings could be less reliable than those of the systematic reviews [3,4].

One option to save time and resources is to limit the reviews to English-language publications. This methodological shortcut is already commonly used in practice. Tricco et al. reported that 49% of their analyzed rapid reviews applied language restrictions [5]. According to Page et al., even systematic reviews often apply language limitations, with 37% of non-Cochrane reviews, including only English-language publications [6]. Another analysis of 69 non-Cochrane systematic reviews showed that only 45% included non-English publications [7].

This approach, however, risks introducing language bias [8] and devaluing research, which has not been published in English [9,10]. In addition, such an approach is not recommended in the method guidelines of leading evidence-based medicine organizations, such as Cochrane [11], the Campbell Collaboration [12], or the Agency for Healthcare Research and Quality [13].

At the same time, the impact of language bias may have been decreasing in recent years, as English increasingly becomes a universal language in scientific research and cooperation. Galandi et al. [14], for instance, detected a large decline in German language publications of randomized controlled trials (RCTs) over the last 10 years, indicating a shift to English as the main publication language in German-speaking countries.

A systematic review identified five methods of studies assessing the impact of restricting systematic reviews to English-only publications on meta-analyses [15,16]. None of these five studies [8,[17], [18], [19], [20]] identified any evidence that language restrictions in systematic reviews of conventional medical topics markedly changed the results of meta-analyses. However, Pham et al. discovered that such language restrictions had a large impact on systematic reviews of complementary and alternative medicine resulting in a 63% smaller effect estimate than language-inclusive reviews [20]. Additionally, stratified analyses revealed that the impact of non-English publications varies according to medical specialties and plays a larger role in fields such as psychiatry, rheumatology, and orthopedics than it does in other areas of medicine [8].

Whether there is a difference in the methodological quality of studies published in English or other languages is unclear. While two methods studies found no significant difference between completeness of reporting (randomization, double-blinding, withdrawals, allocation concealment, drop-outs) of RCTs published in English, French, German, Italian, and Spanish [18,21], Jüni and colleagues showed that trials published in English had a lower risk of bias because of better allocation concealment and blinding [19].

To date, the impact of an English-only approach has been assessed on single outcomes of systematic reviews but not on overall conclusions across multiple bodies of evidence. The Cochrane handbook acknowledges that while the potential impact of non-English publications on effect sizes of meta-analysis might be small, it is difficult to tell how the exclusion of non-English publications affects a given systematic review and its conclusions [22]. Therefore, the aim of our study was to assess whether limiting the inclusion criteria to English-language publications affected the overall conclusions in a set of systematic reviews on diverse interventional medical topics.

Section snippets

Materials and methods

This study is a retrospective analysis of a dataset from a methods study on the impact of abbreviated literature search approaches [23,24]. We developed an analysis plan a priori but did not publish it (see Appendix).

Results

The 59 Cochrane reviews included a total of 2,026 publications referring to 1,281 primary studies [[29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47], [48], [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60], [61], [62], [63], [64], [65], [66], [67], [68], [69], [70], [71], [72], [73], [74], [75], [76], [77], [78], [79], [80], [81], [82], [83], [84], [85], [86], [87]]. Fifty-one percent of the reviews assessed

Discussion

To the best of our knowledge, this is the first study to assess the impact of excluding non-English publications on overall conclusions of systematic reviews across a variety of clinical interventions. Our results indicate that the exclusion of non-English publications from evidence syntheses leads to the same conclusions as a language-inclusive approach.

Our results are consistent with other studies that have focused primarily on the effect of language restrictions on meta-analyses [8,[15], [16]

Conclusion

The inclusion of only English-language publications seems to be a reliable methodological shortcut when conducting a rapid review of clinical topics addressing conventional medical interventions. Further research is needed to distinguish when such an approach is a viable option and when it is preferable to conduct comprehensive, systematic language-inclusive reviews. Decisions that demand the highest possible certainty or reviews on topics for which many relevant studies can be expected to be

CRediT authorship contribution statement

B. Nussbaumer-Streit: Conceptualization, Data curation, Formal analysis, Funding acquisition, Writing - original draft. I. Klerings: Conceptualization, Data curation, Formal analysis, Writing - review & editing. A.I. Dobrescu: Data curation, Formal analysis, Writing - review & editing. E. Persad: Data curation, Formal analysis, Writing - review & editing. A. Stevens: Conceptualization, Data curation, Funding acquisition, Writing - review & editing. C. Garritty: Conceptualization, Data curation,

Acknowledgments

We would like to thank Cochrane for funding this research project, and the Cochrane Editorial Unit for providing data files of the Cochrane reviews. Thank you also to the contacted authors for assessing the impact of the new evidence base on their original conclusions. Furthermore, we would like to thank Sandra Hummel for administrative support throughout this project and Dawn Gartlehner for proofreading the manuscript.

References (95)

  • L.G. Hartling et al.

    EPC methods: an exploration of methods and context for the production of rapid reviews. Scientific resource center

    (2015)
  • A. Cameron et al.

    Rapid versus full systematic reviews: an inventory of current methods and practice in health technology assessment. ASERNIP-S Report No. 60

    (2007)
  • A.C. Tricco et al.

    A scoping review of rapid review methods

    BMC Med

    (2015)
  • M.J. Page et al.

    Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study

    PLoS Med

    (2016)
  • Z. Wang et al.

    Systematic reviews with language restrictions and no author contact have lower overall credibility: a methodology study

    Clin Epidemiol

    (2015)
  • I. Baussano et al.

    Does language matter? A case study of epidemiological and public health journals, databases and professional education in French, German and Italian

    Emerg Themes Epidemiol

    (2008)
  • Methodological expectations of Campbell Collaboration intervention reviews: Conduct standards

  • M. McDonagh et al.

    Avoiding bias in selecting studies. Methods guide for comparative effectiveness reviews

    (2013)
  • D. Galandi et al.

    The demise of the randomised controlled trial: bibliometric study of the German-language health care literature, 1948 to 2004

    BMC Med Res Methodol

    (2006)
  • A. Morrison et al.

    English-language restriction when conducting systematic review-based meta-analyses: systematic review of published studies

    (2009)
  • A. Morrison et al.

    The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies

    Int J Technol Assess Health Care

    (2012)
  • D. Moher et al.

    The inclusion of reports of randomised trials published in languages other than English in systematic reviews

    Health Technol Assess

    (2003)
  • P. Jüni et al.

    Direction and impact of language bias in meta-analyses of controlled trials: empirical study

    Int J Epidemiol

    (2002)
  • B. Nussbaumer-Streit et al.

    Assessing the validity of abbreviated literature searches for rapid reviews: protocol of a non-inferiority and meta-epidemiologic study

    Syst Rev

    (2016)
  • MEDLINE®/PubMed® data element (field) descriptions

  • What field codes can I use in Embase?

  • Excel 2016 [Software]

    (2016)
  • G. Wagner et al.

    Trading certainty for speed–how much uncertainty are decisionmakers and guideline developers willing to accept when using rapid reviews: an international survey

    BMC Med Res Methodol

    (2017)
  • P. Adams Stephen et al.

    Lipid-lowering efficacy of rosuvastatin

    Cochrane Database Syst Rev

    (2014)
  • A. Akl Elie et al.

    Oral anticoagulation in patients with cancer who have no therapeutic or prophylactic indication for anticoagulation

    Cochrane Database Syst Rev

    (2014)
  • A. Akl Elie et al.

    Parenteral anticoagulation in ambulatory patients with cancer

    Cochrane Database Syst Rev

    (2014)
  • A. Akl Elie et al.

    Anticoagulation for the long-term treatment of venous thromboembolism in patients with cancer

    Cochrane Database Syst Rev

    (2014)
  • S. Alkhawaja et al.

    Post-pyloric versus gastric tube feeding for preventing pneumonia and improving nutritional outcomes in critically ill adults

    Cochrane Database Syst Rev

    (2015)
  • L. Anderson et al.

    Exercise-based cardiac rehabilitation for coronary heart disease

    Cochrane Database Syst Rev

    (2016)
  • B.N.G. Andriolo et al.

    Early versus late tracheostomy for critically ill patients

    Cochrane Database Syst Rev

    (2015)
  • J. Barlow et al.

    Parent-infant psychotherapy for improving parental and infant mental health

    Cochrane Database Syst Rev

    (2015)
  • J.S. Birks et al.

    Rivastigmine for Alzheimer's disease

    Cochrane Database Syst Rev

    (2015)
  • H. Blessberger et al.

    Perioperative beta-blockers for preventing surgery-related mortality and morbidity

    Cochrane Database Syst Rev

    (2014)
  • H.M.P. Boardman et al.

    Hormone therapy for preventing cardiovascular disease in post-menopausal women

    Cochrane Database Syst Rev

    (2015)
  • J. Bradt et al.

    Music for stress and anxiety reduction in coronary heart disease patients

    Cochrane Database Syst Rev

    (2013)
  • R. Buchbinder et al.

    Percutaneous vertebroplasty for osteoporotic vertebral compression fracture

    Cochrane Database Syst Rev

    (2015)
  • J. Chen et al.

    Sulfasalazine for ankylosing spondylitis

    Cochrane Database Syst Rev

    (2014)
  • C. Claro Juan et al.

    Amiodarone versus other pharmacological interventions for prevention of sudden cardiac death

    Cochrane Database Syst Rev

    (2015)
  • G. Cormick et al.

    Calcium supplementation for prevention of primary hypertension

    Cochrane Database Syst Rev

    (2015)
  • L. Dowman et al.

    Pulmonary rehabilitation for interstitial lung disease

    Cochrane Database Syst Rev

    (2014)
  • Cited by (0)

    Conflict of interest statement: None of the authors report any conflicts of interest with respect to the topic of this manuscript.

    Funding statement: This work was supported by funds from the Cochrane Collaboration.

    View full text