Original ArticleExcluding non-English publications from evidence-syntheses did not change conclusions: a meta-epidemiological study
Introduction
Systematic reviews that adhere to high-level methodological standards are considered the most reliable and valid support for decision-making in health care. Systematic reviews, however, require a considerable amount of time (up to 2 years) and resources [1,2]. By accelerating the production of evidence syntheses, researchers can better meet the time-sensitive needs of decision-makers. Because rapid reviews often can be completed within a few weeks or months, they have emerged as an alternative evidence synthesis tool to systematic reviews, which tend to consume more time, and thus, resources. A potential risk of speeding up the review process by streamlining the methodological approach is that the generated findings could be less reliable than those of the systematic reviews [3,4].
One option to save time and resources is to limit the reviews to English-language publications. This methodological shortcut is already commonly used in practice. Tricco et al. reported that 49% of their analyzed rapid reviews applied language restrictions [5]. According to Page et al., even systematic reviews often apply language limitations, with 37% of non-Cochrane reviews, including only English-language publications [6]. Another analysis of 69 non-Cochrane systematic reviews showed that only 45% included non-English publications [7].
This approach, however, risks introducing language bias [8] and devaluing research, which has not been published in English [9,10]. In addition, such an approach is not recommended in the method guidelines of leading evidence-based medicine organizations, such as Cochrane [11], the Campbell Collaboration [12], or the Agency for Healthcare Research and Quality [13].
At the same time, the impact of language bias may have been decreasing in recent years, as English increasingly becomes a universal language in scientific research and cooperation. Galandi et al. [14], for instance, detected a large decline in German language publications of randomized controlled trials (RCTs) over the last 10 years, indicating a shift to English as the main publication language in German-speaking countries.
A systematic review identified five methods of studies assessing the impact of restricting systematic reviews to English-only publications on meta-analyses [15,16]. None of these five studies [8,[17], [18], [19], [20]] identified any evidence that language restrictions in systematic reviews of conventional medical topics markedly changed the results of meta-analyses. However, Pham et al. discovered that such language restrictions had a large impact on systematic reviews of complementary and alternative medicine resulting in a 63% smaller effect estimate than language-inclusive reviews [20]. Additionally, stratified analyses revealed that the impact of non-English publications varies according to medical specialties and plays a larger role in fields such as psychiatry, rheumatology, and orthopedics than it does in other areas of medicine [8].
Whether there is a difference in the methodological quality of studies published in English or other languages is unclear. While two methods studies found no significant difference between completeness of reporting (randomization, double-blinding, withdrawals, allocation concealment, drop-outs) of RCTs published in English, French, German, Italian, and Spanish [18,21], Jüni and colleagues showed that trials published in English had a lower risk of bias because of better allocation concealment and blinding [19].
To date, the impact of an English-only approach has been assessed on single outcomes of systematic reviews but not on overall conclusions across multiple bodies of evidence. The Cochrane handbook acknowledges that while the potential impact of non-English publications on effect sizes of meta-analysis might be small, it is difficult to tell how the exclusion of non-English publications affects a given systematic review and its conclusions [22]. Therefore, the aim of our study was to assess whether limiting the inclusion criteria to English-language publications affected the overall conclusions in a set of systematic reviews on diverse interventional medical topics.
Section snippets
Materials and methods
This study is a retrospective analysis of a dataset from a methods study on the impact of abbreviated literature search approaches [23,24]. We developed an analysis plan a priori but did not publish it (see Appendix).
Results
The 59 Cochrane reviews included a total of 2,026 publications referring to 1,281 primary studies [[29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47], [48], [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60], [61], [62], [63], [64], [65], [66], [67], [68], [69], [70], [71], [72], [73], [74], [75], [76], [77], [78], [79], [80], [81], [82], [83], [84], [85], [86], [87]]. Fifty-one percent of the reviews assessed
Discussion
To the best of our knowledge, this is the first study to assess the impact of excluding non-English publications on overall conclusions of systematic reviews across a variety of clinical interventions. Our results indicate that the exclusion of non-English publications from evidence syntheses leads to the same conclusions as a language-inclusive approach.
Our results are consistent with other studies that have focused primarily on the effect of language restrictions on meta-analyses [8,[15], [16]
Conclusion
The inclusion of only English-language publications seems to be a reliable methodological shortcut when conducting a rapid review of clinical topics addressing conventional medical interventions. Further research is needed to distinguish when such an approach is a viable option and when it is preferable to conduct comprehensive, systematic language-inclusive reviews. Decisions that demand the highest possible certainty or reviews on topics for which many relevant studies can be expected to be
CRediT authorship contribution statement
B. Nussbaumer-Streit: Conceptualization, Data curation, Formal analysis, Funding acquisition, Writing - original draft. I. Klerings: Conceptualization, Data curation, Formal analysis, Writing - review & editing. A.I. Dobrescu: Data curation, Formal analysis, Writing - review & editing. E. Persad: Data curation, Formal analysis, Writing - review & editing. A. Stevens: Conceptualization, Data curation, Funding acquisition, Writing - review & editing. C. Garritty: Conceptualization, Data curation,
Acknowledgments
We would like to thank Cochrane for funding this research project, and the Cochrane Editorial Unit for providing data files of the Cochrane reviews. Thank you also to the contacted authors for assessing the impact of the new evidence base on their original conclusions. Furthermore, we would like to thank Sandra Hummel for administrative support throughout this project and Dawn Gartlehner for proofreading the manuscript.
References (95)
- et al.
A retrospective comparison of systematic reviews with same-topic rapid reviews
J Clin Epidemiol
(2018) - et al.
Language bias in randomised controlled trials published in English and German
Lancet
(1997) Including papers in languages other than English in systematic reviews: important, feasible, yet often omitted
J Clin Epidemiol
(2019)- et al.
What contributions do languages other than English make on the results of meta-analyses?
J Clin Epidemiol
(2000) - et al.
Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary
J Clin Epidemiol
(2005) - et al.
Completeness of reporting of trials published in languages other than English: implications for conduct and reporting of systematic reviews
Lancet
(1996) - et al.
Abbreviated literature searches were viable alternatives to comprehensive searches: a meta-epidemiological study
J Clin Epidemiol
(2018) - et al.
Identification and description of controlled clinical trials published in Spanish Gynaecology and Obstetrics journals and risk of bias assessment of trials on assisted reproductive techniques
Eur J Obstet Gynecol Reprod Biol
(2016) - et al.
Significant discrepancies were found in pooled estimates of searching with Chinese indexes versus searching with English indexes
J Clin Epidemiol
(2016) - et al.
Expediting systematic reviews: methods and implications of rapid reviews
Implement Sci
(2010)
EPC methods: an exploration of methods and context for the production of rapid reviews. Scientific resource center
Rapid versus full systematic reviews: an inventory of current methods and practice in health technology assessment. ASERNIP-S Report No. 60
A scoping review of rapid review methods
BMC Med
Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study
PLoS Med
Systematic reviews with language restrictions and no author contact have lower overall credibility: a methodology study
Clin Epidemiol
Does language matter? A case study of epidemiological and public health journals, databases and professional education in French, German and Italian
Emerg Themes Epidemiol
Methodological expectations of Campbell Collaboration intervention reviews: Conduct standards
Avoiding bias in selecting studies. Methods guide for comparative effectiveness reviews
The demise of the randomised controlled trial: bibliometric study of the German-language health care literature, 1948 to 2004
BMC Med Res Methodol
English-language restriction when conducting systematic review-based meta-analyses: systematic review of published studies
The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies
Int J Technol Assess Health Care
The inclusion of reports of randomised trials published in languages other than English in systematic reviews
Health Technol Assess
Direction and impact of language bias in meta-analyses of controlled trials: empirical study
Int J Epidemiol
Assessing the validity of abbreviated literature searches for rapid reviews: protocol of a non-inferiority and meta-epidemiologic study
Syst Rev
MEDLINE®/PubMed® data element (field) descriptions
What field codes can I use in Embase?
Excel 2016 [Software]
Trading certainty for speed–how much uncertainty are decisionmakers and guideline developers willing to accept when using rapid reviews: an international survey
BMC Med Res Methodol
Lipid-lowering efficacy of rosuvastatin
Cochrane Database Syst Rev
Oral anticoagulation in patients with cancer who have no therapeutic or prophylactic indication for anticoagulation
Cochrane Database Syst Rev
Parenteral anticoagulation in ambulatory patients with cancer
Cochrane Database Syst Rev
Anticoagulation for the long-term treatment of venous thromboembolism in patients with cancer
Cochrane Database Syst Rev
Post-pyloric versus gastric tube feeding for preventing pneumonia and improving nutritional outcomes in critically ill adults
Cochrane Database Syst Rev
Exercise-based cardiac rehabilitation for coronary heart disease
Cochrane Database Syst Rev
Early versus late tracheostomy for critically ill patients
Cochrane Database Syst Rev
Parent-infant psychotherapy for improving parental and infant mental health
Cochrane Database Syst Rev
Rivastigmine for Alzheimer's disease
Cochrane Database Syst Rev
Perioperative beta-blockers for preventing surgery-related mortality and morbidity
Cochrane Database Syst Rev
Hormone therapy for preventing cardiovascular disease in post-menopausal women
Cochrane Database Syst Rev
Music for stress and anxiety reduction in coronary heart disease patients
Cochrane Database Syst Rev
Percutaneous vertebroplasty for osteoporotic vertebral compression fracture
Cochrane Database Syst Rev
Sulfasalazine for ankylosing spondylitis
Cochrane Database Syst Rev
Amiodarone versus other pharmacological interventions for prevention of sudden cardiac death
Cochrane Database Syst Rev
Calcium supplementation for prevention of primary hypertension
Cochrane Database Syst Rev
Pulmonary rehabilitation for interstitial lung disease
Cochrane Database Syst Rev
Cited by (0)
Conflict of interest statement: None of the authors report any conflicts of interest with respect to the topic of this manuscript.
Funding statement: This work was supported by funds from the Cochrane Collaboration.