Article Text

Download PDFPDF

The peer review process: a primer for JNIS readers
  1. Joshua A Hirsch1,
  2. Laxmaiah Manchikanti2,3,
  3. Felipe C Albuquerque4,
  4. Thabele M Leslie-Mazwi1,
  5. Michael H Lev5,
  6. Italo Linfante6,
  7. J Mocco7,
  8. Ansaar T Rai8,
  9. Pamela W Schaefer5,
  10. Robert W Tarr9
  1. 1NeuroEndovascular Program, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts, USA
  2. 2Pain Management Center of Paducah, Paducah, Kentucky, USA
  3. 3Department of Anesthesiology and Perioperative Medicine, University of Louisville, Louisville, Kentucky, USA
  4. 4Barrow Neurological Institute, Phoenix, Arizona, USA
  5. 5Department of Diagnostic Neuroradiology, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts, USA
  6. 6Department of Interventional Neuroradiology, Miami Cardiac and Vascular Institute, Miami, Florida, USA
  7. 7Mount Sinai Health System, Icahn School of Medicine, New York, New York, USA
  8. 8Department of Interventional Neuroradiology, West Virginia University, Morgantown, West Virginia, USA
  9. 9Department of Radiology, University Hospitals Case Medical Center, Cleveland, Ohio, USA
  1. Correspondence to Dr Joshua A Hirsch, NeuroEndovascular Program, Massachusetts General Hospital, Harvard Medical School, Boston, MA 02114, USA; Hirsch{at}snisonline.org

Abstract

Peer review of scientific articles submitted for publication has been such an integral component of innovation in science and medicine that participants (be they readers, reviewers, or editors) seldom consider its complexity. Not surprisingly, much has been written about scientific peer review. The aim of this report is to share some of the elements of that discourse with readers of the Journal of NeuroInterventional Surgery (JNIS).

  • Standards

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Introduction

For more than 200 years, clinicians and researchers have published their work in scientific and medical journals.1–3 With changing production paradigms, an ever-greater number of investigators worldwide have contributed scientific articles. In 2010, Bastian and colleagues4 noted that 75 trials and 11 systematic reviews were published every day. In 2012, there were 28 000 scholarly, peer-reviewed journals collectively publishing nearly 2 million articles a year. The PubMed website indicates that almost 21 million articles have been published in biomedical journals.5

In 2009, working with the British Medical Journal (BMJ) group, the Society of NeuroInterventional (NI) Surgery began publication of the Journal of NeuroInterventional Surgery (JNIS). The main purpose of JNIS was to be an academic, peer-reviewed journal focusing on publishing scientific articles to advance the field of NI surgery. As a result, NI specialists look to JNIS for discussions of trials, new techniques, and breakthroughs, among other topics of interest. Therefore, we provide this assessment of the peer review process to advance the JNIS mission.

History

In 1655 Denis de Sallo published the first collection of scientific essays in the Journal des Scavans.6 In this journal, De Sallo reported the findings of others in an effort to communicate knowledge. Peer review did not exist at that time.7 ,8 It was only in 1731 that the Royal Society of Edinburgh published ‘Medical Essays and Observations’ using what is now considered to be the first peer review process.7 ,8 Manuscripts were mostly distributed, by the editor, directly to individuals with expertise in the articles’ subject matter with the purpose of selecting articles for their innovation and interest.7 Like many advances in science and medicine, peer review was not immediately appreciated as a breakthrough.8 An exemplar of that point is Albert Einstein, who published over 300 journal articles between 1901 and 1955, including his miracle year of 1905 when he introduced new ways of understanding space, time, energy, momentum, light, and the structure of matter. In the course of his entire career he probably had only a single paper peer reviewed, in 1936, for a submission he jointly wrote about gravitational waves. The paper received negative reviews, and Einstein rejected in writing both the idea of an anonymous expert reviewer selected by the editor and that expert's comments, eventually submitting the paper elsewhere.9 His response suggests someone unfamiliar with the fundamental tenets of peer review. One can speculate that had Einstein lived today his view of peer review might have been somewhat different.

The Royal Society of Edinburgh was careful in making clear that the stamp of peer review did not necessarily mean the work was better than non-peer-reviewed publications. They thus provided a disclaimer stating that peer review did not guarantee truthfulness or accuracy. They emphasized further that the purpose of the journal was solely to disseminate creative and important ideas.7 ,8 As today, the submitting authors were responsible for the quality and veracity of their own research.7 ,8

As the peer review process has been embraced over time, its development has occurred in fits and starts.6–8 ,10 Different journals apply varying approaches to peer review. Until 1976, the Lancet did not implement peer review. At that time, Journal of the American Medical Association (JAMA) sent their submissions to an internal review panel and, on rare occasions, to outside experts.11 In contrast, the BMJ—with whom JNIS shares a common publisher—has sent every non-editorial submission to a recognized expert since 1893.10 By the late 20th century, peer review has become standard and is used by almost all biomedical journals.8 ,10 ,11

Peer review

The modern peer review process has a variety of constituents.8 ,10 ,11 Authors expect to receive a fair and unbiased critique. Journals need to ensure the integrity and accuracy of the material that they publish. The lay public relies on the integrity of the publication process to inform themselves and their providers of innovative and optimal treatments. The common thread for maintaining the quality and integrity of research is the peer review process. In essence, it provides a formal opportunity for authors to gauge reaction to their work, as well as allowing for revisions and subsequent correction of errors before publication. At its best, peer review allows the editors and expert reviewers to apply with diligence and care their unbiased knowledge, expertise, and interpretation.8 ,10–12

The peer review process, nevertheless, is not without its flaws.8 Cantekin et al13 in an analysis of the peer review, exposed a variety of problems leading to imperfection and bias. Some of these include reviewer conflict of interest, breach of confidentiality, and non-disclosures of funding sources.8 ,14–16 An additional problem that can arise in peer review is reluctance on the part of reviewers or editors to accept papers that express views contrary to currently held norms.15 ,17 ,18

The foundation of the selection process for articles submitted to JNIS, and most other academic journals, is the work of volunteer reviewers. After agreeing to review an article, reviewers are given a deadline to perform their analysis. Distinct comments are solicited for both the authors and the editors. When the associate editor is satisfied with the number and quality of the revisions, he or she will make a recommendation to the editor in chief (EIC). The EIC plays an important role in establishing uniformity between the associate editors. The EIC makes one of the following recommendations: accept, accept pending minor revisions, accept pending major revisions, revise and resubmit, and/or reject (box 1). If revisions are requested, they are returned with an annotated pdf and a cover letter dealing with the reviewer queries, typically, on a point-by-point basis. At JNIS, reviewers are blinded to the authors’ identities, but the editors are not. Other review formats exist, ranging from blinded editors and reviewers, to completely open review.8

Box 1

Various steps in publication of a manuscript in the Journal of NeuroInterventional Surgery (JNIS): submission, peer review, and publication

  1. Pre-submission phase (investigators)

    • Research including data review and synthesis

    • Initial manuscript preparation

    • Internal review by authors

    • Final manuscript signed off by all authors

    • Submission to JNIS, including patient consent forms

  2. Post-submission phase

    • Immediate post-submission phase (journal)

      • JNIS receives the manuscript

      • Editorial manager logs and acknowledges receipt

      • Quality check by editorial manager and editor in chief: articles may be rejected at this point

    • Editors

      • Editor in chief assigns manuscript to an associate editor

      • Editors review manuscript quality and confirm relevance to the journal

      • Editors decide on relevant experts to consider peer review

    • Peer review

      • Peer reviewers examine the manuscript for study design/methodology, validity, accuracy, originality, significance, quality of English, etc

      • Peer reviewers will often ask for additional comments, to have queries answered, for additional references cited, and/or more rigorous statistical or other additional expert review

    • Associate editors

      • Consider reviewer comments

      • Make recommendation to editor in chief to reject/revise/accept

  3. Post associate editor recommendation

    • Editor in chief sends a letter with his/her decision about the paper

      • Authors respond to comments and suggestions

      • Authors revise manuscript

      • Authors resubmit with a cover letter dealing with the reviewer comments/questions and a marked up manuscript

    • Editors

      • Final decision to accept/reject manuscript

  4. Post-acceptance at JNIS

    • Proofs with requested 48 h turnaround

    • Publication online first

    • Publication in print

Adapted and modified from Manchikanti et al.8

The majority of the scientific community considers peer review to be valuable.7 ,8 ,10 ,11 ,19 In a 2009 survey of 4000 people involved in research, although 84% believed peer review to be an important, arguably necessary, process for the dissemination of accurate scientific information, only 69% were satisfied with current peer review systems.19 Despite the questions this survey raised about perceptions of peer review, however, it is noteworthy that fully 91% of respondents believed their last paper had been improved by the peer review process.

Peer review bias

Bias may be defined as a systemic prejudice that prevents the accurate and objective interpretation of scientific studies.8 Indeed, it has been suggested that the objectivity of peer review has been affected in many different ways. These include, but are not limited to, differences in belief or ideology, aversion to new ideas,20 and conflicts of interest. Other potential concerns also include cursory/superficial reviews by reviewers with decreasing motivation and enthusiasm for reviewing over time,21 the occasional need for associate and chief editors to ‘over-ride’ the unanimous recommendation of reviewers to reject papers that may be ‘hidden gems’, and the poor ability to train reviewers to improve the quality of their reviews and eliminate bias.22 ,23

The inability to allow dissenting views to be published in journals has become a major concern.8 ,18 Members of the NI community have had this angst several times in recent years. Specifically, in 2009, when the vertebroplasty versus controlled intervention trials were published, and again in 2013, when the three inconclusive major intra-arterial stroke therapy trials were published in the New England Journal of Medicine, content experts were not given the opportunity to publish contemporaneous counterpoints. Rather, we were left to write short letters to the editor that were frequently rejected—while at the same time these studies became sensationalized by the lay media, to the detriment of our patients.17

Bunner and Larson carried out pioneering work looking at the quality of the peer review process from the perspective of authors and editorial board members.14 Approaching 80%, a large proportion of those who responded to the survey, indicated that they knew what to expect from the peer review process, 19% indicated that they did not have a specific sense, and approximately 3% indicated that they did not know what to expect.14 For manuscripts that were accepted, 84% of authors viewed their reviews favorably, whereas for manuscripts that were rejected, 50% viewed them unfavorably. Perhaps more surprising, almost 30% of authors felt that reviewers did not have an accurate understanding of the paper's topic; an additional 34% felt the reviewer comments were insufficient, and about 30% felt that the reviewers failed to provide suitable suggestions for revision. Most importantly, Bunner and Larson14 indicated that almost 50% felt the critiques were unfair and biased.

Types of peer review bias

Put simply, reviewer bias is a violation of the ideal impartiality that a reviewer or editor has in evaluating a submission. Impartiality in this context is the ability for any reviewer to interpret and apply evaluative criteria consistently for any given manuscript.6–8 ,10–14 ,16 Peer review, however, can be difficult to study systematically, as the process is typically subjective and secretive.22

Gilbert et al24 in 1994 and Jayasinghe et al25 in 2003 showed that specific classes of reviewers are systematically tougher or softer24 ,25 on identical submissions.26 For example, reviewers from the USA tend to be more lenient than their colleagues from the UK, Germany, or Australia.8 Moreover, until quite recently, there has been a well-studied, strong positive bias of reviewers to favor positive, rather than negative study results.27 ,28

Editors must pay specific attention to these issues for impartial assessment and publication.

Content-based bias

As the name suggests, this links directly to content of part or all of the manuscript. Content-based bias may reflect reviewers and editors preferring to see submissions that cite their own work. Cognitive cronyism exists when favoritism is given to submissions that share similar schools of thought.8

Confirmation bias

Confirmation bias is the tendency to gather, interpret, and remember evidence in ways that affirm previously held beliefs. In the peer review system, confirmation bias is understood as reviewer or editor bias against manuscripts where the thesis and/or results disagree with the perspective of the reviewer or editor.8 This type of bias goes towards the heart of the peer review process, wherein evaluations are performed independently of “desires, value perspectives, cultural and institutional norms and presuppositions, expedient alliances and other interest.” In a telling article, Ernst et al,29 found that reviewers who had previously published materials in favor of what they termed a controversial clinical intervention looked at a manuscript where the data supported the use of that technique with a friendlier eye than those who had produced manuscripts that did not support the use of that technique.

Conservatism bias

This is a bias against innovative or landmark research. In its most dangerous form, conservatism bias threatens scientific progress in a very basic and understandable way.20 ,30 Specifically, this type of bias can limit funding and public sharing of theories that deviate from mainstream hegemony and result in authors facing a higher burden of proof requirement to get their publications accepted.

Publication bias

Alleged publication bias is the tendency for journals to publish research demonstrating positive, rather than negative, outcomes.8 ,18 ,27 ,28 If these assumptions are accurate, publication bias raises potential downstream problems because it will—by its nature—lead to errors in effect size measurements in later meta-analyses.30 Publication bias is hypothesized to occur because clinicians and researchers can disagree about the importance of research that produces negative outcomes.30 However, the authors of this manuscript believe that, more recently, a ‘reverse’ form of publication bias may be at play in the neurointerventional literature, in that some notable recent NI published trials have indeed highlighted negative results—gaining great notoriety and providing considerable sensationalism.8 ,18

Bias of conflicts of interest

Perhaps the most self-evident forms of bias, a ‘conflict of interest’ occurs when participants in the publication process have a personal interest that might inappropriately influence their judgment, regardless of whether or not their judgment is actually affected.3 ,8 ,15 ,18 Editors and reviewers may be more open-minded to research from friends, collaborators, or clinicians, particularly if the report supports the professional world view of the reviewer or editor. Conversely, editors and reviewers may be more likely to reject a submission from a clinician whom they dislike, who supports a competing idea, or who opined negatively on a manuscript written by the current reviewer in the past. Conflicts of interest can be difficult to detect.

Forms of peer review

Double-blind peer review

Double-blind peer review is preferred primarily because of its perceived objectivity and fairness.8 ,31 Challenges include the difficulty of making a review truly double blind. In a survey of 1500 editors in chemistry, most respondents felt that double blinding was pointless and, by implication, an unnecessary step, because the particular content and the references chosen indirectly make the author's identity clear. Moreover, a number of empirical studies showed that reviewers can successfully identify authors 25–40% of the time. In a subspecialized field like neurointerventional surgery, we believe this observation has validity.

Open peer review

With this approach, the authors and reviewers are known to each other. This has the obvious benefit of increasing transparency and the possibility of speeding up the review process. Since 1999, the BMJ has asked potential reviewers to identify themselves and declare any competing interests. The reviewers are not disclosed to BMJ readers at the time of publication. One might consider this approach as a single-blind review. JNIS worked with our BMJ publishers to develop the system we use today—namely, double blind, without identifying reviewers and authors to each other.

Fraud and peer review

When one considers the purpose and methodology of peer review, one can argue that it is not necessarily designed to detect fraud. Bearing that in mind, and recognizing that the critique may not be fair, the peer review approach, has been criticized by some participants for its inability to detect fraud.6 ,8 To that point, Godlee et al32 intentionally introduced eight specific, arguably detectable, weaknesses into a research article. They then sent the flawed paper to 200 reviewers. Subsequent analysis of the reviewer comments showed that, on average only two of the inserted weaknesses were identified.

Peer review cannot ensure that the paper does not have any mistakes. Rather, alternative mechanisms of surveillance and veracity confirmation may occasionally need to play a role in ensuring the integrity of academic literature. Indeed, a researcher was recently imprisoned for falsifying data in clinical trials.8

Conclusion

Peer review is a critical element of our routine scientific discourse. JNIS, like many journals, could not function in its current format without it. It has served the medical/scientific community well for over 100 years.6–8 ,13 ,16 For a process that is so fundamental to our specialty and academic activities, however, it is sobering to think how much is assumed and how little we actually know about the biases and limitations inherent to peer review. The editorial staff at JNIS believe in peer review and favor the approach that we have crafted with the BMJ—most notably, that the authors and reviewers are blinded to one another, but not to the editorial staff.

References

Footnotes

  • Contributors JAH and LM composed the initial draft. All authors reviewed and made editorial suggestions that resulted in the final draft.

  • Competing interests None declared.

  • Provenance and peer review Commissioned; internally peer reviewed.