Article Text
PDF
Abstract
Introduction No studies have sought to provide a quantitative or qualitative critique of research in the field of neurointerventional surgery.
Objective To analyze recent publications from the Journal of Neurointerventional Surgery (JNIS) to test a new method for assessing research and collaboration.
Methods We reviewed all JNIS Online First publications from 25 February 2015 to 24 February 2016. All publications—human or non-human research, systematic reviews, meta-analyses, or literature reviews—were included; editorials and commentaries were excluded. For each publication, study design, number of patients, authors, contributing centers, and study subject were recorded. Level of evidence was defined using a new scale.
Results A total of 206 articles met inclusion criteria. Only 4% were prospective studies. Twenty-eight per cent of scientific research featured patient series of nine or less. The majority of publications were categorized as low-level evidence (91%). Forty-seven per cent involved individuals from a single center, with 87% having collaboration from three or fewer centers. International collaboration was present in 19%. While 256 institutions from 31 countries were represented, 66% were represented in only one publication.
Conclusions We queried JNIS Online First articles from a 1-year period in a pilot study to test a new method of analyzing research quality and collaboration. The methodology appears to adequately quantify the studies into evidence tiers that emulate previously published, widely accepted scales. This may be useful for future comparison of peer-reviewed journals or for studying the quality of research being performed in different disease processes or medical specialties.
- Standards
- Political
Statistics from Altmetric.com
Introduction
As the USA healthcare system continues its paradigm shift towards quality-driven reimbursement, there has been a synchronous pivot towards defining, summarizing, and interpreting medical research according to the level of evidence.1 ,2 These distinctions help the medical community to understand the importance, validity, and generalizability of trial results. Unfortunately, producing research of quality that provides satisfactory evidence has become very costly and complex. Multicenter randomized trials are cumbersome, require years to reach enrollment goals, and usually answer only a limited number of questions in a given clinical scenario and thus lack generalizability.3 ,4
Modern level of evidence criteria are derived from an original scale described by the Canadian Task Force on the Periodic Health Examination in 1979 (table 1).5 Since that time, levels of evidence have been modified to include a wider breadth of study types in a number of clinical scenarios. In 2009, the Oxford Centre for Evidence-based Medicine published levels of evidence designed to include studies on diagnostic tests and prognostic markers as well as therapies (table 2).6
Original Canadian Task Force on the Periodic Health Examination level of evidence classification5
Oxford Centre for Evidence-based Medicine 2009 levels of evidence6
Anecdotally, a large portion of clinical research produced in the field of neurointerventional surgery (NI) involves single or multiple institution retrospective case series. Such data are readily available at each center at little cost and are easy to query, analyze, and describe. However, these data are unfortunately subject to critical biases that threaten the quality of the conclusions that can be drawn. To date, no studies have sought to provide a quantitative or qualitative critique of the quality of research produced in a given journal or specialty while simultaneously describing the degree of collaboration among centers. Our study was designed to query the most recent publications in a single journal to test a new method of studying published research quality and collaboration. To do so, we designed a pilot study to analyze the recent publications from the Journal of Neurointerventional Surgery (JNIS), the journal of record for NI research and the official journal of the Society of Neurointerventional Surgery.
Methods
We reviewed original research studies published in JNIS as Online First publications from 25 February 2015 to 24 February 2016. All original research articles—case reports, case series, animal research, systematic or narrative reviews, and meta-analyses published during this time—were included. All non-scientific content, such as editorials, commentaries and healthcare policy, and/or socioeconomic reviews, was excluded. For each article, the number of patients presented (if applicable), the study design, subject matter, number of authors, author location, and number of contributing centers were recorded. Study design was defined as one of the following: randomized controlled trial, prospective series (≥10 patients), retrospective series (≥10 patients), case report, case series (2–9 patients), animal study, non-human or imaging study, systematic review or meta-analysis, nationwide or statewide database analysis, or narrative review. Studies with both a prospective and a retrospective component were considered retrospective for this analysis. In determining the number of centers, all departments or groups within a given institution were categorized as the same. Therefore, all affiliations from different departments in the same university were counted as a single center.
Each publication was graded based on level of evidence using a modified scale for NI research created strictly for this analysis (table 3). This scale was designed to emulate the 2009 Oxford Centre for Evidence-based Medicine scale but was modified to incorporate the vast majority of NI scientific content in a simple and reproducible manner.
Modified level of evidence scale for neurointerventional research
Results
From 25 February 2015 to 24 February 2016, 243 JNIS Online First articles were published. Of these, 31 articles were commentaries, letters to the editor, or editorials. An additional six articles were standards and practices or guideline publications. Therefore, a total of 206 research articles were published as JNIS Online First articles and met inclusion criteria for this analysis. The types of studies published are shown in table 4. Only a small minority represented prospective studies of patients (3.9%).
Types of studies
Institutional representation
The average number of authors for each publication was 6.8 (SD 2.9, range 1–17, 25th centile 5, 75th centile 9). The mean number of institutions represented for each study was 2.1 (SD 1.6, range 1–10, 25th centile 1, 75th centile 3). Of the 206 studies, 97 (47.1%) were single-center studies (figure 1). The vast majority of studies had collaboration of three or fewer institutions (86.7%).
Number of centers represented in studies. The vast majority of manuscripts included three or fewer contributing institutions.
A total of 256 unique institutions (centers) in 31 nations were represented among the 206 articles (mean 1.7 publications for each individual institution, SD 1.3, range 1–8). The vast majority of institutions were involved in three or fewer articles during the index year (91.4%), with almost two-thirds of institutions being represented only in a single publication (66.4%). The most frequently represented institutions were the University of California, Los Angeles (eight publications), University of Massachusetts, Worcester (eight), University of Geneva (seven), Baylor College of Medicine (six), Emory University (six), and Mayo Clinic-Rochester (six).
International collaboration was present in 39/206 studies (18.9%). The eight most frequently represented countries are shown in table 5. Authors from the USA collaborated with authors from at least one of 18 other countries in 20% of manuscripts.
International representation and collaboration
Level of evidence
The quality of the research studies based upon our level of evidence grading scale is shown in table 6. The majority of publications were categorized as low-quality (level 4 or 5) evidence (90.8%) using the proprietary algorithm described above.
Level of evidence of the research studies
Subject
The percentage of publications based on study subject is displayed in table 7. Over two-thirds of publications reported aneurysms or acute ischemic stroke (68%). The levels of evidence represented by published research for each subject are shown on the right in table 7.
Contributing centers and level of evidence based on study subject
Discussion
We examined original scientific articles published Online First in JNIS over a 1-year period in a pilot study to test a new method of analyzing published NI research quality and collaboration. The methodology is simple to perform and appears to adequately quantify the represented studies into evidence tiers that emulate previously published, widely accepted scales. To the best of our knowledge, this is the first time a peer-reviewed journal's individual published research and collaboration between institutions have both been analyzed in this manner. This might be a useful tool for comparing and contrasting peer-reviewed journals in the future or for studying the types and quality of research being performed in different disease processes or medical specialties.
Although the data presented in this pilot study are representative of only a single journal over 1 year, the results are interesting. First, the vast majority of NI published research in JNIS is of lower-evidence human research, with only 4% of studies being large prospective studies and only one study being a randomized controlled trial. Over one-quarter of scientific research published (28%) comprised patient series of nine or less. Second, nearly half of all publications involved individuals from a single center, with the vast majority (87%) having collaboration of individuals from three or fewer centers. Additionally, of the studies categorized as level 4 or 5 evidence, collaboration among centers was low (mean 2.2 and 1.7 centers, respectively). International collaboration was present in only 19% of manuscripts. Finally, while 256 distinct institutions from 31 countries were represented in the 206 JNIS articles, almost two-thirds of centers were represented in a single publication. Overall, these data suggest that most published research was of low evidence with poor collaboration between institutions.
As this pilot study evaluated only original research publications over 1 year from a single journal, it is invalid to generalize the data obtained from this study to the field of NI as a whole. However, as JNIS is the journal of record for NI research with a focus on neurointerventional topics and content with a 2015 Thompson-Reuters impact factor of 2.77 (lying in the middle of the impact factor spectrum among journals that publish NI articles), the results are important. The next step is a wider, more inclusive review of NI published literature from the major journals. This will allow for comparisons between journals, disease types (stroke vs aneurysm, for instance) and provide a better summary of the research produced in the field as a whole. Hopefully, this analysis would generate a rigorous, in-depth overview of the quality of research being performed within the NI specialty.
Studies have previously evaluated the level of evidence in published research in other journals in varied specialties.7–12 These studies collectively suggest a minority of published medical research is of a higher level of evidence. However, these studies are unfortunately one-dimensional and do not evaluate individual research topics within their respective fields nor do they evaluate inter-institutional or international collaboration. Thus, they provide a description of the level of evidence in their target journals but do not offer commentary on the research networks or distribution of resources within the specialty, nor are they designed to track research trends and quality over time. Our pilot study methodology has been designed to provide a means of obtaining robust and in-depth insight into the current state of research within our specialty, which may be repeated annually. Furthermore, later study of citation frequency will identify variables predictive of higher citation rates, which may help researchers understand how to optimize their research output.
Why is it important to understand the types of research being published and the collaboration among centers in our (or any) specialty? Knowledge of the types of studies being published allows governing bodies, physician societies, and funding institutions to understand how best to direct research money and effort. For instance, should only a small proportion of the published research involve multicenter prospective series, overseeing bodies or societies may encourage collaborative prospective studies through grants or funded participation. Further, identification of research trends may allow for the study of societal, institution, and physician-specific limitations to performing cooperative, high-quality research, with the goal of developing strategies to mitigate these obstacles. Additionally, disease processes with poor research representation, or those with low research quality, may be identified and resources shifted to these entities. This might help physician scientists and our researchers to identify an area in need of study. The importance of collaboration in medical research, particularly international collaboration, is beginning to be realized.13 International collaboration may serve as an important metric, particularly given its apparent relationship to citation frequency. Peer-reviewed journals may also use this information to understand their distribution of publications and this could assist with editorial review of scientific articles for publication. For instance, journal editors and reviewers could be encouraged to preferentially accept articles with higher levels of evidence or those that include patient series and authors from multiple centers. Like the impact factor, a level of evidence and collaboration metric might be useful in summarizing a journal's global footprint in its respective field.
It is important to note that level of evidence does not necessarily imply quality. Research with a low level of evidence may still be of significant relevance. Certain criteria for judging research that are important in diseases with significant burden might not be as applicable to a niche field such as NI surgery with a smaller disease footprint. It may be impossible to provide evidence for every device iteration or technique with high levels of evidence. Certainly new devices, especially implants, need to be rigorously tested in regulated multicenter trials before being approved, but a small single-center, well-executed study may also affect paradigms. Occasionally, small case series reporting a new treatment or complication can be as worthwhile as a large clinical trial. Such publications may not represent the top level of evidence but remain important in advancing the field, one small step at a time. Different types of research are needed to answer different clinical questions.14
This study has some limitations. As a pilot study, it was designed to study a single journal's original research publications over 1 year, which represent only a fraction of the research publications within the NI specialty during that period. The level of evidence scale was modified from the Oxford Centre for Evidence-based Medicine and has not been formally validated.
Conclusions
We examined original scientific articles published Online First in JNIS over a 1-year period as a pilot study to test a new method of analyzing published NI research quality and collaboration. This might be a useful tool for comparing and contrasting peer-reviewed journals in the future or for studying the types and quality of research being performed in different disease processes or medical specialties. Overall, this methodology aims to provide a quantified analysis of the current state of research in our specialty. The next step is an analysis including research publications from all the major journals publishing NI content.
References
Footnotes
Contributors Each author listed above should receive authorship credit based on the material contribution to this article, their revision of this article, and their final approval of this article for submission to this journal. Alyssa Pierce assisted with the editing and revision of this manuscript.
Competing interests AMS: Penumbra Inc—consulting, shareholder; Stryker—consulting; Pulsar Vascular—consulting; Blockade Medical—shareholder. JAH: Medtronic—consulting; Codman Neurovascular—Data and Safety Monitoring Board. JM: National/International PI/Co-PI for the following trials: THERAPY (PI), FEAT (PI), INVEST (Co-PI), COMPASS (Co-PI), LARGE (Co-PI), COAST (Co-PI), POSITIVE (Co-PI). On the steering committee for the MAPS trial; Rebound Therapeutics, TSP Inc, Cerebrotech, Lazarus Effect, Pulsar, Medina— consultant; Blockade Medical, TSP Inc, Lazarus Effect, Medina—investor.
Provenance and peer review Not commissioned; internally peer reviewed.