Article Text

PDF

Bibliometric indices: defining academic productivity and citation rates of researchers, departments and journals
  1. Rebecca M Garner1,
  2. Joshua A Hirsch2,
  3. Felipe C Albuquerque3,
  4. Kyle M Fargen1
  1. 1Department of Neurological Surgery, Wake Forest University, Winston-Salem, North Carolina, USA
  2. 2Department of Interventional Neuroradiology, Massachusetts General Hospital, Boston, Massachusetts, USA
  3. 3Department of Neurosurgery, Barrow Neurological Institute, Phoenix, Arizona, USA
  1. Correspondence to Dr Kyle M Fargen, Department of Neurological Surgery, Wake Forest University, Winston-Salem, NC 7157, USA; kfargen{at}wakehealth.edu

Abstract

There has been an increasing focus on academic productivity for the purposes of promotion and funding within departments and institutions but also for comparison of individuals, institutions, specialties, and journals. A number of quantitative indices are used to investigate and compare academic productivity. These include various calculations attempting to analyze the number and citations of publications in order to capture both the quality and quantity of publications, such as the h index, the e index, impact factor, and Eigenfactor score. The indices have varying advantages and limitations and thus a basic knowledge is required in order to understand their potential utility within academic medicine. This article describes the various bibliometric indices and discusses recent applications of these metrics within the neurological sciences.

  • bibliometrics
  • h index
  • academic productivity
  • citation analysis
View Full Text

Statistics from Altmetric.com

Introduction

Recent focus on academic productivity has led to the creation of a number of bibliometric indices analyzing the quantity and quality of scholarly research.1–5 These indices attempt to determine a researcher’s academic contribution with a single value, or in some instances with a few values. In 2005, Hirsch, a physicist, proposed the h index, a metric designed to capture the quality and quantity of the scientific output of a researcher.2 Since then, a number of variations on the h index have been proposed, as well as metrics designed to complement or replace the h index. These indices have been used throughout the sciences and academic medicine to evaluate the academic productivity of researchers, institutions, medical specialties, journals and more. For instance, NIH funding for academic neurosurgeons has been shown to correlate with higher bibliometric indices.6 In the neurological sciences a number of indices have been used, including the h index, e index, g index, m quotient, hc index, and i10 index. A large study evaluated these indices for academic neurosurgeons and demonstrated important relationships with academic rank, gender, and subspecialty.7

In addition to analyzing individual researchers, there are measures examining the impact and influence of journals. Garfield proposed the journal impact factor (JIF) in 1955 and rankings of journals by impact factor are published annually by Thomson Reuters in the Journal Citation Reports.8 Other values have been created to quantify the impact and influence of a journal such as the Eigenfactor score (ES), the article influence score (AIS), the SCImago journal rank (SJR) indicator, and the source normalized impact per paper (SNIP).9–12

The purpose of this review is to summarize the most commonly used bibliometric indices and describe their applications, strengths, and limitations.

Individual authors

The h index

The h index is the most popular metric designed to encapsulate the quantity and quality of a researcher’s publications.2 The initial metric most investigators consider is total citation count. However, this is a blunt instrument. Citation count can be heavily influenced by a small number of highly cited papers. In addition, the number of papers published by a researcher can be influenced by producing a large number of low impact papers that are not cited frequently rather than a moderate number of impactful papers that are cited frequently. In response to this, the h index attempts to mitigate the influence of highly cited papers and to exclude infrequently cited papers. The h index combines publication number (quantity) and citation counts (quality) into a single value.

The h index is defined such that an author who has published n papers, each of which has been cited at least n times, has an h index of n. Therefore, if a researcher has an h index of 10, he has published at least 10 papers which each have 10 or more citations. Papers that do not have at least h number of citations are not included in the h index value, and once a paper is included, any excess citations it exceeds over h will not increase the h value. The h value implies that the researcher has been cited at least h2 times. All the papers included in the h index evaluation (ie, all papers with greater than h citations) are considered part of a set called the h core. Fundamentally, given the exponential relationship, moves to a higher index value become more difficult with progressively higher h indices.

The h index is powerful in that it generates a single value encompassing both quantity and quality of research, but it does have some distinct weaknesses. Because the calculation of the h index does not include a term accounting for the age of an article, the value can only increase or remain the same as time passes. Therefore, it can advantage older researchers who have had a longer period in which to publish, even if they are no longer actively publishing and contributing to the scientific field. Khan et al have demonstrated that the h index increases with academic rank for neurosurgeons.7 This disadvantages younger researchers who have had less time in which to publish and who can thus appear less productive when compared with older researchers who may no longer be productive. The h index can also be influenced by self-citation due to researchers citing their own papers. In addition, citations of papers that are in excess of the value of h are not considered, lessening the effect of highly cited papers. It has been shown that values for the h index vary across specialties and thus make it difficult to compare outside one’s own field.13

The m quotient

In order to account for researchers of varying age, Hirsch also proposed a term known as the m quotient.2 The m quotient allows for comparison of researchers at different stages of their careers. The m quotient is defined by dividing the h index by the number of years since a researcher’s first publication. By normalizing the h index across length of career, the m quotient attempts to better allow evaluation of researchers of differing ages. Although the m quotient accounts for the age of the researcher, it has similar weaknesses to the h index. In addition, if two researchers have published the same number of articles with the same citation counts, the m quotient calculation may penalize the researcher who began publishing earlier in his career (ie, college or residency) when compared with the researcher who began publishing later.

The g index

To overcome some of the shortcomings of the h index, Egghe defined a bibliometric index called the g index in 2006.1 The h index ignores papers which are cited much more highly than a researcher’s other papers. Once a paper has been included into the h core (the papers which have greater than h citations), any excess citations are ignored. The g index is intended to give more weight to highly cited papers in order to account for their seemingly disproportionate impact on the field. The g index is calculated by ranking a researcher’s papers in decreasing order of citations and defining g as the number g of papers having greater than or equal to g2 citations cumulatively. A g index of 10 implies that the top 10 papers have a cumulative amount of 100 citations. The papers included in the g index do not require individual citation counts greater than g, as long as the cumulative number of citations for the papers included is still greater than g2. Therefore, g is always greater than h because it includes citations in excess of the number used for inclusion into the h index. The g index allows for differentiation of authors with similar h scores by accounting for citations greater than the h2 citations used for the evaluation of the h index and allowing papers with high citation counts to increase the value of the index. Thus, the g index can be increased by a few highly impactful papers, despite a researcher’s other papers having low citation counts. In addition, it is somewhat more difficult to calculate the g index and it is less widely used.

The e index

To complement the h index, Zhang introduced an index called the e index.4 The e index is designed to be independent from and a complementary value to the h index. It accounts for the excess citations of papers used in the evaluation of the h index, in addition to the citations included in the h2 value. Once a paper has attained h citations, any excess citations are not considered in the calculation of the h index. The e index is defined as the square root of the net excess citations of papers included in the h core. In use with the h index, the e index accounts for the cumulative number of citations of papers in the h core. Thus, the e index allows for evaluation of the total number of citations, so that in situations where researchers have the same h index, it can be determined which has the more highly cited and impactful papers.

The contemporary h index (hc index)

The contemporary h index (hc index) is designed to account for the age of articles by giving greater weight to more recent papers. Sidiopoulis et al proposed it in order to generalize the h index to allow for evaluation of “brilliant young scientists.”3 Comparison among researchers of different ages using the h index is difficult because the h index increases over the length of a researcher’s career. The hc index is defined by creating a citation score through multiplication of the citation count of a paper by four and division by the number of years since publication of the paper. A researcher has an hc index equal to the n number of articles that have citations scores greater than hc. The hc index uses the age of a paper to alter its effective citation count to create an index that is similar in numerical value to the h index, but gives greater weight to newer papers. By accounting for the age of the papers, the hc index can distinguish active older researchers who are still producing work as well as younger scientists with greater potential.

The i10 index

Google Scholar provides an index called the i10 index, which is defined as the number of papers with at least 10 citations.5 While Google Scholar is the only database that uses this index, it can be a useful value in assessing a researcher’s academic output.

Table 1 shows the various metrics used to evaluate academic productivity for individual researchers.

Table 1

Bibliometric indices for individual researchers

Examples of bibliometric studies for individual researchers in neurosciences

A large study computed bibliometric indices for academic neurosurgeons and compared the relationship of academic productivity to academic rank, gender, and subspecialty. It found that the indices increased with academic rank and showed variation based on gender, which disappeared when separated by academic rank. The study also found a significant difference in index values for subspecialties, with functional/epilepsy, peripheral nerve, radiosurgery, neuro-oncology/skull base, and vascular specialties having higher index values. Using Scopus, the mean h index calculated was 14.6 with a range of 0–76 and the mean m quotient was 0.71 with a range of 0–3.67. Using Google Scholar, the mean g index was 26.5 with a range of 0–131 and the mean hc index was 9.3 with a range of 0–42.7 A different study conducted in 2011 calculated the mean h index for neurosurgery program faculty members using Scopus and found that the mean h index for assistant professors, associate professors, and full professors was 5.6, 9.7, and 16.0, respectively.14

Tomei et al used the h index to evaluate gender differences among neurological surgeons and found that men had greater academic productivity. However, when separated by academic rank, there was no significant difference between men and women.15 The h index and its variants have also been used to evaluate pediatric neurosurgeons in North America and found that pediatric neurosurgeons affiliated with a fellowship program had higher bibliometric indices. Male pediatric neurosurgeons were also found to have statistically higher bibliometric indices.16

Extending beyond individual researchers

Additionally, the indices can be used to evaluate groups rather than individuals, such as medical specialties or departments within institutions. This can be done by evaluating the output of multiple researchers and compiling median or cumulative index values. Institutional h, e, and g indices over the previous 5 years have been computed to assess the academic productivity of neurosurgical residency programs. The study found that large departments which were the most academically productive had relatively equal publication outputs from faculty members.17 Sonig et al evaluated the departmental h and e indices for neuroendovascular programs in the USA, showing that the h index can be calculated for a particular subspecialty. The mean h index calculated for neuroendovascular departments was 38.86 with a range of 12–103. They found no association between h index and affiliation with a comprehensive stroke center or with accreditation.18 In addition, many studies have used bibliometric indices to evaluate neurosurgical programs across the world.19–22

Altmetrics

With the increasing use of social media by the public, academicians have begun to use social media outlets to broadcast and advertise for their work. An expanded online presence widens readership to include the larger public community thereby increasing impact, which may lead to an increase in article citations. Following the trend of social media use and online blogging, the field of altmetrics (‘alternative metrics’) has emerged. Altmetrics measure the online activity associated with papers through analysis of social media platforms, blogs, media outlets, and more. Altmetrics are computed and presented by companies and websites such as PlumAnalytics, Altmetric, and Impactstory, among others.23–25 Wang et al have analyzed altmetric scores for the ‘highest trending works’ in neurosurgical journals and have shown that social media presence is correlated with higher almetric scores.26 Additionally, many journals have recognized the power of social media platforms in enhancing readership and now actively engage online followers with journal content. The Journal of Neurointerventional Surgery (JNIS) has implemented measures using social media platforms including Twitter and Facebook to advertise for scholarly articles, podcasts and other content and have demonstrated a significant increase in readership and online traffic.27 28

Journals

A journal’s influence can be evaluated by calculating the total number of times it has been cited (citation count) or by calculating the average number of citations per paper published in the journal. In addition to these values, the h index can be extended to evaluate journals by calculating the h index using all the papers published in a particular journal for a given period of time.29

Journal Impact Factor (JIF)

For decades, the JIF has been used to evaluate the influence of journals. It is published annually by Thomson Reuters in the Journal Citation Reports and has been discussed in detail in a prior issue of JNIS.30 When evaluating the citation count for journals, smaller journals may appear to be less influential because they publish fewer papers. The JIF aims to eliminate this bias by dividing the citation count of a journal by the number of papers published in the journal.31 The JIF value is therefore the frequency with which the average paper in a journal has been cited.

JIF is defined as the total number of citations to papers published in a particular journal in the previous 2 years divided by the total number of papers published in the journal in the previous 2 years. For instance, the current JIF for JNIS is 3.55, which indicates that in the last 2 years each published article generated, on average, 3.55 citations.

The JIF includes only journals indexed in the Web of Science database and is calculated for papers published in a given journal during the previous 2 years. Citation count per article can vary across disciplines, thus making it difficult to use the JIF when comparing journals in different fields. While the short time period of 2 years can make it difficult to assess the long-term impact of journals, Thomson Reuters does also publish a 5-year JIF which calculates the average citations to a journal over the previous 5 years.32

Eigenfactor Score (ES)

The JIF places equal value on citations received from all sources. However, a certain citation source can carry more weight and influence than another. To account for this, Bergstrom proposed the ES.10 The ES gives greater importance to the citations in journals cited more often by other journals, which are therefore assumed to have greater influence.

The ES is calculated using an iterative ranking algorithm that views citations among journals as a network and functions in a way similar to that used by Google to rank pages in response to search terms. It can be understood by imagining a ‘reader’ randomly walking through a database of journals, following citations from one paper to another. The ‘reader’ will visit the more influential journals more frequently and this frequency with which the journal is visited is then expressed as a percentage, resulting in the ES. A score of 1 indicates that the journal has 1% of the total influence of all the journals in the database. This gives the proportion of citations a journal receives rather than the absolute number of citations received, allowing for better comparison across disciplines. The score is calculated for each year and covers the previous 5-year interval. The ES is calculated by the Eigenfactor Project.33 It analyzes journals contained in the Thomson Reuters Journal Citation Reports.

Article Influence Score (AIS)

The ES is dependent on the number of papers published in a journal and can be increased by increasing the size of a journal. Bergstrom also created the AIS, which accounts for the average influence of an article in a journal, to allow for comparison of journals of varying size.9 The AIS is calculated by dividing the ES by the total number of papers published in a given journal in the previous 5 years. The scores are then normalized so that the average paper in the database has a score of 1. An AIS of 2 indicates that the average paper in the journal is twice as influential as the average paper in the database. By accounting for the average influence per paper in a journal, the AIS is directly comparable to the JIF. Like the ES, the AIS only analyzes journals contained in the Thomson Reuters Journal Citation Reports.

SCImago Journal Rank (SJR)

Similar to the ES and AIS, the SCImago Research Group created the SJR.11 The SJR is also computed using an algorithm that calculates journal influence. The SJR uses a random walker probabilistic model to give greater weight to citations from more influential journals. It conveys the number of citations, weighted by journal influence, which are received by the average paper in a journal. It is calculated using the papers published in the previous 3 years. The SJR is computed using the Scopus database and is published in the SCImago Journal and Country Rank.

Source Normalized Impact per Paper (SNIP)

Moed created the SNIP in order to measure the influence a journal has while also accounting for the variation in citation norms among disciplines.12 The SNIP uses the average number of citations in the discipline in which a journal belongs to weight the JIF. The SNIP is a journal’s average number of citations per paper divided by the ‘citation potential’ of the journal’s subject field, which is defined as the set of papers which cite the journal. The citation potential is the average number of citations per paper normalized by the median journal in the journal’s subject field. SNIP is computed using the Scopus database and considers papers published in the previous 3 years.

The journal bibliometric indices vary in the years over which they are computed and in the database queried. The indices provide similar information and there are no data to support superiority of one journal index over another. Various bibliometric values for JNIS are provided for illustrative purposes in table 2.

Table 2

Most recent journal bibliometric values for JNIS

Databases to query

Web of science

The Thomson Reuters Web of Science database covers journals published beginning in 1900. The journals in the database include the sciences as well as social sciences, arts, and humanities. Web of Science’s citation reports can compute the h index for an author and Thomson Reuters publishes the Journal Citation Reports which cover JIF and ES for journals.34

Scopus

Elsevier’s Scopus database includes journals from the sciences, social sciences, arts, and humanities. The database provides citation information for all papers published after 1996 and is currently increasing citation information for papers published before 1996. Scopus is updated daily. It computes the h index for authors and identifies individual authors with unique profile pages with tools allowing analysis of citation trends.35

Google scholar

Google Scholar includes academic papers from a variety of sources; however, it does not index journals. New papers are added several times a week; nevertheless, records can take months to update. Google Scholar creates author profiles and computes h and i10 indices.5

Variability in database results

While the h index and its variants can be calculated using three databases (Web of Science, Scopus, and Google Scholar), the results are unlikely to be equivalent. It has been shown that the value of the h index for individuals differs depending on which database is used.36

For illustrative purposes, we used the three databases to calculate the h index, m quotient, g index, and e index for one of our junior authors (KMF) and found significant variation in the values provided (table 3). The number of identified citations in Google Scholar was more than double that identified in Web of Science, resulting in substantial differences in h index calculation, which is particularly troubling given that the author has a unique name with no other authors in the world having the same first and last name. This variability is further amplified when performing queries of individuals with very common names. In fact, searching individuals with more common names, where unique identifiers are not available, often results in significant confounding due to difficulty in differentiating between specific authors with the same name. Further, it is likely that the variability in h index and citation count increases as the number of publications increases, suggesting that more prolific academicians may have even more variability between databases. Therefore, in its current state, academic productivity results determined via searches from these databases must be interpreted with consideration of the inherent limitations to each.

Table 3

Variation in bibliometric indices calculations based on database for a junior author.

Conclusions

Bibliometric measures are being used throughout medical research to assess the quality and quantity of academic productivity. Measures of academic productivity are especially important in the field of neurointerventional surgery because of its highly competitive nature. They can be used to compare and rank departments and institutions as well as play a role in determining academic advancement and funding. These objective measures provide interesting insights into the trends of academic productivity across individuals, departments, specialties, and journals. The h index remains the most commonly used metric for this purpose but it has important limitations. A number of indices have been created in order to address the weaknesses of the h index, but these are not widely used and have their own intrinsic limitations. Further, the databases used to compute these indices can provide widely differing values, as seen in our computation of the h index for one of our authors. A knowledge of the indices and their limitations is necessary for the modern researcher when considering measures of academic productivity and their use in medicine.

References

View Abstract

Footnotes

  • Contributors Drafting the article: RMG. Critical revision of article: All authors. Final approval of article: All authors.

  • Competing interests None declared.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.