Evaluating Scientists: Citations, Impact Factor, h-Index,
Online Page Hits and What Else?
M Jagadesh Kumar
Editor-in-Chief, IETE Technical Review, Department of Electrical
Engineering, IIT, Hauz Khas, New Delhi-110 016, India
How to cite this article:
M. J. Kumar, “Evaluating Scientists: Citations, Impact Factor, h-Index, Online Page Hits and What Else?” IETE Technical Review, Vol. 26, pp.165-168, 2009
Identifying the key-performance parameters for active scientists has always remained a problematic issue. Evaluating and comparing researchers working in a given area have become a necessity since these competing scientists vie for the same limited resources, promotions, awards or fellowships of scientific academies. Whatever method we choose for evaluating the worth of a scientist’s individual research contribution, it should be simple, fair and transparent. A tall order indeed!
One common approach that has been used for a long time is to calculate the number of citations for the publications of a scientist and also see the impact factor of journals in which these publications have appeared. This approach, universally used as a decision-making tool, does have its limitations.
1. Citation Count
The number of citations for each publication of a scientist is readily available from different sources, e.g., Web of Science, Google Scholar and Scopus. It is generally believed that the impact of a researcher’s work is significant on a given field if his or her papers are frequently cited by other researchers. Usually self-citations are not included in such citation counts. However, using citation count alone to judge the quality of research contributions can be unfair to some researchers. It is quite likely that a researcher will have poor citation metrics (i) if he or she is working in a very narrow area (therefore with fewer citations)or (ii) if he or she is publishing mostly in a language other than English or mainly in books or book chapters (since most citation tools do not capture such citations).
2. Impact Factor
Publishing in a journal, such as Nature or Science, which has a high impact factor is considered very prestigious. In our profession, which deals with electronics and communications, it is a dream for many to publish in IEEE journals because some of the IEEE journals do have a high impact factor and their reviewing procedure is very tough. Impact factor is a measure of how frequently the papers published in a journal are cited in scientific literature. Impact factors are released each year in the Journal Citation Report by the Institute of Scientific Information (ISI) . Since its first publication in 1972, the impact factors have acquired wide acceptability in the absence of any other metric to evaluate the worth of a journal.
However, there are limitations in using the impact factor as a measure of the quality of a journal, and hence the quality of research of a scientist who publishes in a high-impact factor journal. For example, many people may read and use the research findings appearing in a given paper but may not cite these because they do not publish their work. In other words, impact factor measures the usefulness of a journal to only those who read and cite the paper in their publications, leaving out a large number of other practitioners of the profession who have not published but yet benefited from the research findings of a paper published in that journal  .
There are more than 100,000 journals published from around the world. However, ISI database includes only a small percentage of these journals. Therefore, if you publish in any journal which is not a part of the ISI database or if your papers are cited in the journals not listed in the ISI database, it will not add up to the impact factor calculation. Impact factors can also be manipulated. For example, in some journals, authors are forced in a subtle way to cite other papers published in the same journal. Therefore, blind usage of citation and impact factor indicators may not result in a correct evaluation of the scientific merit of a researcher.
3. The h-index
To overcome the problems associated with the citation metric and impact factor, in 2005, Jorge Hirsch of the University of California at San Diego suggested a simple method to quantify the impact of a scientist’s research output in a given area , . The measure he suggested is called the h-index. In the last few years, it has quickly become a widely used measure of a researcher’s scientific output. Without getting into the mathematical rigor of this approach, the meaning of the h-index can be explained as follows. Suppose a researcher has 15 publications. If 10 of these publications are cited at least 10 times by other researchers, the h-index of the scientist is 10, indicating that the other 5 publications may have less than 10 citations. If one of these 10, out of the 15, publications receives, let us say, 100 citations, the h-index still remains 10. If each of these 15 papers receives 10 citations, the h-index is again only 10. The h-index will reach 15, only if each of all the 15 papers receives a minimum of 15 citations. Therefore, to calculate the h-index of a scientist, find the citations of each publication, rank them according to the number of citations received, and identify the first ‘h’ publications having at least ‘h’ citations. To have a reasonably good h-index it is not sufficient to have a few publications with hundreds of citations. The use of h-index aims at identifying researchers with more papers and relevant impact over a period of time.
3.1 Limitations of the h-index
Caution needs to be exercised while calculating the h-index. The value of the h-index you get depends on the database used for calculating the number of citations. If you are using the ISI database, the same limitations that we have seen for calculating the impact factor will also apply here since ISI database considers only those citations in the journals listed in the ISI database. In general, it is found that Google Scholar gives a higher h-index for the same scientist when compared to Scopus or Web of Science. The scientific impact of any researcher can be calculated using Harzing’s freely downloadable tool called “publish or perish”.
There are several studies in literature to make the h-index more universally valid, but there is no consensus on using these corrections. For example, the introduction of the g-index is an effort to give some weightage to the highly cited papers , , . In a recent study, Liu has pointed out the case of two Nobel prize winners, each of whose h-index is less than that corresponding to a “successful scientist” . However, they still got the Nobel prize. Young researchers, whose research time span is short, are bound to have lower h-index values. A further limitation of the h-index is that it does not diminish with time and therefore cannot detect the declining research output of a scientist. Sometimes, the h-index may give rise to misleading information about a scientist’s contribution. For example, a researcher with 10,000 citations may have an h-index of 10 because only 10 of his/her papers have received a minimum of 10 citations; while another researcher with 650 citations may have an h-index of 25 because each of his/her 25 publications has received a minimum of 25 citations. In spite of all these limitations, there is now enough evidence to show that the use of the h-index has become popular and acceptable.
3.2 Finding Your h-index
One way of overcoming the limitations of the database used by the Web of Science, Google Scholar and Scopus is to first develop a habit of periodically collecting all the citations of your papers from different sources, including the above three sources. You can then rank them and pick up the top ‘h’ publications with a minimum of ‘h’ citations. This will give you the h-index of your scientific output. You however have to maintain a list of all your citations and the complete bibliographic information on the citing source, irrespective of whether it is a book, conference paper, journal paper, PhD thesis, patent or non-English source. The carefully maintained bibliographic data will be a proof for the reliability and authenticity of your h-index calculation. Just to give you an idea, the peak h-index of many Nobel prize winners in physics during the last two decades is around 35 to 40 
4. Mentoring Abilities
Recently, Jeang has argued that in addition to the above performance metrics, we should also measure the mentoring abilities of a scientist . If the coauthors of a scientist are his or her own trainees or students and if they continue to make a scientific impact after leaving their supervisor, it does point to the quality of the mentoring by the scientist and to the impact made by the scientist, as a result of his/her mentoring abilities, in a given area during a given period. This is a very important but totally neglected aspect of the contribution made by a scientist or an academic. However, we do not yet have a well worked out formula to measure such mentoring abilities.
5. Online Page Hits
In recent times, most journals have gone online, with open access, and it is very easy to keep track of the number of visitors to the journal’s website. For example, in IETE Technical Review, you can see how many times an article has been viewed, emailed or printed. A recent study shows that high viewership does lead to high citations, and highly cited articles do not necessarily have high viewership. The online viewership data includes (i) those who simply read and (ii) those that read and also publish citing the paper they have read . The citation data includes only the latter group, while the viewership data includes both the groups. Therefore, it may be appropriate to use the number of views for a paper as a measure of its impact and popularity provided the website avoids counting the repeat page hits from the same computer within a given period.
6. Skewed Performance Metrics
Whatever performance metrics we may use, it appears that authors from developing countries do face certain constraints in terms of achieving higher performance indices and therefore recognition for themselves and their country. It is quite possible that authors from advanced countries may tend to cite publications from organizations located in their own countries, leading to a disadvantage for authors working in difficult situations, with less funding opportunities . This is bound to affect the h-index of scientists working in developing countries. Since there is a limited page budget and increased competition in many “high-profile” journals, it is not always possible to publish in these journals. One way to overcome this problem is to encourage and give value to papers published in national journals. There are many scientists from developing countries such as India working in highly developed countries with advanced scientific infrastructure and huge funding. These scientists should seriously consider publishing their work in journals originating from their native countries. This will bring an international flavor to the national journals, attracting more international authors and ultimately making them mainstream international journals. When these journals become more visible and easily accessible through their online versions, there is a chance that papers published in these journals are more often cited. This way, the skewed calculation of the h-index and other performance metrics for scientists from developing countries may be minimized.
Exuberant dependence on single numbers to quantify scientists’ contribution and make administrative decisions can affect their career progression or may force people to somehow enhance their h-index instead of focusing on their more legitimate activity, i.e., doing good science. Considering the complex issues associated with the calculation of scientific performance metrics, it is clear that a comprehensive approach should be used to evaluate the research worth of a scientist. We should not rely excessively on a single metric. Since the h-index is now becoming more popular and is simple to calculate, we should use it judiciously by combining it with other metrics discussed here.
As always, please do not hesitate to contact me and let me know your views.
1. Available from: http://science.thomsonreuters.com/index.html
2. O. Yoshiko, and A. Makoto, “Pitfalls of citation and journal impact factor devises in research evaluation,” Journal of Science Policy and Research Management , vol. 20, pp. 239-58, 2005.
3. Available from: http://arxiv.org/abs/physics/0508025
4. J.E. Hirsch, “An index to quantify an individual′s scientific research output,” Proceedings of the National Academy of Sciences of the USA , vol. 102 (46), pp. 16569-72, 2005.
5. Available from: http://www.harzing.com/pop.htm
6. L. Egghe, “How to improve the h-index,” The Scientist , vol. 20 (3),
p. 14, 2006.
7. L. Egghe, “An improvement of the h-index: The g-index,” ISSI
Newsletter , vol. 2 (1), pp. 8-9, 2006.
8. L. Egghe, “Theory and practice of the g-index,” Scientometrics ,
vol.69 (1), pp. 131-52, 2006.
9. S.V. Liu, “Real discrepancy between h-Index and Nobel prize-winning,” Logical Biology, vol. 5 (4), pp. 320-1, 2005.
10. K.T. Jeang, “H-index, mentoring-index, highly-cited and highly-accessed: how to evaluate scientists?” Retrovirology , vol. 5, Article Number: 106, Nov 2008.
11. A.W.A. Kellener, and L.C.M.O. Ponciano, “H-index in the Brazilian Academy of Sciences – comments and concerns,” Anais da Academia Brasileira de Ciκncias (Annals of the Brazilian Academy of Sciences) , vol.80 (4), pp. 771-81, Dec 2008.