Background in bibliometrics and their use in research evaluation Background in bibliometrics and their use in research evaluation

What are bibliometrics?

The term 'bibliometrics' refers to statistical measurements of research publications and can be applied to a journal, author or article.

Why does this matter?

The digitisation of journals has meant that citation can be measured much more easily.  Although there is debate on the usefulness of such measures, and statistical analysis of citation information has varying importance in different research fields it is likely that it will continue to have influence.  The 2014 Research Excellence Framework funding process used citation to 'inform' the expert panel review of research in some areas.  RAND Europe produced a 2009 report on the use of bibliometrics in health research, but of interest to other disciplines,  highlighting how these should be used in combination with other factors such as case study analysis and peer review.

Some of the common measures used: 

Citation analysis - a calculation of how many times an article is cited elsewhere - this generates a measure for the individual article.  It is based on the idea that an article that is cited has research that is useful to others.   This will be influenced by the age of an article.  Concerns have been expressed that individuals could self-cite to boost their rating however investigation of self-citation during the REF pilot did not suggest that it is statistically significant in distorting citation rates.

H-index - a calculation of an individual's productivity and impact in their field.  A researcher has an index of h if s/he has published h articles which have been cited h times.  The H-index will vary depending on the source of the citation information - ie Scopus and Web of Science will produce different figures as they draw on different journal bases.

Impact factor - a calculation of the importance of a journal, using the number of citations divided by the number of articles in the 2 previous years as a measure.  This is published in 'Journal Citation Reports' from Thomson Reuters, to which UEA library subscribes.  

However see this blogpost for a discussion of an alternative to journal impact factor.

Eigenfactor Score - Eigenfactor Score has been developed to measure scientific journal citations in what is suggested is a more robust way than the impact factor.   It uses analysis similar to Google page ranking to identify ratings.  

The key bibliographic databases at UEA, Web of Knowledge and Scopus provide citation information and tutorials on usage of this. 

What about Google Scholar?

Google Scholar is in effect a bibliographic database that is freely available.  Whereas WOK and Scopus publish their sources, Google Scholar does not list where its articles are from.  However a study¹  in New Zealand demonstrated a correlation between the research quality output and Google Scholar citations.  Harzing considers it a worthwhile resource, but adds the caveat that less mainstream work may not be recognised, and low citation score within Google Scholar does not necessarily correlate with lower quality research. 

In 2011 Google released Google Scholar Citations, which allows researchers to track citations to their work, and also to calculate some citation metrics.  Information on signing up

¹Smith, A.G. Benchmarking Google Scholar with the New Zealand PBRF research assessment exercise (2008) Scientometrics, 74 (2), pp. 309-316 

Other web resources

Thomson Reuters who publish Web of Knowledge provide presentations explaining some of the issues to consider when measuring bibliometrics.

This is a presentation from Thomson Reuters entitled: 'Using bibliometrics: A guide to evaluating research performance with citation data'

A group of Irish universities have provided useful tutorials in a package called MyRI (My Research Impact) on understanding bibliometrics & their effect on your research impact

More detailed explanations and analyses of terms are displayed in this table copied from Table 2.2 in  Ismail, S., Nason, E., Marjanovic, S., & Grant, J. (2009). Bibliometrics as a tool for supporting prospective R\&D decision-making in the health sciences: Strengths, weaknesses and options for future development. Distribution: RAND Corporation.
Retrieved from and reprinted with permission of Rand Europe.