In the Shadow of Bibliometric Analysis
Posted by gazjjohnson on 19 March, 2009
As anyone who’s following me on Twitter knows, the last week or so has been rather dominated by my work on bibliometrics. Let me state up front here, I’m not a bibliometrician (sounds worryingly close to mathamagician to me) nor statistician, rather I’m a former scientist who spent a lot of time working with stats in another life. I sat in on a meeting about statistical teaching last week which served to rather poitnly remind me of all the things I used to know how to do (linear regression, chi squared, two tailed T-tests etc).
On the other hand I’ve always quite enjoyed working with data collection and simple anaysis; when I was a library researcher at Univ Warwick I spent quite a bit of time doing just this. So this does mean that any outputs that I produce aren’t going to be stunningly complex, but they should help people to get a picture based on fact. This, and my role as LRA personage involved in the Research Excellence Framework (REF) preparations, are doubtless why I was tapped by Louise to run a bibliometric profile of the Chemistry dept.
Bibliometrics, in case you didn’t know, is the analysis applied to texts or information. In this case I was asked by the Dept. to run a sample profile of their publication outputs; in an attempt to establish where they stand in relation to the rest of the academic world. In practise this meant taking a sizable sample (half the departmental academics) and looking at which journals they’ve published in over the last 9 years (2001-date). This is a key range for a number of reasons – firstly due to the suggestion that the REF will take account of publications back to this date. It’s also due to the fact that Journal Citation Reports (JCR) only goes back to 2000 online, so it’d be harder work to analyse publications beyond this point.
Now whilst the results are naturally confidential at this point I can tell you about what I sample in brief
- Article outputs– Number of articles produced and indexed within the time frame.
- Citation counts – Number of references to articles produced.
- H Index– The Hirsh Index quantifies both the actual scientific productivity and the apparent scientific impact of a scientist. It is increasingly viewed as a major indicator of academic esteem. Anything over 100-120 and you’re into Nobel laureate territory.
- Journal Impact Factors – A measure of how often all articles in a specific journal have been cited. Usually the most common “How important is this journal?” value.
- Cited Half Life – measures the number of years, going back from the current year, that account for half the total citations received by the cited journal in the current year.
From these I’ve been able to profile the departmental research publications as a whole, as well as getting an idea about the impact of the individual contributions to it. Quite looking forward to discussing the results with Chemistry in the near future.
The biggest challenge (data collection aside, which currently is very long winded) is knowing when to stop. I’m still very new to bibliometrics, and my inner scientist kept suggestion other ways to contrast the data or to analyse it. Essentially I could have been at this for weeks. And since we’re still not quite sure what metrics the REF will be using there didn’t seem much point in going to far with the first attempt.
There’s also the question of benchmarking. Raw stats on our depts are all well and good – but where do they stand contrasted with the rest of the world? That’s something that I might need to follow up on, but would likely be a far more time consuming operation than a week’s work. For now the Chemists might just need to trade notes on H Factors held by comparator academics, in contrast with their own.