EMALINK – Bibliometrics & Research Visibility

A rather belated report on this May Emalink event:

What are Bibliometrics and why should I care?  Ian Rowlands (University of Leicester)

• Bibliometrics can be very sterile & specialist so they must be used in a context that makes sense.
• Citation data – indicates relationships of influence, deference & usage – a bit like social media networks.
• Bibliometrics have to help the institution or individual in the research process.
• BUT bibliometrics just one small par of the puzzle and tools available.
• How much information is there really out there about research inputs & outputs?
• Data can be variable e.g. to pick up on Univerisity of Leicester citations then authors need to put University of Leicester in their address.
• Currently it is difficult to deal with the variety of research outputs e.g. data, software, plays…
• New tools emerging e.g. Readermeter from Mendely to see if your papers have been socially bookmarked.
• IMPACT of research – very important for REF but citations do not always translate to real world impact – need to go beyond bibliometrics.
• Some types of citations have greater ‘weight’ in terms of impact e.g. citation in a NICE guideline directly impacts how healthcare is provided.

Enhancing research visibility at Loughborough (Lizzie Gadd)

• In 2011 Loughborough found it had slid down the THE World rankings and needed to improve their citations count.
• The Plan to improve citations = library to run sessions on publishing & promoting research, VC commissioned Academic Champion for bibliometris, promote visibility of good research in high impact journals, recruit & retain good researchers, ciations taken into account when promoting, use ResearcherID and Google Scholar profiles to improve citations & impact & use research repository.
• Training Implementation = publish or perish sessions for new academics, lunchtime bibliometrics seminars in Depts/Research groups, 1to1 appointments ion request and online tutorials on citation tools and impact tools.
• Plus provide bibliometric data to support staff and promote bibliometrics training through staff conferences, webpages, blogs & newsletters.
• The Vision for the future = joined-up thinking (work with research office, IT service etc), research visibility focus (databases of research kit, data and publications).
• Already seeing improved citations.

Some good ideas that could be implemented elsewhere.

Research training will be high on our agenda once we get our Library Research Services team fully in place, headed up by our own bibliometrician Ian Rowlands. I’ll be moving over into that team later this year.

REF Pushed back to 2013/14

It’s not come as a big shock to anyone that the new Government have made good on one of their pre-election promises to push back to the REF-2012.  You can read more about it and the response from the sector at the following locations:

• THE on the anticipated delay
• HEFCE Response
• BBC News takes a slightly different tac

What this means for Leicester will be interesting to explore over the coming weeks, with especial reference to the preparations for REF that are already underway and the LRA in particular.

Ready for REF CERIF Workshop (King’s March 2010)

This Tuesday I travelled down with Steve Loddington of the Research Support Office, to King’s College London’s Waterloo Campus to attend a Ready4Ref (R4R) CERIF workshop. Following an overview of the day from Mary Davies (Kings) the day proper began. What follows are my notes and comments on the sessions, hopefully slides from the event will be available online shortly.

CERIF4REF, Richard Gartner, Kings College
CERIF standard is very complex, almost too complex for most users to fully understand. The RAE2008 was used to shape CERIF as REF2012 standards remain as of yet unannounced. Repositories and CRIS outputs on systems that adhere to the standard can be feed through CERIF4REF and provide a single output to the REF assessors (in principle). There is a data dictionary that defines the standard and the elements within it. Going to take RAE data from Kings and process as part of a trial to check that this works well.

CERIF, CRISes & Research Databases Marc Cox
Marc talked about King’s CRIS, developed in house 2004-7, and developed originally as a research management tool; although the RAE overtook and drove it towards administrators rather than academics which was not the original intent. Took data from HR, student, awards and finance & publications from TR WoS (author ID a problem) – now use the WoK API to take data, although that was quite a challenge. At the moment administrators (mostly) and academics (few) are keeping the publications up to date.

CERIF is a standard data model that describes research entities and their inter-relationships, originally developed with support of EU. It is architecture independent. 4 main data fields from RAE2008 taken for CERIF4REF. A number of system and data tweaks were needed to these four research data fields to make it compatible with CERIF. RA1 data was relatively easy, although RA2 data was more difficult to map. RA3a/b and RA4 couldn’t be mapped without the base data which created them.

Benefits from the approach however included RAE forms generated from style sheets that can be cross compared with php scripts to check accuracy. Next steps are to generate real King’s data in CERIF xml format and exchange data with other CERIF compliant systems.

Using ISI Web of Science Data in Repositories, Les Carr
EPrints has had plug ins that do this for a while on an individual basis, but due to change in licenses now have access to API for direct deposit. SWORD based ISI Deposit, for EPrints was examined, although as Les noted the technology wasn’t at the heart of the issue as all repositories work in a similar fashion in the big picture. There is a need for a repository editorial step – which is a manual step, so can be like drinking from a fire hose – too much data flooding in and how can you deal with it with established workflows. The data download may not be straight forward exercise, e.g. student papers and non-peer-reviewed items are listed on WoS as well as academic papers. Les showed an example of selecting one academic and the process to go through to weed out the non-relevant items (a manual process) – 38/items ingested initially a minute, about 10 minutes for manual process and removal of duplicates and irrelevant. Questions of how to use this – monthly update? On a per user basis?

Les moved on to look at repositories as a CRIS – since repositories manage research or teaching or academic outputs and are broader in description and purpose. But what about other databases and information resources across campus (Finance, HR, Grants database. CRISes pull all the disparate data together and present a unified view of it; which includes the repository. Eprints has attempted to accommodate the CERIF data – not just publications but projects and organisations.

E.g. previously a project was added in the metadata – now they are objects in their own right, linking from metadata record to a page about the project itself; with contributors rather than authors. Data can be exported and imported in CERIF format. This joined up integrated resource can help develop research case studies for demonstrating impact and output. I imagine useful though this is, it does add yet another load to the already busy repository administrators workflows. However, I can see a significant advantage to the repository that offers this kind of joined up service. I doubt Leicester will go this route, given our interest in a separate CRIS systems at the heart of the research management agenda.

Discussions
After a brief Q&A session we moved onto lunch. After lunch we broke into two discussion groups, one looking at the perceived benefits or flaws in CERIF; along with the practicality of auditing and standardising institutional systems with it. These sessions then reported back on the points that had been raised. Notably on average for those in attendance having data information systems that could be audited and made compatible with the CERIF standard was a reasonably attractive opportunity, however there were mixed concerns on the technical expertise being available in house at short notice to participate. When it came to staff resource available to take part in such an audit, virtually the entire group felt that this was the biggest obstacle to overcome.

Overall this was an interesting day, and while it was more on the CERIF data standard than the REF itself as I had hoped I was still able to take away some points for further thought.

[Edit: Slides from the event are now available here: http://www.kcl.ac.uk/iss/cerch/projects/portfolio/r4r.html]

Terms of REFerence – workshops and news from HEFCE

Those of you interested in the preparations for the REF (which with my LRA hat on I’ve been somewhat involved in here at Leicester) will probably be interested in the following publications from HEFCE website

• Publication of discussions from the ProVCs for Research March workshop
• Presentation: the issues and challenges for discussion by David Sweeney, Director of Research, Innovation and Skills, HEFCE

The second of these makes for an easier read, though those of you very interested in the REF will find a lot of information in the ProVCs workshop papers.  I must confess most of that paper covers areas that I’ve got very little interaction with, interesting though the insights into the thought processes at that level might be.

David Sweeney’s presentation though gives a rather thorough examination of why we need the REF (or something like it) stemming from the lessons from the RAE 2008, from the funders’ point of view.  And if the funders think it is important, then those of us in HE need to think about it a lot as well.

From what I’ve seen so far – it’s either going to be very, very complex in gathering the data, esteem factors etc and take a lot of  staff time to support.  Or it’s going to end up being quite easy for most people, but a fair amount of work for a very few administrators.  Role on the June workshop at Kings to tell us more!

Thompson Reuters InCites Bibliometrics Meeting (part 3)

[Continued from parts 1 and 2]

Emma Swann (Target Account Manager) started to talk more about InCites. Previously TR was able to produce these kinds of reports as on-demand CD outputs. InCites is the Web based version of this, in basic and premium versions.

Basics offers citation reports and indicators (benchmarking contrasts with other institutions). Premium is a more customised service, that is set up working with an institution. She demonstrated the information that the system would pull out, which was rather impressive – stuff that would take me several days to generate could be produced more readily. I was glad to have worked at the coal face generating this data, as it enabled me to readily see the importance of the information proceed. That said there were some metrics I’d produced [link] that weren’t evidenced produced. Possibly InCites could still produce them, but I’d need some time hands on before I could say that. All the same the time the system would save in generation of this information would allow the manual discovery of this information if push came to shove.

Emma showed how it was possible to generate custom benchmark reports for a range of institutions at an author, discipline or institutional level. It was even possible to rank all of an institution’s researchers easily.

Basic InCites package includes:

• 1 standard citation report with all current metrics back to 1981
• One standard indicators report
• Quarterly data updates
• Internal distribution only

Premium package includes all this plus:

• Allows posting of data on external website
• Can use researcher ID (RID)
• PubMed ID match
• Match retrievable service with WoS

In terms of cost (depending on institutional JISC band A-E):

• Basic
  • U$D10-25k
• Premium
  • U$D16-40k
• API Citations
  • U$D6-15k

Wellcome talked about their funders point of view, and how identification of authors was only the start. What they wanted was a system that would interact with their own databases allowing them to call up extensive data on researchers they fund – in essence answering the questions “Is this person worth what we are funding” along with “Are there areas of funded research excellence that we are not funding but should”.

Comments from the HEFCE representative continued the discussions about on unique RIDs and carry forward between institutions. UCL commented that with 1,100 address variants it was a real problem IDing researchers. Some researchers identify with a unit or division and not an institution more readily and thus ensuring all are covered can be a problem.

The HEFCE rep suggested that bibliometrics probably won’t be used for Arts/Humanities and many social sciences reviews, noting that the finer detail of bibliometrics has a long way to go in being resolved. For example some subject areas publish in low level journals, because the whole field is publishing in these journals – it’s all relative.
HEFCE also commented that the 2010 bibliometrics exercise may be developmental rather than a full review, but this while this is not a certainty, the time taken for the pilot was far in excess of what they expected. They have a commitment to run something, but it might not be what everyone initially expected. It should be possible for institutions to see where they stand with the first real funding related REF taking place in 2013. Autumn consultation workshops will be run in 2009 to inform REF, and then later workshops to inform submission guidelines. A comment was made that provided your publications management system is robust and embedded in custom and practice then you may have less to worry about w.r.t. REF.

On the subject of competing products that do a similar job to InCites, the representative from UCL suggested that there were other resources they were looking at, but declined to name them.

Finally HEFCE talked about the CERIF metadata schema, which may well be a REF requirement. It has a high use on the continent, but much lower in the UK. Scandinavian countries have been using it a lot for example. EuroCRIS and JISC are involved and advocates of it. Noted that a number of Scottish Universities are piloting it.

As you’ve seen in this and the previous parts of this post, this was an especially information rich day with a lot useful, and sometimes surprising, information coming out. What does this mean for the REF and bibliometrics at Leicester? I think it’s too soon to say, but it certainly means I’ll be having a lot more conversations about them in the near future I suspect!

Thompson Reuters InCites Bibliometrics Meeting (part 2)

[Continued from Part 1]

Jon continued mentioning that an API is available to extract the data from InCites– although it does need to be enabled by Jon or Emma as a request. Another ability of the software is the creation of create researcher ID (RID) profiles (unique). Using the WoS article matching tools it is possible to find records and create links to them. From this you can create local database of publishing output.

A comment from the UCL representative was that academics haven’t been that keen to sign up for these IDs and noted that there are other unique identifiers such as HEFCE. Jon countered that with these IDs it is much easier to ID researchers and works. A short debate followed discussing the practicality of academics keeping these unique IDs between institutions. It was felt that for simplicities sake most institutions would issue new unique IDs to new staff, which rather made the usefulness of this aspect of the service somewhat diminished.

For repositories that use the UT tag on various records, it is possible for InCites to make use of local data (if your research IDs and data are clean/clear enough).

Can evaluate citations counts from institutional repository contents possible to purchase other institutions data, but can’t expose the information. There was a discussion noting that what the REF wants is driving this as a central process, but unclear what HEFCE wants – hence everyone is adopting a wait and see until the June results of the REF pilot are presented. It was noted that whilst the word bibliometrics is much muttered, but in terms how, what and why remains very unclear within most institutions.

The question of citations from patents was raised (e.g. Derwents Citations Index) – would these be of value? The answer wasn’t currently clear. The question of staffing challenges was raised, which whilst the Thompson tools would help wouldn’t alleviate all the challenges.

UCL spoke about their experience as a pilot for the REF – the experience has been a valuable one and a bit of a shock too. Went beyond where just REF wants them to go, and where the data would be of see and found that the systems they had weren’t good enough and neither was the data. More pressingly there was a need for a cultural overhaul in how researchers record their research output and usage. As a result they have a separate a project to address this.
The HEFCE rep present commented that academics still keep sourcing their own research data from different sources for different purposes, and that this was not necessarily a good thing.

[Continued in part 3]

Thompson Reuters InCites Bibliometrics Meeting (part 1)

On Monday I travelled down to London to a meeting organised by Thompson Reuters (TR), providers of the ISI Web of Knowlegde and Journal Citation Reports to name but a few key resources. Principally the day was to introduce and give an overview of their new InCites bibliometrics product (to be launched in May 2009), but thanks to the mix of people there from universities (myself, Kingston, UCL) and funders (Wellcome, HEFCE) some very interesting discussions around the subjects demonstrated were presented. All of this naturally is related to the REF.  As this is a long report on a very full discussion, I’m going to break it up into multiple posts.

The day and background to the product was introduced by a senior manager, who talked about the breadth of coverage of TR’s products, noting the importance of everyone having access to the same quality data for evaluation. He mentioned that the acquisition of Evidence (based in the UK) has allowed the provision of services and tools along with access of the data themselves, even the generation of customised reports. Finally he talked about TR’s development of grant application systems as one of their next major launches.

Jon Stroll (Key Account Manager) presented on aspects of research analysis including using data integration within institutional work flows such repositories. Focussing first on the Web of Science (WoS) and mentioning that the conference database is now also a citations database. However, Jon was unsure if the REF will take account of this data. In terms of quality TR use manual and machine harvesting of each article’s data. InCites is essentially a benchmarking citation analysis tool. He noted that currently the HEFCE Pilot,the US National Science Foundation and EU are all using the data from Thompson Reuters

Key questions that InCites can help answer include:

• Over all published output in 10/20 years
• Impact and how frequently has it been cited
• Which papers are the most influential and how do they compare to the benchmarks?
• Who are the top (H Indexed) authors and therefore where should research funding be focussed
• What research do our researchers pull on – are they citing the right material

The extended license allows for use of ISI bibliographic data within IRs (populate the meta data using ISI WoS data). The current policy is now that you can use the data and expose externally with no additional charge. The only mandatory requirement is that you have a WoS subscription. A comment from Susan Miles that the repository community are currently unaware of this and would really benefit from knowing.

[To be continued in Part 2]

In the Shadow of Bibliometric Analysis

As anyone who’s following me on Twitter knows, the last week or so has been rather dominated by my work on bibliometrics.  Let me state up front here, I’m not a bibliometrician (sounds worryingly close to mathamagician to me) nor statistician, rather I’m a former scientist who spent a lot of time working with stats in another life.  I sat in on a meeting about statistical teaching last week which served to rather poitnly remind me of all the things I used to know how to do (linear regression, chi squared, two tailed T-tests etc). 

On the other hand I’ve always quite enjoyed working with data collection and simple anaysis; when I was a library researcher at Univ Warwick I spent quite a bit of time doing just this.  So this does mean that any outputs that I produce aren’t going to be stunningly complex, but they should help people to get a picture based on fact.  This, and my role as LRA personage involved in the Research Excellence Framework (REF) preparations, are doubtless why I was tapped by Louise to run a bibliometric profile of the Chemistry dept. 

Bibliometrics, in case you didn’t know, is the analysis applied to texts or information.  In this case I was asked by the Dept. to run a sample profile of their publication outputs; in an attempt to establish where they stand in relation to the rest of the academic world.  In practise this meant taking a sizable sample (half the departmental academics) and looking at which journals they’ve published in over the last 9 years (2001-date).  This is a key range for a number of reasons – firstly due to the suggestion that the REF will take account of publications back to this date.  It’s also due to the fact that Journal Citation Reports (JCR) only goes back to 2000 online, so it’d be harder work to analyse publications beyond this point.

Now whilst the results are naturally confidential at this point I can tell you about what I sample in brief

• Article outputs– Number of articles produced and indexed within the time frame.
• Citation counts – Number of references to articles produced.
• H Index– The Hirsh Index quantifies both the actual scientific productivity and the apparent scientific impact of a scientist. It is increasingly viewed as a major indicator of academic esteem.   Anything over 100-120 and you’re into Nobel laureate territory.
• Journal Impact Factors – A measure of how often all articles in a specific journal have been cited.   Usually the most common “How important is this journal?” value.
• Cited Half Life – measures the number of years, going back from the current year, that account for half the total citations received by the cited journal in the current year. 

From these I’ve been able to profile the departmental research publications as a whole, as well as getting an idea about the impact of the individual contributions to it.  Quite looking forward to discussing the results with Chemistry in the near future.

The biggest challenge (data collection aside, which currently is very long winded) is knowing when to stop.  I’m still very new to bibliometrics, and my inner scientist kept suggestion other ways to contrast the data or to analyse it.  Essentially I could have been at this for weeks.  And since we’re still not quite sure what metrics the REF will be using there didn’t seem much point in going to far with the first attempt.

There’s also the question of benchmarking.  Raw stats on our depts are all well and good – but where do they stand contrasted with the rest of the world?  That’s something that I might need to follow up on, but would likely be a far more time consuming operation than a week’s work.  For now the Chemists might just need to trade notes on H Factors held by comparator academics, in contrast with their own.

Finding an author’s h-index – a step by step guide

The h-index (Hirsch Number) is a metric that is increasingly becoming of interest to researchers, especially in the light of the REF.  An h-index is “a number that quantifies both the actual scientific productivity and the apparent scientific impact of a scientist“.  You can work it out manually, but to be honest you’d need to be mad or a bibliometrics fiend to want to.

I’ve been asked by a few people how to find it, and each time I totally forget how!  So in the light of this, here’s my step by step guide to discovering an author’s h-index automatically using that wonderful Web of Knowledge tool!

1. Go to Web of Knowledge  and click on the big green button
2. Click the Web of Science tab at the top of the screen
3. Rnter the author’s name in the format surname initial* (e.g. raven e*)
4. Change the search option from the drop down menu to Author
5. Click Search
6. At the top right of the results is the option to Create Citation Report. Click this.
7. The analysis appears, along with the person’s relative h-index.

It seems simple, but I was scratching my head using WoK until I discovered that I need to just use Web of Science, not the whole WoK in order to get the value.  And so, now you know!  It is worth noting you do have to be fairly exact in your author naming conventions, as the citation report will not run for more than 10, 000 result records.

I did wonder if between steps 6 and 7 about selecting individual papers from the list of results, but it appears that this has no effect on the citation analysis; for example selecting 5 papers from a list of 120, 000 doesn’t enable me to run the citation reports – it appears to run in an all or nothing manner.  Or maybe there’s a trick here I’m missing?

Publicationslist.Org – is it useful?

A former collegue of mine, working in the Open Access field, mentioned this service to me over the weekend – PublicationList.org.  In their own words this site

PublicationsList.org exists to let researchers and research organizations maintain a reliable web-based record of their academic output without any fuss.

It rang a bell, and I thought back to the demo from Symplectic a few weeks back for their software; which certainly seems to offer some similar functionalities.  The difference being Symplectic is hosted and maintained for an individual institution.  PL.Org on the other hand is institution independant.

Helpfully SHERPA have blogged about PublicationList.Org in far more detail than I’m going to go into.  So is this service useful?  6,000 academics think so; and whilst that’s pretty small potatoes on a global scale it certainly is a reasonable mass.  What do the rest of you library/repository types think?  Will this service have a broader impact?