26/7/2006

‘Quality’

Filed under: — Bravus @ 1:35 pm

We’re currently undergoing an activity at our university – and it will soon be expanded to all Australian universities – whereby the attempt is made to measure not just the quantity of publications but the quality.

Sounds like a good idea on the surface – just counting published papers was never all that useful as a measure of productivity, just because it’s very easy to publish crap, or to publish essentially the same article multiple times.

The devil, of course, is in the detail. No-one has time to actually sit down and read all the papers to check for quality. Citations are a decent measure, but they take years to occur, and there aren’t really good measures of citations in the social sciences like there are in the sciences.

So they’ve pretty much decided to use ‘quality of the journal in which the paper is published’ as a proxy for ‘quality of the paper’. It’s at second remove, but presumably it’s easier to publish good stuff (and harder to publish crap) in a good journal with high standards and good peer review.

Trouble is, who decides on the quality of the journals? And how narrowly specific should it get (e.g. if journals are chosen and ranked across the whole education faculty, how many science education journals will get in and who will judge their quality?) Presumably these are questions that could be answered in good and subtle ways, but the signs so far aren’t encouraging…

And the hammer is too big: at this stage, a paper published in a Tier One journal is worth five times as much as one published in a Tier Two journal. So if you want to publish in the latter, you have to write 5 times as many papers… Just makes it a lot harder to get published in the ‘good journals’… and likely also kills off startup journals that could potentially make a big contribution.

I’m all for measuring quality, and measuring achievement in better ways, but when ‘quality’ is just reduced back to quantity (a weighted number of papers in different levels of journals), and when ‘quality’ is defined in terms of a proxy several steps removed, just because that’s what’s easy to measure…

One response to “‘Quality’”

  1. Bravus » Feels Good says:

    […] We’re currently doing a ‘Research Quality Assessment’ (RQA) at our university – I think I ranted about it before. As one stage of it we were asked last week to send in our total publications over the past 5 years. I had 2 books, 3 book chapters and 10 journal articles. I was feeling a bit sheepish about the journal article total: I expect more from myself than 2 a year, but realised that books take more time and have impacted on the pace in articles. […]

Leave a Reply

Your email address will not be published.