22/9/2008

Fundamentally Flawed Numbers

Filed under: — Bravus @ 10:04 am

Some of us in the School of Education did an activity last week where we had to look at some statistics about our programs and compose a response to be sent to the Faculty and the University. There were all sorts of stats, on enrolments and attrition (students who start but don’t finish courses), grades, student satisfaction and so on.

If there were any places where our stats were unusually low we were required to explain the reasons for it. So for one semester – the first semester of this year – the student satisfaction surveys were unusually low. For the immediate preceding semester last year they had been unusually high.

As we tried to figure out the reasons for these odd results, we realised that we tend to administer those surveys to only some courses each semester, not to every course. It’s almost random which courses and lecturers and students are included in the average scores for each semester.

As anyone who understands anything about statistics will tell you, that makes the results almost useless, and certainly not able to be meaningfully compared with one another semester by semester. We understand this. The people we are going to report to also understand it. And yet, here we are, in the position of having to explain ourselves.

Look, I have no problem with issues of accountability. I don’t even have a real problem with a numbers-based approach to accountability, although I think a mixed-method approach that mixes numbers with some narrative or other description is probably richer.

But what I really object to – and it happens at all levels, including use of student results to rate schools and teachers – is when everyone concerned *knows* very well that the numbers are fatally flawed, but yet because the numbers are there just treats them as though they are facts and need to be accounted for – in some way other than through actually recognising that they are meaningless.

(I like to think I would have written this rant even if the numbers showed us up in a good light…)

One response to “Fundamentally Flawed Numbers”

  1. Donna says:

    David – couldn’t agree more with your rant!!!!

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.