Exams and Goodhart’s Law

I blogged recently about the difficulty of finding the right assessment system. Afterwards, in a throwaway remark on twitter, I said that finding the right assessment format was a bit like finding the right system for exchange rates – fiendishly difficult because you want one system to serve a multitude of purposes.

There are other ways in which I think economics and assessment face similar problems. Goodhart’s Law is a famous law that was invented by Charles Goodhart, a former Bank of England adviser. He observed that ‘as soon as the government attempts to regulate any particular set of financial assets, these become unreliable as indicators of economic trends.’ Extrapolating from that, you get the more general Goodhart’s Law: when a measure becomes a target it loses all value as a measure.

Exactly the same has happened with the 5 A*-C including English and Maths measure. Once upon a time it probably was a fairly reliable measure of how well a school was doing. Since it has become the government’s favourite target of how well a school is doing, it is now much less reliable as a measure. Schools have attempted to game the target using GCSE equivalencies, different exam boards and focussing their attention on C/D borderline pupils.

Campbell’s Law expresses the same point but in the field of social science. It was elaborated by the US social scientist Donald T Campbell and has been used in the US to explain the problems there with high-stakes testing. Campbell’s formulation is: ‘The more any quantitative social indicator (or even some qualitative indicator) is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.’

It’s for this reason that I think accountability measures should use more than one measure. I suggested a possible model for this a while back in this post and in this one. In the comments on the first post, Mr Chas made the following wonderful point:

Once you get to 4 or five measures, as the writer here says, you can ‘cheat’ on maybe one, but then as soon as you do it kicks the others way out of kilter. Like trying to squeeze into jeans a size too small. The fat has to go SOMEWHERE, it just pops up in a different place!

Of course, since we wrote that the government have adopted something similar – four key measures of accountability rather than one, plus an unspecified ‘destination measure’ to look at what pupils do after they leave school. I think this has to be an improvement on the previous system.