The adverb problem

In a previous blog I looked at how hard it is to set exams based on criteria. I quoted Tim Oates:

The first issue is the slippery nature of standards.

Even a well-crafted statement of what you need to get an A grade can be loaded with subjectivity – even in subjects such as science. It’s genuinely hard to know how difficult a specific exam is.

It’s genuinely hard to set exams based on criteria, and it is also hard to mark exams only using criteria. Most of the exam criteria for essay based questions suffer from what I call the ‘adverb problem’. Often, the difference in criteria between the top few bands is the difference of an adverb – getting a band 3 may require you to write ‘well’; getting a band 4 requires you to write ‘fluently’. Etc. Here’s an extract from the WJEC exam board criteria for English.

Band 3: there is some use of devices to achieve particular effects

Band 4: devices to achieve particular effects are used consciously and effectively

And this:

Band 3: plot and characterisation are convincingly sustained

Band 4: plot and characterisation are effectively constructed and sustained

This last one is particularly interesting as I would think that ‘convincingly’ is better than ‘effectively’, but that’s by the by. The point is these adverbs are extremely vague and their meaning lies in the eye of the beholder. The only way we can give these criteria any kind of concrete and specific meaning is by referring to actual samples of pupil work – which is of course what we all do, in internal and external moderation days. Criteria are only given meaning by reference to individual pupil performance. That is, criteria are given meaning by norms.

Let me give another example of this. When I was first marking a set of creative writing coursework tasks, I came across one that I felt deserved the top band. One of the criteria for the top band was something like ‘appropriate and controlled variation of sentence structure.’ This pupil’s sentence structure was fairly (there is another adverb for you) well controlled and appropriate, but there were moments where it wasn’t completely secure. Did it deserve the top band? I asked my mentor, and she told me what an exam moderator had said to her, which was that when awarding the top band, you have to take into account what it is realistic for a 15 year old to achieve. I have no idea if this really is official exam board advice, but it seemed reasonable enough. On that basis, I gave it the top band. Yes, the piece of work was not flawless, but on the basis of what is realistic for 15 year olds to achieve, it deserved the top band. (Of course, this does raise the question of how a teacher is supposed to know what is realistic for 15 year olds to achieve – in this case, I had my mentor, who had experience of marking hundreds of exam scripts, backing up my judgment. In general, this is a reason why it is important for all teachers to have a good idea of the whole of the attainment range, and it is why exam moderation meetings often involve looking at sample scripts from across the range. Tom Sherrington makes a good case for why and how we should do more of this here.) The point is that here, we were giving criteria meaning by reference to norms. Suppose, for the sake of argument, that my norm in this case was not what it was possible for 15 year olds to achieve but what it is possible for Booker Prize winners to achieve. Clearly, I also expect them to vary their sentences in an appropriate and controlled way. That is, the same criterion applies to them. But if Booker Prize winners were the norm, I’d interpret that criterion in a completely different way. I could apply the same criterion to KS2 pupils, where I would again interpret it in a different way.

When judging performance on complex tasks, we don’t just want to see if the pupil can do it or not – we want to make a judgment about how well they can do it. You can judge if a pupil is capable of doing something or not in isolation of the performance of others. You can judge a pupil very clearly on the criteria ‘is able to write a creative story’. Either they can or they can’t, and you can judge that in isolation of other pupils. But we want exams to tell us more than that. For more complex tasks, performance isn’t pass/fail – it’s on a continuum. Most pupils can write a creative story – we want the exam to give us some idea of how well they do that. And that judgment is inevitably bound up with the performance of others. Criteria can help guide that judgment, but they can’t do the whole job on their own. The key point here is not different adverbs, but the comparative and superlative forms of the same adverb – not effectively, convincingly, fluently, but well, better, best.

0 responses to “The adverb problem”

  1. […] this post I spoke about some of the general difficulties with criterion-referenced exams. In this one I looked at how this was particularly applicable to English creative writing assessments. A common […]

  2. […] to encourage accuracy. Daisy Christodoulou has written eloquently about what she calls ‘the adverb problem‘ and this is something of which we should be ever mindful. Our assessment is a lot more […]

  3. […] that this system runs the risk of the ‘adverb problem’ as highlighted by Daisy Christodoulou here. I have wrestled with this conundrum for a while now: what is the best way to effectively judge a […]

  4. […] creating our own version of the ‘adverb problem’, as identified by Daisy Christodoulou in her excellent assault on the inefficacy of levels. We obviously wanted to avoid this lack of clarity at all […]

  5. […] Once this had been established, I started to look at the new specimen assessment materials to see how these AOs were going to be assessed.  Exploring the criteria in Band 4 (year 11) and then Band 3 (year 10) (appropriate for our school context), I placed the assessment criteria / descriptors next to the relevant AOs.  I did the same for A level and then used the APP grids for KS3 focusing on old level 6a criteria for year 9 (again, appropriate for our school context) and worked backwards.  This was an interesting exercise as I was clearly able to see the progression and development of a skill over key stages.  I noted, interestingly, that the descriptors at A level seemed more simplistic in comparison to those at GCSE although we know inherently that they aren’t.  Once this was done, I focused in on the progression of a skill and edited the wording (without messing with the assessment criteria at GCSE) to ensure that the progression of the skill over key stages would be clear for staff to see.  I re-edited this after feedback from Phil Stock to remove fluffy terms such as ‘some’ which are subjective and open to interpretation so I tried to be as specific and focused as I could. More information about this can be found here at Daisy Christodoulou’s blog which Phil sent me over: […]

  6. […] (such as applying a mark scheme to an essay, discussed perceptively by Daisy Christodolou in The Adverb Problem), but able to make comparative judgements of quality reliably and quickly. With this in mind, and […]

  7. […] Christodoulou has written clearly on the problems of adverbs. While there are few adverbs in the new frameworks, we still have statements to judge such as […]

Leave a Reply

Your email address will not be published. Required fields are marked *