Comparative judgment: practical tips for in-school use

 

I have blogged a bit before about comparative judgment and how it could help make marking more efficient and more reliable, and help to free the teaching of writing from tick box approaches. I think CJ has the potential to be used for national assessments – that’s why I’m working with Dr Chris Wheadon and the No More Marking team on a national experiment to moderate Key Stage 2 writing assessments using comparative judgement. However, whatever happens nationally there are ways you can use CJ within your own school. The No More Marking website allows you to use comparative judgment for free. The website is very easy to use and it is definitely worth a go if you are interested.  Here are some suggestions based on what we have done at Ark, plus some practical tips.

Primary writing assessments

There are no interim frameworks outside Y2 & Y6, so using CJ for writing in Y1, 3, 4, & 5 feels as good a solution as any. If you want to try and measure progress across these years, you could get pupils to do a task now and judge it now. Then keep the tasks and use them again in a judging session at Christmas or this time next year and see how they compare to the same pupils’ work at that point in time. We haven’t done this yet but it feels like a really powerful way of showing pupils the progress they are making. I know lots of schools already keep portfolios of pupils’ work across time, so you could use these to start the process.

KS3 English assessments

I’ve personally found it easier to judge writing assessments than to judge literature essays. Others have said the same. We recently got some very high reliability scores when judging a set of allegories that had been written by our Y8 pupils.

KS4 English exams

We haven’t used CJ for KS4 tasks yet, but it would certainly be possible. You could try and judge entire exams, or just pick out individual questions. I feel that individual questions would be easier to judge, and that you would get more accurate results for them. I think you would also get more interesting discussions and feedback afterwards when sharing the results.

KS3 history assessments

CJ has worked just as well for us in history, although again, I found judging history essays to be harder and slightly more time consuming than judging writing tasks. We did some judging on essays on the Battle of Hastings. This is a classic Y7 task and it was interesting for me to see the different ways different teachers had approached it.

And here are some general practical tips

• Go to No More Marking, set up an account for free, and then, on the dashboard, create a new task.
• You will need to upload all the scripts from your pupil. You can upload them as a jpg or pdf. If the pupil work is on paper, you’ll need to scan it – if you have a copier with this facility that shouldn’t be too difficult. The slightly fiddly bit is making sure every separate pdf or jpg has a pupil name or identifier in the title. This means that when you get the results, you will be able to easily see which pupil has which mark. Alternatively, you can use the QR coded answer sheets that nomoremarking.com provides. The bar coded sheets automatically recognise which pupil is which from the bar code on the scan and match them to their results.
• How many judgments do you want per script? If reliability is very important, you will want 10 judgments per script. If it is less important, you can get away with 5. It can feel nice to aim high and try and get a reliability score of 0.85 or more, but there are a couple of things to consider. First, what type of reliability are you getting at the moment without comparative judgment? You probably don’t even know or have a way of finding out. So if you can get a reliability score of 0.75 from doing 5 judgments, that’s more than likely to be an improvement on what you are doing currently. You might be able to get up to 0.9 by doubling the judgments, but you will need to consider whether it is worth doubling the amount of time. It will depend on what you are using the results for. I am starting to think that in some cases, doing 5 judgments per script as a quick sift and then meeting as a team to discuss the results and set standards might be the best way forward.
• Do you want to do the judging together as a group, or send out the links to people? To begin with, I’ve found it quite powerful to have people in the same room doing the judging. Whatever you choose to do, I also think it is worth having a group follow up session where you discuss the scripts and think about why certain scripts were better than others, and what the teaching and learning implications. As I have said before, the two immediate benefits of CJ are that it saves time and it is more reliable. But the longer term benefit is freeing teaching from the tick box. If you don’t meet after to discuss the results and implications, then you are making it harder to achieve that.
• Do you want to include exemplars or not? These can make it easier to apply the standards or grades once you have the results. But I would only use this if you are sure they are at a certain standard. Also, be careful – if you are putting in a script that you feel sure is a C grade, do you think it is a top or bottom C? The C-grade is a large grade so you need to be sure.
• I would recommend trying to get scripts from more than one class to begin with (if possible even more than one school). One of the nice things about the CJ tasks we have done is how they make it quick and easy for teachers to see how other teachers and pupils have attempted similar tasks.