Closed questions and higher order thinking

I know that Andrew Old often writes about the way that open questions are often, wrongly, seen as superior to closed questions – ie, it’s seen as being better to ask pupils questions that have lengthy answers and many possible answers rather than those that only have one straightforward right answer. I think one of the reasons for this is that open questions are seen as being effective promoters of higher-order thinking. That is, asking a question like ‘Why was there a war in Europe in 1914?’ is likely to lead to higher quality thinking than ‘When did Britain and France sign the Entente Cordiale?’

I would make two points here. First, it is true that closed questions are very effective at testing facts and knowledge. However, ‘higher order thinking’ such as analysing, evaluating and synthesising is in fact made up of ‘lower order’ facts and knowledge. To know why the First World War happened you have to have a very good knowledge of chronology and knowing when the Entente Cordiale was signed is an important, if small, part of that wider question. Indeed, the question ‘Why was there a war in Europe in 1914?’ requires a lot of facts to be able to answer successfully, and  one of the things that allows you to be able to analyse a deep, open question like this successfully is how many facts you have and how well you know them. This is as true of the academic level as the classroom level. I am currently reading Christopher Clark’s book on just this question. In the introduction, he notes that one of the problems with this topic is just how many facts have to be understood and marshalled, and that they are all in so many languages.  One of the things which makes Clark’s book so brilliant is that he has spent several decades mastering very many of these facts.  There is no way that a 20-year-old could write a book like this, however innately smart and innately good at analysing they were. Clark’s superb and revisionist analysis of the causes of the First World War springs from his detailed mastery of the facts. His excellent analysis is a function of his wide and deep knowledge.

Second, closed questions can in fact be extremely good at testing analysis and evaluation – in fact, they can often be more efficient and effective than open questions. For proof of this, take a look at the British Columbia leaving exam. (The link is to an absolutely excellent report by Common Core which features extracts from curriculums and exams from around the world.)

The BC leaving exam in history asks pupils to write essays, yes. But it also asks them to answer multiple choice questions. Here’s a typical example.

15. How did the Soviet totalitarian system under Stalin differ from that of Hitler and Mussolini?
A. It built up armed forces.
B. It took away human rights.
C. It made trade unions illegal.
D. It abolished private land ownership.

I think this is an excellent question which very definitely asks for higher order thinking. The reason why I think this question is so good is that it tests a finer gradation of understanding. Everyone knows the Nazis and Soviets were evil, and because they were evil, it is easy for pupils to just think that their regimes were the same. And of course the regimes were very similar. But they were different in interesting ways too, and this question probes that. A pupil who got this question right would have understood something important. A pupil who didn’t would have misunderstood something quite important.

And it isn’t just this question. I went through all of the literature and history multiple choice questions on this paper and all of them made me think hard about deep issues. I couldn’t have rote learned or memorised the answers to any of them. So I would say that the closed questions can be very effective at promoting higher order thinking. Of course, I wouldn’t suggest only having these types of questions – but as I said, the BC exam has a judicious mix of these and essay questions.

I know another technical criticism of MC questions is that it is too easy to guess them, but I think this criticism can be simply countered by having a penalty for wrong answers and/or a very high pass rate.

Update: I’ve since written two more posts on this topic, here and here.

15 responses to “Closed questions and higher order thinking”

  1. You’ve nailed it again! GO DAISY, GO!

  2. Barry Naylor says:

    Daisy

    Fascinating topic, making me think. Having taught AP in US system I do believe it is possible to write valid and reliable test items testing higher order skills.

    “I think this is an excellent question which very definitely asks for higher order thinking.”

    I will have a guess having never studied either political system but having seen and read quite a bit about WW2, Cold War etc. My guess would be D but no great confidence.

    What makes you confident that answering the question quoted requires higher order thinking. Any chance you could explain briefly.

    Ta

  3. Q. What is a Soviet?

    A. A council
    B. Evil
    C. An example of lower level thinking

  4. Barry Naylor says:

    I take it that’s a no.

  5. Barry Naylor says:

    Didn’t mean that to seem rude. Sorry.

    I appreciate that you are very busy and time management etc. may dictate that you do not provide an individual reply.

    Have a great weekend.

  6. […] my previous blog post I gave an example of what I thought was an excellent multiple choice question, taken from the […]

  7. […] my last posts on multiple choice questions (here and here), Kris Boulton and Joe Kirby have pointed me in the direction of Robert Bjork’s work on […]

  8. […] of this model is the lack of challenge and access to higher order thinking skills, but as noted here, multiple choice questions can be extremely challenging if constructed carefully. Besides, as shown […]

  9. […] In this blog I’m going to focus on DT Willingham for no other reason than it also links to previous blogs. On twitter, it seems, the DT Willingham meme is everywhere. The meme posits the view that higher order thinking skills equate to knowledge. There seems to be little room for thinking skills such as: interpretation, empathy, analysis, synthesis, evaluation or some humanist or esoteric notion of cleverness. Assessing how well we think, therefore,  is correlated to assessing how much knowledge we have. This seems to be the reasoning behind Daisy Christodoulou’s understanding of multiple choice q… […]

  10. Interesting. Re: The risk of ‘guessing’. You mention penalising wrong answers. Another interesting idea is linking with ‘confidence questioning’ (students can gain extra marks if they feel ‘confident’, but lose more marks if they are wrong on those Qs).

  11. […] pick B’ used to be my approach. However, a series of blogs by Joe Kirby, Phil Stock and Daisy Christodoulou have caused a change of heart in my […]

Leave a Reply

Your email address will not be published. Required fields are marked *