Seven Myths: The evidence base, part I

In Seven Myths about Education, I make two claims: first, that in English education, a certain set of ideas about education are predominant; second, that these ideas are misguided. Finding the evidence to prove the second point was relatively straightforward. It is scientifically well-established that working memory is limited and that long-term memory plays a significant role in the human intellect. This has clear implications for classroom practice, implications which others have made and which I was happy to recap.

However, there was no one evidence base which could prove (or indeed, disprove) the first claim. Instead, I identified a range of different evidence bases to prove my point that in English education, the seven ideas I discuss are predominant. (In doing so, I also defined clearly what these ideas meant in theory and practice.) Here’s a recap.

• The writings of prominent theorists, and the proof that such theorists are indeed prominent (eg book sales, presence on government committees, the judgment of their peers, etc.)
• The advice given in popular teacher training textbooks, and the proof that such textbooks are popular (eg book sales, presence on reading lists)
• The National Curriculum
• Ofsted reports – I created an appendix of 228 exemplar lessons that were described in Ofsted subject reports from the last three years. These exemplar lessons were in turn drawn from the thousands of lesson observations done by Ofsted inspectors over the previous 3-5 years. See here for more information about this and a link to the appendix, which is available for free online.
• Popular non-governmental curriculums (and the evidence of such popularity, eg by number of schools applying it)
• Examples of lessons from popular resource sharing websites

I could have included a lot, lot more examples from the latter category which would have proved my point, and then some. I deliberately chose not to because it is hard to tell how popular or influential such lessons are. Had I used a lot of these, it would have been easier to accuse me of simply cherry picking the worst examples I could find on the entire internet. So I went easy on these, and only used a few examples from such websites, and then only from websites I could demonstrate were popular.

Given this range of evidence, I found it odd that some people criticised the book as being reliant on anecdotal evidence. As I say in the introduction, I do add in the odd anecdote to try and liven up the text, but only when I have clearly established that my anecdotal experience is in line with the evidence. In his review of my book, Tom Sherrington refers to my book being based on ‘personal anecdotal experience…from a very specific teaching situation’. That is absolutely not the case. It’s also particularly baffling that Tom Sherrington would say this given that the only counter-evidence he brings to bear is his own anecdotal experience. His strongest counter-argument is that ‘for me, the myths just don’t ring true’.

Interestingly, when I first started writing the book, my instinct was that my own personal anecdotal experience would bear little relation to the wider system, simply because I couldn’t believe that an entire system would have endorsed beliefs that were so completely at odds with all the available evidence. I actually set out planning to write a book that critiqued some parts and aspects of the education system. It was a genuine surprise to me to see that clear examples of bad practice were being endorsed as good practice on a system-wide level.

In a largely positive review, Michael Fordham did criticise my use of Ofsted reports as ‘a reliable and unproblematic account of pedagogy’. I think he is right to say that Ofsted reports are problematic. In the book I do in fact discuss the strengths and weaknesses of Ofsted reports, conceding that,

The one flaw with these reports is that Ofsted inspections are pre-announced. This means that teachers can, and do, put on a show for Ofsted. It has long been a complaint of many teachers that the kind of lesson Ofsted grade as outstanding is simply not possible to repeat consistently. So, when I give an example of an outstanding lesson from an Ofsted report, I do not mean to suggest that lessons exactly like this are going on all the time in every lesson. However, given the power we have seen Ofsted have, given the cottage industry of Ofsted preparation and given the fact that most schools will organise their own internal observation systems around Ofsted criteria, it is still fair to say that if Ofsted demand a certain type of lesson, that matters.

Also, I am not just using Ofsted reports as evidence for how teachers teach. I am also using them as evidence for how teachers are told to teach.

So, Ofsted’s inspection reports and subject reports are a fairly reliable guide to what actually happens in schools and a very reliable guide to what teachers are told to do.

Given that I am trying to prove what the dominant views are in the English education system, finding out how teachers are told to teach is just as important as finding out how they actually do teach.

Finding completely reliable and valid evidence of how teachers in England teach is always going to be difficult. Even a well-funded research study would run into methodological problems. Any large-scale observation programme, however low-stakes, would face the Hawthorne effect. However, whilst finding completely reliable and valid evidence is hard, I still think the Ofsted reports, backed up by the other sources listed above, offer a fairly reliable and valid picture.

Also, finding reliable and valid evidence of how teachers in England are told to teach is nothing like as methodologically difficult. Essentially, it is as straightforward as looking at what Ofsted and the government tell teachers. I would argue that even if teachers completely and utterly ignored this advice, the advice would still tell us something important about the education establishment. And in any case, there is good evidence that teachers and schools do not and can not ignore Ofsted. Since I first published the book, I think events have reconfirmed the central importance of Ofsted and their judgments in the education system.

I am always interested in discovering new sources of evidence about classroom practice. If people think there are other important sources I’ve missed, I would really like to know. In Tom Sherrington’s case, he seems to be suggesting that his own personal experience is a more valid and reliable source of evidence than the list I’ve outlined above. In Michael Fordham’s case, he doesn’t put forward any alternative. If anyone wants to suggest any others in the comments thread, I’d be glad. (Two very interesting sources which in the end I didn’t use were the PISA and TIMSS teacher, school and student surveys. These are really fascinating and have a lot of interesting data in them. Liz Truss and Laura McInerney, amongst others, have used these surveys to make interesting points. But they didn’t have the level of detail about classroom practice that I was looking for. TIMSS have done video surveys of some classrooms, but unfortunately not, so far as I know, of English classrooms.)

In any case, as I will go on to discuss in my next post, a lot of the arguments about the evidence base for my first claim are actually just a rather illogical smokescreen for people who are really interested in challenging my second claim.