Five questions to ask an education guru

Posted: September 26, 2015 | Author: gregashman Blog: Filling the pailBlog:

Imagine that you are a teacher in a school and you have just sat through a presentation by an education guru of some sort. Let’s call her ‘Marion’. If it is safe to do so, what questions should you ask Marion at the end of the presentation? I have a few suggestions.

1. What are you actually suggesting that we should do?

A typical education guru will say plenty of things that don’t actually lead to concrete proposals. She may spin anecdotes about a fantastic school that she recently visited where everything was marvelous.

Alternatively, she may construct terms and then set about defining these terms. You may find yourself hearing about ‘Productive Change Capacity’, how this is made up of constructive professional dialogue, openness to the views of others etc. and how this can be contrasted with ‘Resistive Change Capacity’ which is made of restricted professional dialogue, closed mindedness and so on. I’ve just made-up these two terms but education literature is full of such tautologies that lead us nowhere in particular.

Instead, if Marion really has anything to offer then she should be able to describe what might change as a result of taking on her ideas. Of course, you don’t expect Marion to know everything about your context and so she might need to ask a few questions too. However, she clearly should have something to offer or there really is no point.

2. What problems do your proposals solve?

Presumably, implementing Marion’s ideas will make things better in some way. Otherwise, why would we bother? So it seems reasonable to ask what current or typical problems these ideas will solve. This also places the ideas on a testable basis. If they solve a particular problem then what will that look like? How will we know that the problem is solved? Perhaps our students will read more at home or they might become better mathematical problem solvers. Perhaps they will feel better about school. Some of these things are easier to measure than others but any meaningful change should have observable consequences.

3. What would convince you that you are wrong?

Testability leads to a key principle of science that is largely absent from education discussions; the idea of falsifiability. In fact, it is so absent that you are likely to need to persist in order to get an answer to this question. I would predict that Marion’s first response would be to explain why her ideas are not wrong, why they are well grounded in theory or research or whatever. So you’ll probably need to clarify with a follow-up question.

The problem is that many supposed educational theories can explain all possible sets of circumstances. We will see an example shortly. If Marion really cannot think of anything that would convince her that her ideas are wrong then we have something more akin to an unshakable belief than something based on evidence.

Strong theories are always falsifiable. Their strength comes from the fact that, despite this, nobody has managed to demonstrate that they are wrong. There is a common story that tells of how the evolutionary theorist, J. B. S. Haldane, was asked what would falsify the theory of evolution and he answered that finding fossilised rabbits in Precambrian rocks would do it.

4. Does adopting part of the approach give part of the benefit?

One way that a proposed intervention can become unfalsifiable is when it only works if implemented fully and with 100% fidelity. If it works, great. If it doesn’t work then that is because you didn’t do it properly. Either way, the rightness of the original intervention remains unchallenged.

Something like this happened with a huge differentiated instruction study in the U.S. It didn’t work but the authors concluded that this was because the intervention was not implemented correctly, leaving the principle of differentiated instruction unchallenged.

Now it may be the case that some approaches will work with a perfect implementation and will not work or will cause harm if implemented with anything less than this. If this is true then ask yourself how much practical value there would be in adopting this course of action. It seems unlikely in the extreme that you will get any team of teachers anywhere to implement something with complete faithfulness to the originators’ intentions.

However, if implementing part of the program delivers part of the benefits then this seems a much better prospect. You can imagine different teachers having strengths in different elements, at least to begin with. And as the benefits accrue then you might start to win over the skeptics.

5. What are the negative effects?

If a consultant cannot describe the negative effects of their proposed initiative then this is either because there aren’t any, the consultant is badly informed, the consultant is dishonest or it has never been attempted before and you are the guinea-pigs.

I cannot think of any intervention, even those that I would recommend, that have no negative consequences. For instance, a push for explicit instruction would meet with some teacher resistance that would have to be effectively managed.

No questions asked

Of course, you will not need to ask any of these questions if they are answered in the presentation. Some of the best educationalists that I have seen will preempt most of these points. Dylan Wiliam, for instance, has spent a lot of time thinking about the negative impacts of attempts to embed more formative assessment and has tried to develop programs that provide benefits if implemented only in part. I have heard him talk of teachers aiming to embed one new practice per year.

And we shouldn’t be too harsh. Just because a consultant tells an anecdote, it does not mean that she is wrong. Anecdotes enliven presentations and often make them more bearable. A consultant promoting a genuinely effective approach might not have been challenged by such questions before; the education community is often too polite and credulous.

However, if after a thoughtful pause, you don’t get a straight answer then I’d be keen to investigate further before plunging into the next whole-school initiative.

“We’ll need to have a think about that, Marion.”

If you are interested in reading more about the evaluation of educational initiatives and ideas in a way that is accessible to teachers then I recommend Dan Willingham’s book, “When can you trust the experts?”