Over domme PISA-analyses van PISA- kopstuk Andreas Schleicher: deel 1

Reclaiming Schools
More PISA myths about top-performing school systems

Posted on February 20, 2015 by terrywrigley

A recent article circulated by the head of PISA Andreas Schleicher claimed to dispel “7 big myths about high-performing school systems“. These include “the myth that disadvantaged pupils are doomed to do badly in school”. Expressing the issue like this already distorts the argument: not doomed perhaps, but highly likely! On top of this, the choice of Shanghai to prove his point is a tendentious use of evidence.

Here, and in two later instalments, Pat Thomson and Terry Wrigley take issue with Schleicher’s logic and show the importance of questioning conclusions derived from big numbers.

More PISA myths about top-performing school systems (part 1)

Analysis by Pat Thomson and Terry Wrigley

Andreas Schleicher, OECD’s director of education and skills (the man in charge of PISA), recently sent a challenging article to BBC News website – surprisingly published under Business rather than Education. Under the title ‘Seven big myths about top-performing school systems’, his press release creates more myths than it dispels.

Schleicher’s statement seems to have been written for the USA, but by an editorial sleight of hand, it is related to England by adding an introductory sentence: ‘Education Secretary Nicky Morgan says she wants England to get into the top five of the international PISA tests for English and Maths by 2020.’

Statistical manipulations

Schleicher uses data from PISA to compare countries with each other. While large-scale statistical data can be revealing there is also a danger of superficial and misleading analysis.

Problems often occur when people use educational statistics. One of the most common errors is when a correlation is taken as a cause. For example, we might say that poverty directly causes poor school attainment, instead of noting that a very complex set of relationships and actions produces the correlation between these two sets of data, income and school attainment. Understanding the processes by which poverty converts into educational disadvantage is essential; these involve a tangle of cultural, political and historical factors which may not be the same in different societies.

Another common problem with statistics occurs when one single factor is correlated with another for the purposes of comparison –to take our previous example, poverty is correlated with attainment. If all countries are compared with each other across only these two measures, this ignores important differences which influence the relationship between them. For example, relative poverty may have a different type of impact on young people’s attitude to education in different societies; and class size matters more in some circumstances than others. We should be wary of simplistic conclusions. Simple comparisons are often, as the saying goes, like comparing apples with oranges.

Schleicher’s article makes both of these fundamental errors simultaneously.

On top of this problem, Schleicher’s press release engages in various rhetorical devices to weight the argument, including setting up a ‘straw man’ to refute.

Here are the myths Schleicher seeks to topple, and our responses to his argument.

Myth no 1 Disadvantaged pupils are doomed to do badly in school

When a teacher holds this fatalistic position, it does indeed reinforce disadvantage. Nevertheless it is statistically irrefutable that disadvantaged pupils in general are far more likely to underachieve. In fact, PISA shows not a single country where that isn’t the case, though they vary in (a) the amount of poverty (b) the degree to which this impacts on achievement.

Schleicher’s main evidence here is that the 10% most disadvantaged 15-year-olds in Shanghai have better maths skills than the 10% most privileged students in the United States. Undoubtedly there are some good things happening with maths teaching in Shanghai, but Schleicher is hiding some crucial factors.

Firstly, the social context of Shanghai. Over many years, this city-state has seen itself as an elite intellectual powerhouse. It has granted citizenship only to highly qualified outsiders. Currently 84% go into Higher Education and its people place very high value on children’s education. (See the OECD’s more detailed analysis for other important factors.)

Secondly, what kind of ‘disadvantage’ is being referred to here? Most manual and routine work in Shanghai is carried out by non-citizens – migrants from other parts of China. These now amount to 54% of children starting primary school. Compared with the rest of China, Shanghai regards itself as generous for providing these migrant workers’ children with an education, but it is only up to Year 9: with few exceptions, in order to pursue their education from age 15 they have to return to their regions of family origin. Data from Tom Loveless of the Brookings Institute shows 6 out of 10 leaving the city by age 15; those who remain go into low-skill employment or attend vocational schools, though some exceptions are made for exceptionally capable students. This means that by the time children sit the PISA test, most non-citizens’ children are no longer at school. Consequently ‘the 10% most disadvantaged’ from among the PISA sample are nowhere near the ‘most disadvantaged’ members of the society at large.

Finally, school learning is heavily supplemented by homework and private tuition. One estimate is that tuition for high school students costs on average around 30,000 yuan annually, plus 19,200 for other activities such as tennis or piano. This is higher than the average Chinese worker earns in a year (42,000 yuan).

To use Shanghai’s divided society as an example of successfully countering poverty-related underachievement is highly dubious.