|
Veel kritiek op PISA en topman Andreas Schleicher: statistische blunders, misbruik van PISA door OESO e.d.
1.Er verschijnen steeds meer bijdragen over foute statistische berekeningen in de PISA-studies.
In vorige bijdragen in dit facebook verwezen we al naar een recente studie van de univ. van Helsinki waaruit bleek dat Finland allesbehalve een onderwijsparadijs is inzake qualit...y en equity zoals het voorgesteld wordt in PISA-publicaties. De Finse onderzoekers stelden met eigen tests vast dat de Finse 15-jarigen vrij zwak presteerden voor wiskunde en andere basisvakken - en dit in tegenstelling met de PISA-voorstelling. Ze stelden de voorbije 11 jaar ook een grote achteruitgang vast en slaan alarm. De Finse beleidsmakers werden blijkbaar misleid door de hoge PISA-score en verwaarloosden de niveaubewaking.
2. Hadley schrijft over PISA en Schleicher: It is absurd that someone with no academic status as a statistician can have so much influence on the educational research. The world's greatest experts on statistics have heavily criticised Schleicher's methodology in peer reviewed papers and in articles in the press. Anyone who doubts this should Google any one of Professor Spiegelhalter from Cambridge, Professor Svend Kreiner and Professor Karl Bang Christiensen from Copenhagen, Professor Harvey Goldstein from Bristol and Oxfords Professor Jenny Ozga, look at their academic record and see what they say about PISA. They are not shrills in the pay of the teachers' unions or left wing zealots trying to harm our children's education. They are gifted independent people who simply know what they are talking about with no axe to grind, who can recognise an abuse of statistics when they see it. Mr Schleicher simply does not really understand what he is talking about and it is embarrassing that so many people take so much notice of him.
3.In The Guardian lazen we gisteren in een bijdrage van Peter Wilby volgende kritieken.
*But does Pisa have sufficiently robust data to justify its growing weight? Many critics think not. For one thing, the tests don't work as people think they work. You would expect all pupils to answer the same questions. In fact, according to an analysis by Copenhagen University in Denmark, only 10% of those who took part in Pisa 2006 were tested on all 28 reading questions, and about half weren't tested on reading at all. The OECD feeds real scores into a statistical device called the Rasch model so that it can work out "plausible values" for children who weren't tested. Schleicher says: "We want to test lots of different things but we have limited time. So we give the students different tests with overlapping content." He says it's a long-established statistical technique, enhanced by modern technology. But some statisticians insist it can't work for Pisa, because different test items work differently in different countries. Pisa's league tables, they say, are almost meaningless. In 2006, the UK could have finished anywhere between 14th and 30th on reading, Canada anywhere between second and 25th, Japan anywhere between eighth and 40th.
*A second objection to Pisa is that it can't take account of social, economic and cultural differences. Can it make sense to compare, say, Peru, which has high levels of child labour and limited internet access, with western European countries? Can UK schools really learn anything from east Asian countries, with their deeply ingrained respect for authority?
*Pisa critic: Schleicher's work threatens a global standardisation of education, wiping out school systems that were embedded in diverse local cultures, values and traditions. "The very meaning of public education is being recast," write the American academics Heinz-Dieter Meyer and Aaron Benavot, editors of Pisa, Power and Policy, a collection of learned papers published this year, "from a project aimed at forming national citizens and nurturing social solidarity to a project driven by economic demands." Schools, they argue, are increasingly "subject to the imperatives of efficiency, calculability, predictability and control".
Until recently, almost nobody questioned the merits of Pisa and its league tables, only their interpretation. Now, as the world's schoolmaster announces the latest planetary exam results, we can expect the political arguments to intensify.
4. We citeren ook nog even uit davidts' blog
The PISA methodology is complex and rather opaque, in spite of the substantial amount of material published in the technical reports. Briefly:
Individual students only answer a minority of questions. Multiple plausible values are then generated for all students assuming a particular statistical model, essentially estimating what might have happened if the student had answered all the questions. These plausible values are then treated as if they are the results of complete surveys, and form the basis of national scores (and their uncertainties) and hence rankings in league tables. But the statistical model used to generate the plausible scores is demonstrably inadequate it does not fit the observed data. This means the variability in the plausible scores is underestimated, which in turn means the uncertainty in the national scores is underestimated, and hence the rankings are even less reliable than claimed. Here's a little more detail on these steps.
(1). Individual students only answer a minority of questions.
Svend Kreiner has calculated that in 2006, about half did not answer any reading questions at all, while "another 40 per cent of participating students were tested on just 14 of the 28 reading questions used in the assessment. So only approximately 10 per cent of the students who took part in Pisa were tested on all 28 reading questions."
(2). Multiple plausible values are then generated for all students assuming a particular statistical model
A simple Rasch model (PISA Technical Report , Chapter 9) is assumed, and five values for each student are generated at random from the 'posterior' distribution given the information available on that student. So for the half of students in 2006 who did not answer any reading questions, five 'plausible' reading scores are generated on the basis of their responses on other subjects.
(3). These plausible values are then treated as if they are the results of surveys with complete data on all students
The Technical Report is not clear about how the final country scores are derived, but the Data Analysis manual makes clear that these are based on the five plausible values generated for each student: they then use standard methods to inflate the sampling error to allow for the use of 'imputed' data.
Secondly, PISA uses imputation methods, denoted plausible values, for reporting student performance. From a theoretical point of view, any analysis that involves student performance estimates should be analysed five times and results should be aggregated to obtain: (i) the final estimate; and (ii) the imputation error that will be combined with the sampling error in order to reflect the test unreliability on the standard error.
All results published in the OECD initial and thematic reports have been computed accordingly to these methodologies, which means that the reporting of a country mean estimate and its respective standard error requires the computation of 405 means as described in detail in the next sections. There does seem to be some confusion in the PISA team about this - in my interview with Andreas Schleicher, I explicitly asked whether the country scores were based on the 'plausible values', and he appeared to deny that this was the case.
(4). The statistical model used to generate the plausible scores is demonstrably inadequate.
Analysis using imputed ('plausible') data is not inherently unsound, provided (as PISA do) the extra sampling error is taken into account. But the vital issue is that the adjustment for imputation is only valid if the model used to generate the plausible values can be considered 'true', in the sense that the generated values are reasonably 'plausible' assessments of what that student would have scored had they answered the questions.
A simple Rasch model is assumed by PISA, in which questions are assumed to have a common level of difficulty across all countries - questions with clear differences are weeded out as dodgy. But in a paper in Psychometrika, Kreiner has shown the existence of substantial Differential Item Functioning (DIF) - i.e. questions have different difficulty in different countries, and concludes that the The evidence against the Rasch model is overwhelming.
The existence of DIF is acknowledged by Adams (who heads the OECD analysis team), who says The sample sizes in PISA are such that the fit of any scaling model, particularly a simple model like the Rasch model, will be rejected. PISA has taken the view that it is unreasonable to adopt a slavish devotion to tests of statistical significance concerning fit to a scaling model.. Kreiner disagrees, and argues that the effects are both statistically significant and practically important.
(5) This means the variability in the plausible scores is underestimated
The crucial issue, in my view, is that since these 'plausible values' are generated from an over-simplified model, they will not represent plausible values as if the student really had answered all the questions. Kreiner says The effect of using plausible values generated by a flawed model is unknown.
I would be more confident than this, and would expect that the 'plausible values' will be under-dispersed, ie not show a reasonable variability. Hence the uncertainty about all the derived statistics, such as mean country scores, will be under-estimated, although the extent of this under-estimation is unknown. It is notable that PISA acknowledge the uncertainty about their rankings (although this is not very prominent in their main communications), but the extra variability due to the use of potentally-inappropriate plausible values will inevitably mean that the rankings would be even less reliable than claimed. That is the reason for my scepticism about PISA's detailed rankings.
|
