The Minister for Education Alan Tudge has claimed the UK as the new benchmark for education performance while ignoring serious flaws in the reporting of its results. He has also misrepresented data on school funding and student results in Australia.

Mr Tudge told The Age education summit in April that the UK has dramatically improved its education results in the Organisation for Economic Co-operation and Development’s (OECD) Programme for International Student Assessment (PISA) over the past 10 years despite cutting school funding.

However, the UK PISA 2018 results are significantly over-stated because of several flaws, most notably very low school participation and high student exclusions from the tests.

The flaws are exposed by Professor John Jerrim of the University College of London in a paper to be published in the academic journal Review of Education.

His analysis comprehensively refutes Mr Tudge’s claim. Professor Jerrim says: “There is clear evidence of an upward bias in the PISA 2018 data for England and Wales, with lower-achievers systematically excluded from the sample”.

Professor Jerrim estimates that the combination of student exclusions, school non-response, student non-response and technical details about eligibility criteria meant that about 40 per cent of the target UK student population did not participate in PISA 2018.

This was the fourth lowest participation rate of the 79 countries participating in PISA 2018. Only Panama, USA and Brazil had lower rates.

He shows that the PISA 2018 data for England (which accounts for 84 per cent of the UK sample) clearly under-represents lower achieving students and over- represents higher achieving students.

For example, 21 per cent of the PISA sample were low achievers compared to 29 per cent for the total population of the age group.

Another issue analysed by Professor Jerrim is the school response rate. The OECD requires that 85 per cent of sampled schools agree to take part in the study.

However, both England and Northern Ireland failed to meet this standard with only 72 per cent and 66 per cent respectively participating.

The overall rate for the UK was 73 per cent. While there is provision to include replacement schools, the PISA technical criteria require a very high participation rate from such schools which was not met by the UK.

Even the UK Department of Education admits that the UK “did not fully meet the PISA 2018 participation requirement” [Achievement of 15-year-olds in England: PISA 2018 results, p. 188] because of the high school non-response.

However, the OECD waved this through and agreed that the UK data should be included as fully comparable to other countries.

Professor Jerrim says the OECD has a very weak adjudication process to decide whether to include a country’s data in the PISA results.

Professor Jerrim also shows that the UK had a high rate of student exclusion from the tests. Students can be excluded from the tests in various ways.

Schools may decide not to test some students included in the sample, others may be declared ineligible because they moved school between the time the sample was designed and the time the test was implemented, parents may not consent for students to participate and some students in the sample may be absent on test day.

The OECD technical standards state that within-school exclusions should total less than 2.5 per cent of the target population and that the combination of school-level and within-school exclusions should not exceed five per cent of the target population.

The UK failed to meet these standards. The within-school exclusion rate was 3.3 per cent and the total exclusion rate was 5.5 per cent.

Professor Jerrim notes that a strict application of PISA’s data quality criteria would have led the UK to be removed from the study, as it was from PISA 2003, for similar breaches.

These exclusion rates are much higher than for many other countries participating in PISA 2018. The average within-school rate was 1.4 per cent and the total exclusion rate was three per cent.

The total exclusion rates in Japan and South Korea were 0.1 per cent. Such differences are likely to bias cross-country comparisons of PISA performance.

The overall high non-participation in the UK has clear potential to bias its PISA results. It creates large uncertainty about the results. This is likely to affect the reliability of comparisons to other countries and how results have changed over time.

Professor Jerrim estimates that the average PISA scores in England and Wales were inflated by the high non-participation rate in PISA 2018.

The average PISA mathematics score for England was 504 points – significantly above the average across OECD countries.

He estimates that had a truly representative sample of the population taken the tests England’s score would have been about 494. This is roughly the same as the OECD average and Australia’s score.

Professor Jerrim concludes that the OECD adjudication process for deciding whether a county’s results should be accepted in a PISA cycle should be more transparent.

“ ... the OECD needs to reconsider its technical standards, the strictness of which these are applied, and its data adjudication processes...the processes currently in place flatter to deceive and are nowhere near robust enough to support the OECD’s claims that PISA provides truly representative and cross-nationally comparable data,” Professor Jerrim noted.

His analysis of the dodgy PISA results in the UK raises the broader issue of the validity of international comparisons when there are so many loopholes for countries to rig their results.

He states: “There remain many ways for countries to not test pupils who are technically part of the target population, with lower-achievers disproportionately likely to be removed from the sample.”

Apart from using the dodgy UK PISA results, Mr Tudge also fudged school funding data in Australia in claiming its school funding per student, adjusted for inflation, increased by 60 per cent since 2000.

This is far from the truth. After adjusting for flaws in data from the Report on Government Services, we estimate the actual increase for all schools from 2001-02 to 2018-19 was only 19 per cent, that is, an average increase of just over one per cent per year.

The increase for private schools was over double that for public schools. Government funding per student in private schools increased by 34 per cent compared to only 15 per cent for public schools.

If school funding is failing to deliver better results as Minister Tudge claims, this is mainly because money is being wasted on more privileged private schools instead of helping schools to overcome disadvantage in education.

Mr Tudge also fudged Australia’s school results by highlighting the decline in PISA results for 15-year-old students and ignoring improving results in Year 12.

The decline in PISA results is questionable because student motivation and effort is likely to be a factor in the decline.

In contrast to Year 12 assessment, the PISA tests have no consequences for students as they don’t even get their results.

The OECD says that 73 per cent of Australian students participating in PISA 2018 did not fully try in the tests.

While there is no direct evidence of an increasing proportion of students not fully trying in the PISA tests over time, there is indirect evidence.

PISA data show that student dissatisfaction at school amongst 15-year-olds in Australia increased four-fold from eight per cent to 32 per cent between PISA 2003 and 2018.

This large increase in student dissatisfaction may have led to lower motivation and effort in PISA over time.

The OECD says that the relationship between a feeling of belonging at school and performance in PISA is strong for students with the least sense of belonging [OECD 2016, p. 122].

Students who feel they do not belong at school have significantly lower levels of achievement in PISA than those who do feel they belong.

Mr Tudge’s claim of declining school results is contradicted by other more significant data. The percentage of the estimated Year 12 population that completed Year

12 increased from 68 per cent in 2001 to 79 per cent in 2018, although there is an unexplained drop-off in 2019 [Report on Government Services 2007 & 2021].

The proportion of 20 to 24-year-olds who attained a Year 12 Certificate or equivalent increased from 79 per cent in 2001 to 89 per cent in 2019 [ABS, Education and Work, 2011 & 2020].

OECD data also shows that Australia had one of the larger increases in the OECD in the proportion of 25-34 year-olds who attained at least an upper secondary education.

It increased by 19 percentage points from 71 per cent in 2001 to 90 per cent in 2019 [Education at a Glance 2002 & 2020].

These are indicators of an improving education system, not a deteriorating one. They are clearly inconvenient for Mr Tudge because he ignores them and relies solely on questionable figures that misrepresent Australia’s education performance.

Minister Tudge’s fudges are designed to deny public schools the funding increases needed to ensure all students received an adequate education and to improve equity in education.

Instead, the Morrison Government has provided billions of dollars in special deals for private schools and conspired with state governments though bilateral funding agreements to continue to under- fund public schools.

No more fudges, Mr Tudge. Your fundamental task as the Commonwealth Minister for Education is to better support public schools and disadvantaged students to deliver improvements in equity in education.