Using Star data to measure the impact of lockdown on learning inequalities – a UK first

By Renaissance UK,

It’s not news to anyone that educational outcomes for many young people have suffered during lockdown and the effective total closure of schools for many pupils since Easter. It’s commonly held that this also not likely to be felt equally – and that given the unequal access to non-school learning during this period, educational inequality will rise significantly.

Most notably, the Education Endowment Foundation has suggested that the past ten years’ worth of effort by teachers, schools, and educators across the country to close the socio-economic attainment gap between better off children and their more disadvantaged peers may be reversed during these three months of closure.

But how can such a theory be verified?

Over the last few months, various pieces of information have become available showing the discrepancy in inputs into home learning – that is, access to IT equipment, number of hours of learning undertaken by different pupils, access to live lessons, resources available for pupils, or extent of parental supervision. But what hasn’t been available until now is any output data, which could be analysed to explore for any discrepancies. Given the replacement of external standardised GCSEs and A Levels and their replacement with centre-assessed grades, and the postponement of this year’s Year 2 SATs exams, there will be no national standardised data to use.

Professor Anna Vignoles and Professor Simon Burgess – two of the country’s most renowned education economists – have been commissioned by the Royal Society as part of the DELVE project. DELVE – Data Evaluation and Learning for Viral Epidemics – is a multi-disciplinary group, convened by the Royal Society, to support a data-driven approach to learning from the different approaches that countries are taking to managing the pandemic. This group formally provides input into the UK government’s response to Covid-19 through SAGE. Professors Burgess and Vignoles were asked to explore this issue further and to understand the size of the problem we have to address in terms of catching-up that loss. They put out a call for any schools who had undertaken internal tests during lockdown, and who had comparable data for pre-lockdown.

Our contribution

Renaissance were delighted to be able to offer anonymised access to Star Assessment data from across our suite of schools to help with this project.

We were able to use our unrivalled depth of data gathered from all of our participating schools during the 2019-2020 academic year, and our data on the pupils and schools that have taken the tests, including their FSM data, to provide a uniquely rich dataset which tracks pupil performance both before and after lockdown, and show – for the first time, empirically – what has been happening in the UK.

The findings from this analysis have been published today, by the Royal Society – and can be seen here.

As Professors Burgess and Vignoles say in the report:

As far as we know, this is the first time that outcome data for the period of the Covid-19 closure has been analysed to show the impact of the lockdown in the UK. This data is from an online learning platform, where young people between the ages of 5 and 16 from subscribing schools can take Star tests on reading and understanding. We are grateful to Renaissance Learning for providing anonymised pupil and school test data for this report, including a pupil ID reference that is used for internal Renaissance purposes only, a date of test, minutes spent per test, and the test scores. In all, around 160,000 data points were provided for the period September 2019 through mid June 2020. There is no generally set time frame for taking the tests so a week-by-week or month-by-month analysis might be misleading: for example, it might be that the more able students tend to take tests later in the year. But clearly, one very salient factor for test taking will be the school lockdown and the strong encouragement from schools to use online resources. So we simply look at the distribution of test scores before and after the lockdown, for a fixed set of pupils who take tests 1 or 2 times pre lockdown, and also take tests 1 or 2 times post lockdown. This is designed to reduce the scope for selection issues to drive the results.

The results are show below:

Burgess and Vignoles show:

The graph plots the interquartile range for this group shows that the difference between the scores of high-performers and low-performers increases markedly after the schools lockdown. This is particularly clear for primary school children, less so for secondary school children. To scale the IQR, the pre-lockdown average scores were respectively 378 (for Year 3), 541 (Year 5), 734 (Year 7) and 870 (Year 9).

In other words, the interquartile range – that is the difference in raw scores between the score of the pupil at the 25th percentile, and the score of the pupil at the 75th percentile – has increased significantly for all 4 year groups studies pre and post lockdown.

  • For Year 3 pupils, the gap has grown from 190 points to 290 points (around a median score of 378 points) – a rise of 52%
  • For Year 5 pupils, the gap has grown from 230 points to 320 points (around a median score of 378) – a rise of 39%
  • For Year 7 pupils, the gap has grown from 350 points to 390 points (around a median of 541), a rise of 13%
  • For Year 9 pupils, the gap has grown from 440 points to 500 points (around a median of 870 points), a rise of 13%

We therefore see here, for the first time, the widening of the gap – with the biggest gaps in primary phase. The gap between the lower performers (at the 25th percentile) and the higher performers (at the 75th percentile) has grown.

Burgess and Vignoles conclude:

This picture fits well with the increased inequality seen in learning inputs over the lockdown period, and is concerning for the future learning outcomes for these cohorts and their life chances beyond education.

John Moore, Director at Renaissance UK, was pleased to be able to support the study. He notes: “This is a brilliant example of what Renaissance can offer as a public good. We have an immense amount of data and a wide breadth of schools across England. In keeping with our mission to Accelerate Learning for All we were happy to contribute our analytical power to dig into this and make a real contribution to public policy”

This graph – and the accompanying analysis in the report – provides, for the first time, concrete data in the UK to both demonstrate, and begin to quantify, unequal learning loss. It shows the scale of the challenge which schools in the UK will face when they return for all pupils in September. At Renaissance, we will continue to offer support through a suite of products and services to do what we can to help once again address and begin to narrow this gap.

Renaissance UK

Renaissance is a leading provider of learning analytics - enabling teachers, curriculum creators and educators to drive phenomenal student growth. Renaissance's solutions help educators analyse, customise and plan personalised learning paths for students of all ages and abilities, allowing time for what matters - creating energizing learning experiences in the classroom. Founded by parents, upheld by educators and enriched by data scientists, Renaissance knows learning is a continual journey - from year to year, and for a lifetime. For more information, visit www.renlearn.co.uk.

Monthly newsletter