COVID-19: the measure of learning losses

How much learning did primary school children in poorer schools lose due to the lockdowns resulting from the Covid-19 pandemic? Three different studies on early-grade reading from no-fee schools across South Africa show that in 2020, grade 2 students lost between 57% and 70% of a year of learning, and grade 4s between 62% and 81%. There is also evidence from the grade 4 sample that girls and those with stronger initial reading proficiency have been most negatively affected. The key question for education authorities now is how to mitigate the long-run implications of these losses.

Introduction[1]

A year after the World Health Organization declared the outbreak of the Covid-19 pandemic, about half the world’s students were still experiencing complete or partial school closures[2]. Children from poorer countries have missed substantially more classroom instruction time than children from high-income countries[3]. Learning losses amount to more than actual days lost. They also include the “deterioration” of knowledge that is forgotten over time[4]. The larger the extent of these combined losses in early grades, the more detrimental the impacts, not only for later learning but for future life outcomes, human capital and ultimately economic growth[5]. School closures may also increase the risk of dropout for vulnerable children[6].

Among developing countries, the average number of schooling days lost has been high[7], yet the ability of their education systems to respond to school closures and support remote learning has been limited. Education responses to the crisis depend crucially on home learning environments, parental ability to support learning, digital connectivity and skills; all attributes that divide richer from poorer countries[8]. Even before the pandemic, developing countries were facing a learning crisis. Children battled to keep up with curricula demands and classes were characterised by high variability in learning levels[9]. School closures will likely amplify this gap.

This article is based on our paper that examines the impact of Covid-19 disruptions on learning in South Africa using data collected pre- and post-Covid-19 related school closures[10]. We draw on three different studies on early-grade reading from no-fee schools located in three South African provinces. Using a statistical method known as difference-in-difference, we estimate short-term learning losses in reading for grade 2 and 4 students from under-resourced school environments.

Our results highlight significant short-term losses in learning in 2020 due to Covid-19 disruptions, of between 57% and 81% of a normal school year. South Africa simply has no option but to engage in significant remediation efforts in the coming years and to avoid future school disruptions as much as possible.

 

Background

Previous international research on school closures has documented evidence of long-term negative effects with children showing lower educational attainment, lower earnings, higher unemployment and a higher likelihood of being in lower skilled occupations in adulthood.

Several researchers have used this evidence to predict the impact of Covid-19 school closures on learning.Kaffenberger (2021) simulates learning losses using data from seven low- and middle-income countries. Conservatively, school closures lasting one-third of a normal year during grade 3 will result in a one-year deficit by grade 10.

In South Africa, Gustafsson and Nuga-Deliwe (2020) predict below-expected grade 12 outcomes lasting at least until 2022, and possibly as far as 2031 if there is no successful catch-up strategy.

Since these initial modelling exercises, it has become evident that school disruptions have been longer than initially expected, extending well beyond 2020.

The world has never experienced disruptions on this scale. Not only are there concurrent shocks of a widespread global economic downturn but they are being experienced simultaneously across the globe. School closures have happened at a time of heightened economic uncertainty, falling household incomes, rising unemployment and psychological costs associated with increased health and mortality risks. The unique challenges underscore the need for empirical evidence of actual Covid-19-related school losses to inform policy responses to the crisis.

This article contributes to the evidence base on learning losses in developing countries. We use longitudinal data collected on early-grade reading from three different provinces in South Africa to estimate learning losses in grades 2 and 4 due to Covid-19 related disruptions to schooling in 2020.

 

Data

To identify learning losses in grade 2 and 4, we drew on three longitudinal studies of early grade reading in Nguni home languages (isiXhosa, Siswati and isiZulu) and English across no-fee schools in the Eastern Cape, KwaZulu-Natal, and Mpumalanga. These are among the poorer provinces in the country, with 61% of households in the Eastern Cape, 50% in KwaZulu-Natal, and 54% in Mpumalanga being reliant on social grants[11]. In terms of educational outcomes as proxied by the end of secondary school pass rate in 2020, all three are also of the mid- to bottom-ranking provinces, ranking eighth, fourth and sixth out of nine provinces respectively[12].

To estimate learning losses at the grade 2 level, we use data collected in three waves from an evaluation of the Funda Wande (FW) coaching intervention[13]. This study was set in the three urban and peri-urban districts in the Eastern Cape. At the beginning of 2019 (wave 1), 10 grade 1 (FW cohort 1) and 10 grade 2 (FW cohort 2) students were randomly selected for assessment from 57 isiXhosa quintile[14] 1 to 3 schools, commonly referred to as no-fee, schools. These same students were re-assessed in the final term of 2019 (wave 2) and in the first term of 2021 (wave 3).

The second Early Grade Reading Study (EGRSII)[15] was conducted between 2017 and 2020 in 180 no-fee schools in two districts in the Mpumalanga province. Reading outcomes were tracked for a single cohort of students from 180 schools over five data collection periods from the start of grade 1.

The Early Grade Reading Study allows us to calculate learning gains over a period of Covid-19 disruptions. However, to identify what learning trajectories would have looked like in a normal year we require a counterfactual sample. For this we draw on assessment data from the Story Powered Schools (SPS)[16] study. Using similar assessments to EGRSII, 10,233 grade 2 to 4 students were assessed in isiZulu, isiXhosa and EFAL (English as a First Additional Language) from 354 no-fee schools in predominantly rural KwaZulu-Natal and Eastern Cape province schools at the beginning of 2018 (wave 1) and term 3 of 2019 (wave 2).

We compared the reading gains of students who were in grade 2 in 2020 (the Funda Wande Covid group) against students who were in grade 2 in 2019 (the FW counterfactual group) in the same schools. We also compared the gains of students who were in grade 4 in 2020 (EGRS II Covid group)[17] against students who completed grade 4 between 2018 and 2019 (SPS counterfactual group).

All the samples are drawn from quintile 1 to 3 schools. Nationally, three-quarters of South African children attend these no-fee schools. Additionally, the Nguni home languages assessed among these sample students are widely spoken in South Africa[18]. Computer access among the student samples is low, at only 13% to 34%.

The groups also had an equal mix of girls and boys.

Finally, we noted attrition rates across the assessment waves. Across the grade 4 Covid and counterfactual groups, they are similar at around 12% to 14%. Among the grade 2 sample, attrition is notably higher among the Covid group at 20.6% than among the counterfactual group (3.6%). This is due to the longer period between the assessment waves (5 terms versus 4), and significant reports of children changing schools or not returning to school at all during the pandemic period.

 

School days lost

When President Cyril Ramaphosa announced lockdown measures to limit the spread of Covid-19 on 15 March 2020, this included the closure of schools from 18 March to 14 April 2020. In fact, schools only gradually started re-opening from 1 June 2020 and were closed again from 27 July to 24 August 2020. During June and July 2020 there was a staggered return of students starting with grades 7 and 12. Grade 2 and 4 students were some of the last grades to return. Once schools re-opened on 24 August 2020, social distancing requirements meant that most could not operate at full capacity. Schools implemented platooning or rotational school attendance schedules. This added to the school days lost. Additional days were also lost due to individual school decisions to reopen late or close earlier than scheduled for the year.

Table 1. School days in 2020 for grade 2 and 4 students in the majority of study schools

		Eastern Cape (FW): Grade 2			Mpumalanga (EGRSII): Grade 4
Term	Dates	Maximum possible school days per DBE regulations	School days for majority of sample schools	School days taking rotational timetabling into account	Maximum possible school days per DBE regulations	School days taking rotational timetabling into account, assuming official school timetable followed*
Term 1	15 Jan – 18 Mar	46	46	46	46	46
Term 2	8 Jun – 24 Jul	5	0	0	0	0
Term 3	24 Aug – 23 Oct	44	44	22	53	26
Term 4	2 Nov -15 Dec	32	25	12	32	16
Total		127	115	80	131	88
% of 2019 days (199 days)		64%	58%	40%	66%	44%

*Actual school days for the majority of EGRSII grade 4 students in 2020 is unknown.

 

The first and fourth column of Table 1 shows the maximum possible days that schools could be open for grade 2 and 4 students in 2020.  Schools were meant to teach up to 15 December 2020. In the FW sample, most of the schools had stopped teaching for grade 2s by 4 December 2020. Accounting for fewer school days and rotational schedules, grade 2 students would have had 80 school days, which is only 40% of the 199 school days in the 2019 school calendar. We don’t know how many days the EGRSII schools were open, but assuming a maximum number of scheduled days and rotational schedules, then at best grade 4 students could have attended school for 44% of a normal school year. 

Absenteeism

Data from the FW schools also suggests that a significant portion of students didn’t return to school in 2020 after the initial school closure. Between the end of 2019 and the beginning of 2021,15% of sampled students left the school. Of these students, the school reports that 29% had dropped out and were not attending school at all.

In the FW study, one in five teachers report that at least half their students were absent in term 3 (Table 2). This improved by the fourth term with only 8% of teachers reporting such high levels of absenteeism. There is less available information on absenteeism in the EGRSII study but one indicator is that between 75% and 79% of grade 2 and 3 teachers in term 4, 2020 indicated that the number of students attending lessons in their grade was lower than normal. 

Table 2. Percentage of grade 2 students absent on days they were meant to attend school after schools reopened from COVID-19 closures, FW schools

	Term 3, 2020	Term 4, 2020
None	4%	9%
Just a few (1 to 4)	53%	63%
Quite a few (5-10 students) but less than half	23%	21%
About half	12%	4%
More than half	4%	4%
Most were absent	5%
Total teacher responses = schools in study	100%	100%

Source: FW, Eastern Cape sample. Teacher responses from 57 schools.

Method

The statistical tool we used – difference-in-differences - relies on the assumption that, in the absence of Covid-19, the pandemic cohort would have experienced a counterfactual achievement gain identical to the observed achievement gain in the pre-pandemic cohort.

For the grade 2 group, we essentially compared pandemic learning trajectories against pre-pandemic trajectories within the same school.

For grade 4, we matched student characteristics at end of grade 2/beginning of grade 3 across the pre-pandemic and pandemic cohorts. In addition to reading fluency, we matched on school quintile, gender and baseline household possessions (computer, television, refrigerator, car and flush toilet inside the household).

 

Results

For the Covid group who were in grade 2 in 2020, we compared their gains in letter-sound knowledge to a counterfactual (pre-pandemic) group of grade 2s in 2019 in the same schools. Almost a year after Covid-19 related school closures, the Covid group is correctly sounding only 37 letters per minute on average at the beginning of grade 3 (2021) in contrast to the counterfactual group, which averaged 46 correct letters per minute in grade 2, term 4 (2019).

We also examined oral reading fluency (ORF). We were unable to observe pre-pandemic trends for the Covid group as most students were unable to read at the start of grade 1. But pre-pandemic performance is similar across the two groups. A year after the onset of the pandemic, however, substantially worse average ORF outcomes are observed for the Covid group at the start of grade 3 in 2021, compared with the counterfactual group at the end of grade 2 in 2019. 

In a similar vein, we compared learning gains for EGRSII students who were in Grade 4 in 2020 against those of SPS students. Focusing on the SPS counterfactual group together with the SPS grade 2 and 4 cohorts, we see roughly linear growth in correct words read per minute in home language from the first term of grade 2 to the third term of grade 5. The pre-pandemic trend for EGRSII is parallel but we observe a sharp break from the trend over 2020.

We found a significant negative impact of the pandemic on reading in grade 4. Learning losses for grade 4 are around seven words per minute for both home language and English. These reading fluency gains are between 19% to 38% of the gains of the counterfactual group. Expressed differently, the estimated learning losses in grade 4 are between 62% to 81% of a year of learning. 

Heterogeneous effects

Our findings suggest that, for the grade 2s, the pandemic had the most severe impact on letter-sound reading among the least proficient students but there were no differential impacts in reading text. The grade 4 students with higher initial reading proficiency seem to have been most severely impacted by school closures. Students in the middle of the distribution of initial reading proficiency suffered greater home-language reading losses than their peers at the bottom and top end. For English reading, compared with students in the lowest tercile, learning losses were 62% and 80% higher for the students in the second and third tercile respectively.

The results by gender are also significant. At each assessment point across grades and studies, girls outperform boys. Across all three reading fluency tasks, female students were substantially more impacted by the pandemic. In grade 2, girls had learning losses of around nine words per minute in contrast to losses of six words per minute for boys. Learning losses in grade 4 were 20% and 27% higher for girls than boys in home language and English reading respectively. 

 

A way forward

Our research establishes substantial learning losses in reading for grade 2 and 4 students.

Grade 2 students lost between 57% and 70% of a year of learning, and grade 4s lost between 62% and 81%. By comparison, the number of contact teaching days lost in these students’ schools due to a combination of school closures and rotational timetabling schedules was about 56% to 60% of what they would normally have received. This implies learning to schooling loss ratios of between 1 to 1.4.

In fact, our grade 2 results may even underestimate learning losses. The grade 2 students who are not included in the estimations because they dropped out or because of higher absenteeism, have weaker pre-pandemic reading outcomes than those who were reassessed in a Covid-19 period. This implies higher ratios of learning lost to school days lost and thus a more significant deterioration of knowledge over school closures.

Students have been impacted differently.  Although girls in South Africa typically perform better than boys, both in terms of their reading levels and trajectories within a normal school year, we find some evidence from both the grade 2 and 4 samples that girls’ reading ability is disproportionately negatively affected relative to boys. One possible explanation for this is that those who were benefiting more from being at school pre-pandemic are those that lose the most as a result of closures. The grade 4 results show that the reading trajectories of children who were doing better before the pandemic (in reading terciles 2 and 3) are more negatively affected than those students with weaker pre-pandemic reading performance (tercile 1).

Our results show how learning trajectories have been impacted in under-resourced school environments, where most children have no access to remote learning opportunities. They highlight the vital importance of ensuring schools remain open and that children attend school daily. However, a limitation of this study is that we are unable to apportion these learning losses to factors beyond schooling disruptions, such as shocks to household incomes, rising hunger, and the impacts of pandemic-related events on children’s socio-emotional well-being[19].

To add to the disruption caused by the 2020 school closures, the opening of schools for the 2021 academic year was delayed by a month in response to a second wave of Covid-19 infections. On return, social distancing requirements were still in force. By mid-2021, well after these learning losses were observed, the students in the study samples would still only be attending school every second day. To counter the detrimental long-term consequences of learning losses in the lower grades, the Department of Basic Education directed that all primary school children should return to school daily from 26 July 2021 when schools reopened after the winter break. It has also implemented a campaign to vaccinate all teachers and school support staff. However, new evidence indicates that large proportions of schools were still implementing rotational timetables in the second half of 2021.

While it is almost certain that learning deficits will increase if there are more school disruptions, it is unclear whether these gaps will remain static, grow, or narrow over time. But it is likely that as children are moved up grade levels, those who are behind will continue to learn less each year. Consequently, teachers will have to increase their focus on remediation.

In response to the loss of learning, the department has developed a three-year curriculum recovery plan, which sets out a strategy to re-focus on the teaching of foundational and core content.

Successful implementation of this plan will, however, depend on the levels of agility possible within a large public system and teachers’ ability to adapt and adjust.

 

References

Angrist, N., de Barros, A., Bhula, R., Chakera, S., Cummiskey, C., DeStefano, J., Floretta, J., Kaffenberger, M., Piper, B., & Stern, J. (2021) Building back better to avert a learning catastrophe: Estimating learning loss from COVID-19 school shutdowns in Africa and facilitating short-term and long-term learning recovery. International Journal of Educational Development 84, 102397. https://doi.org/10.1016/j.ijedudev.2021.102397

Avanesian, G., Mizunoya, S. & Amaro, D. (2021) How many students could continue learning during COVID-19-caused school closures? Introducing a new reachability indicator for measuring equity of remote learning. International Journal of Educational Development. 84, 102421. https://doi.org/10.1016/j.ijedudev.2021.102421

Azevedo, J.P., Hasan, A., Goldemberg, D., Iqbal, S.A. & Geven, K. (2020). Simulating the potential impacts of COVID-19 school closures on schooling and learning outcomes: A set of global estimates. Policy Research working paper no. 9284, Washington, DC: World Bank, 2020.

Department of Basic Education. (2021)National Senior Certificate 2020 Examination Report. Available: https://www.education.gov.za/Portals/0/Documents/Reports/2021%20NSC%20Reports/2020_NSC_REPORT.pdf?ver=2021-02-25-114414-893 

Favara, M.1., Freund, R., Porter, C., Sánchez, A & Scott, D. (2021) Young lives, interrupted: Short-term effects of the COVID-19 pandemic on adolescents in low- and middle- income countries. COVID Economics. Issue 67, 4 February 2021: 172-198

Gustafsson, M. & Nuga, C. (2020) How is the COVID-19 pandemic affecting educational quality in South Africa? Evidence to date and future risks. NIDS-CRAM Insight Brief I

Hanushek, E.A. & Woessmann, L., (2020) The Economic Impacts of Learning Losses. OECD Education Working Papers. https://www.oecd.org/education/The-economic-impacts-of-coronavirus-covid-19-learning-losses.pdf

Hossain, M. (2021) Unequal experience of COVID-induced remote schooling in four developing countries. International Journal of Educational Development, 85, 102446. https://doi.org/10.1016/j.ijedudev.2021.102446

Kaffenberger, M. (2021) ‘Modelling the long-run learning impact of the COVID-19 learning shock: Actions to (more than) mitigate loss.’ International Journal of Educational Development 81, 102326. https://doi.org/10.1016/j.ijedudev.2020.102326

Kuhfeld, M., Soland, J., Tarasawa, B., Johnson, A., Ruzek, E., & Liu, J. (2020) Projecting the potential impacts of COVID-19 school closures on academic achievement. Educational Researcher 49(8): 549-565

Smith, W. (2021) Consequences of school closure on access to education: Lessons from the 2013–2016 Ebola pandemic. International Review of Education 67: 53–78

Statistics South Africa. (2020) Statistical release P0318: General Household Survey 2019. Available: http://www.statssa.gov.za/publications/P0318/P03182019.pdf.

UNESCO (2021). One year into COVID-19 education disruption: Where do we stand?https://en.unesco.org/news/one-year-covid-19-education-disruption-where-do-we-stand. Retrieved 11 May 2021

United Nations (2020). Policy Brief: Education during COVID-19 and beyond. Washington

Wills, G., Patel, L., Van der Berg, S., & Mpeta, B. (2020) Household resource flows and food poverty during South Africa’s lockdown: Short-term policy implications for three channels of social protection.

NIDS-CRAM wave 1 policy paper No 12.  https://cramsurvey.org/wp-content/uploads/2020/07/Wills-household-resource-flows-and-food-poverty-during-South-Africa%E2%80%99s-lockdown-2.pdf

World Bank (2018.)World Development Report 2018: Learning to Realize Education’s Promise. Washington, DC: World Bank. doi:10.1596/978-1-4648-1096-1. License: Creative Commons Attribution CC BY 3.0 IGO

 

[1] This article is based on the authors’ paper, Covid-19 learning losses: Early grade reading in South Africa, published in the International Journal of Educational Development, October 2021. It can be found here: https://doi.org/10.1016/j.ijedudev.2021.102480

[2] UNESCO, 2021

[3] United Nations, 2020

[4] Angrist et al 2021; Azevedo 2020

[5] Ibid; see also Hanushek and Woessmann, 2020 and Kaffenberger, 2021

[6] Smith, 2020

[7] Angrist et al op cit, UNESCO, 2021

[8] Avanesian et al 2021; Hossain 2021

[9] World Bank, 2018;  Kaffenberger, 2021

[10] We acknowledge funding for this paper from the Economic and Social Research Council (ESRC) [grant ES/T007583/1] and the Allan Gray Orbis Foundation Endowment. The paper also draws on data from multiple early grade reading assessments (EGRA) in South Africa. Without these data, analysing learning trajectories would not be possible. Accordingly, we acknowledge the funders and implementers of these EGRA related reading projects. The Funda Wande evaluation was conducted by SALDRU and funded through the Allan Gray Orbis Foundation Endowment, the Michael and Susan Dell Foundation, the Zenex Foundation and the FEM Education Foundation. The Early Grade Reading Study II (EGRS II) was conducted by the South African Department of Basic Education and funded through the United States Agency for International Development (USAID).The Story Powered Schools (SPS) data was undertaken as part of the External Impact Evaluation of SPS that NORC at the University of Chicago conducted under the USAID Reading and Access Evaluation Contract (PN 7617.010.01 GC-10F-0033 M/AID-OAA-M-13-00010).

[11] Statistics South Africa, 2020

[12] Department of Basic Education, 2021

[13] Funda Wande, an NGO that works in partnership with the Eastern Cape Department of Education, develops videos and print materials to equip and train early grade teachers on how to teach reading in African languages. Trained teachers in the Eastern Cape intervention are supported by ongoing in-classroom expert coaching.

[14]  Quintiles refer to Department of Basic Education proxies for the socio-economic status of a school. Quintile 1 to 3 schools are not allowed to charge fees and thus serve the poorest three quarters of students, while Quintiles 4 and 5 schools may charge fees.

[15] EGRSII aims to support and strengthen the teaching of early grade reading in English as a second language through the provision of structured lessons plans, learning and teaching support materials and coaching.

[16] SPS is a reading for enjoyment programme run by Nal’ibali that provided home language reading materials and weekly visits from young adults trained in running and supporting activities to promote a culture of reading for enjoyment in primary schools.

[17] EGRSII followed grade 1 students from 2017 and by 2019, 19% of students were still in grade 1 or 2. We exclude these students and restrict the sample to students who were in grade 3 in 2019.

[18] As the most widely spoken of South Africa’s 11 official languages, isiZulu and isiXhosa comprise the first language for about 23% and 16% of the population respectively (Statistics South Africa, 2012).

[19] Wills et al 2020; Favara et al 2021

Download article