Assessing the Academic Impact of COVID-19 on Summer 2021 Summative Assessments

Aug 06, 2021

Data from the Spring 2021 Summative Assessments Enhances Our Understanding of the Impact of the Pandemic

As we approach 18 months into the COVID-19 pandemic, evidence regarding the academic impact on students is still relatively scarce; information we have largely comes from interim assessments (e.g., Renaissance STAR or NWEA MAP). Beginning in Fall 2020, several interim assessment vendors leveraged their assessment data to examine the academic impact of the pandemic in the United States. Up to now, those data comprise the entirety of what we know about pandemic-related academic impact. Data from the 2021 summative assessments, which were conducted in the Spring, is now significantly increasing our understanding of the academic impact of the pandemic.

The Center for Assessment works extensively with numerous state departments of education on their assessment and accountability systems. As part of our ongoing work with states, the Center is currently consulting with a dozen states to understand the academic impact of the COVID-19 pandemic on their students and education systems. We are primarily using 2021 state-level summative assessment data in ELA and mathematics, English Language Proficiency (ELP) assessment data, college entrance (e.g., PSAT/SAT) data, and interim assessment data to assess the impact. In combination with extensive pre-pandemic data from these assessments, we are conducting analyses examining the pandemic’s impact on both academic attainment and the academic growth of students.  

Recognizing the urgency of getting these results quickly, we have been preparing and validating our methods for analyzing academic impact for almost a year. With data now arriving from states, we felt it would be of interest for the field to get some early insight into the initial findings from these data.

State Summative Assessment Data Show the Academic Impacts are Large

The academic impacts observed on students are significant. Both status- and growth-based analyses point to unprecedented impacts on student academic achievement. Using effect size estimates associated with Hurricane Katrina in Louisiana as the benchmark for events with an impact on student achievement (see endnotes), we observe impacts two to four times as large. In terms of performance metrics often used by states:

  • Decreases in percent proficient on state summative tests range from 5 to 11 percent in ELA from 7 to 15 percent in mathematics. 
  • Decreases in growth on state summative tests (as measured by median pre-Covid baseline SGPs) range from 5 to 20 (i.e., median baseline SGP between 45 and 30) for ELA and range from 10 to 30 (i.e., median baseline SGP between 40 and 20) for mathematics, where an SGP of 50 would indicate student growth consistent with previous years. 

The drops are in many cases staggering. States are using these impact data to help districts and schools “right-size” interventions, recognizing that catching students up will be a substantial, multi-year effort.

Academic Impacts are Consistently Large Across States 

Of the states we have examined thus far, the magnitude of the academic impacts observed has been consistent in both ELA and mathematics, across all grades tested, and across most demographic subgroups. As we receive more data, we will want to investigate whether state COVID-related policies were associated with differential academic impact on students. 

A prominent concern this year regarding the interpretation of assessment results is whether participation rates might impede analysis and interpretation. The states that we are currently working with vary in terms of participation, ranging between excellent (> 95% participation) to low (50% to 70% participation). Even prior to applying multiple imputation and propensity score adjustments for low participation, academic impact results in low participation states are consistent with those found in higher participation states. 

Academic Impacts are Consistently Large Across Grades

Unlike the interim assessment findings, analyses associated with state summative assessment results in ELA and mathematics show similar, large levels of academic impact across the examined grades and content areas. 

Whereas interim assessment results demonstrated the most impact among elementary school students, and much more modest impacts for middle school students, state summative assessment results demonstrate larger and more uniform impacts across all grades.  

One notable exception to this pattern occurred with ELP (WIDA ACCESS): Academic impacts for elementary school students (grade 1 to 6) were, on average, large to severe, while impacts decreased for middle school grades and were negligible for high school.

Academic Impacts are Larger in Mathematics Than in ELA

Corroborating findings from 2020-2021 interim assessments, we find that the academic impacts associated with mathematics on the summative assessments are larger than those for ELA. Depending upon the grade and content area, academic impacts in math are up to twice as large as ELA. 

Importantly, this is not to suggest that impacts in ELA are negligible. Instead, in most cases, ELA impacts are moderate to large. Impacts associated with ELP results are comparable in size to those associated with mathematics. 

Academic Impacts are Consistently Large Across Demographic Subgroups

Academic impact patterns among traditional demographic subgroups (e.g., ethnicity, gender, and socio-economic status) indicate consistently large and negative impacts. Observed differences in impact amongst these groups tend to be small, particularly considering the overall impact observed. 

An area of concern from the onset of the pandemic was whether different groups of students would suffer different levels of academic hardship (and impact) due to myriad education-related factors brought about by the pandemic. There was concern, for example, that given the transition to remote learning for most students those without adequate resources to learn effectively at home would be at a disadvantage. Initial data limitations do not permit this analysis, but with the potential for more nuanced opportunity-to-learn data in some states becoming available, we hope to investigate this more thoroughly in the coming months.

Academic Impacts Are Not Consistently Large by Academic Subgroup

One of the more interesting findings thus far shows disparities in impact between lower and higher achieving students; and these disparities depend in many cases upon the content area, grade, and state.

In grades 7 and 8 mathematics, it is students performing at or above proficient who are impacted most substantially, not the lowest-achieving students. In ELA grades 7 and 8, we have seen the opposite so far: low achievers tend to be more substantially impacted than high achievers. 

Within several grades in ELA and mathematics, students with prior (i.e., 2019) attainment in the lowest decile indicated modest to no academic impact. For the lowest-achieving students, many of whom were already performing well below grade level, it may not be possible to precisely pick up the extent of the academic impact through the state assessment.

Academic Impacts Vary Widely by School and District

When overall impacts at the state level are large, it follows that most schools and districts will also experience large academic impacts. In general, we find that approximately 25% of schools demonstrated large to severe academic impact in ELA, and more than 50% of schools demonstrated large to severe academic impact in mathematics. 

In a few instances, schools – some of whom were high performing pre-pandemic – demonstrated no impact or even improvement. Follow-up analyses in the coming months with finer-grained data on mode of education and attendance will attempt to better understand this variability. 

Academic Impacts That Are Small or Non-Existent for Large Subgroups of Students Raise Red Flags. 

The pandemic has the potential to tell us at least as much, if not more, about the quality of education some groups of students were receiving prior to the pandemic than it tells us about who overcame adversity and learned during the pandemic. Because pandemic impact analyses must rely on pre-pandemic results, little to no impact could imply either of these two extremes:

  • Great support and effort to overcome pandemic related learning obstacles for these students in 2021, or 
  • Lackluster pre-pandemic learning such that the pandemic did little to impact an already impoverished situation. 

If the pandemic pulled the “educational rug” out from under a lot of students, as my colleague Chris Brandt pointed out to me, “many students never had a rug under them to begin with.”

Case in point, results for high school ELP students from several states show typical or better learning during the pandemic than before the pandemic. Given the large, negative impacts observed for elementary school ELP students, we fear the latter extreme is at play.

The First Chapter in a Long Data Story

States are just now receiving data from their state summative tests and are beginning to glean information on the impacts of the pandemic on students. The early results are stark and sobering. As was the case with Hurricane Katrina, the recovery will require extraordinary effort, resources, and innovative thinking over an extended period. 

Our state partners in this work are steadfast in their determination to quickly and accurately identify these students, schools, and districts so that they can assist in supporting them in their efforts to ameliorate these impacts. Just as the pandemic and its impact on education are unprecedented, so will have to be the recovery. 

Notes on the impact of Hurricane Katrina

Sacerdote (2012) reported an effect-size impact on student achievement of -0.16 while Payne, McCaffrey, Kalra, and Zhou (2008) reported an effect size of -0.06. Depending upon content area, grade and academic subgroups, impact effect sizes indicated by 2021 summative assessment data range from -0.20 to -0.40.