Reflecting on Lessons Learned in Assessment and Accountability in Education in 2022
Identifying the Problems to Support Learning Recovery
To wrap up 2022, Scott Marion provides his take on student learning recovery following the pandemic’s impacts on assessment and accountability in education and emphasizes the need to identify problems to properly support recovery efforts.
As 2022 draws to a close, we hope that the pandemic’s most egregious interruptions to “normal life” are behind us. As countless studies have shown, however, its effects on students’ learning and their social-emotional well-being aren’t going away anytime soon. Supporting student recovery has been and will continue to be a major focus for us here at the Center for Assessment. But if that focus is going to produce positive results, we need to clearly identify the problems we’re trying to solve.
The most transformative scientists are those who ask the richest questions. When Einstein was just 15 years old, for example, he asked himself what he would see if the train on which he was riding was traveling at the speed of light. Asking this amazing question guided Einstein’s life’s work and led him to revolutionize theoretical physics.
In our more humble domains of educational design and evaluation, we must focus on getting at the root of the problem we’re trying to address because, as my colleagues Juan D’Brot and Chris Brandt advised this year, we don’t stand a chance of launching the right programs or evaluations unless we clearly identify the problem to be solved.
Role of Assessment and Accountability in Education in Learning Recovery
Needless to say, the most pressing issue for all of us in education is to help school and district leaders accelerate student learning at extraordinary rates. High-quality assessments can play an important role in monitoring and supporting this recovery. A key aspect of defining the recovery problem is understanding where students are currently and how far they need to go.
Unfortunately, many of the reports we saw this year did not include the level of detail necessary to help us understand the scope of the problem. Damian Betebenner, Leslie Keng, and I offered this guidance for reporting 2022 results, which we hope states will incorporate into their regular reporting going forward.
Two NAEP releases this fall—the long-term trend results in early September, and the most important one in years, the “main” results on Oct. 24—portrayed a comprehensive picture of the deep trouble we’re facing. Unfortunately, the release also sparked many weak claims about the effects of remote instruction on student learning. These claims were an example of getting distracted from the real problem, as I wrote here.
Supporting learning recovery was the focus of our annual conference (RILS) in September. Aside from the sheer enjoyment of seeing so many friends and colleagues in person, we collectively wrestled with how we can best use assessment and accountability systems to monitor and support educational recovery. My Center colleagues, along with several state and district leaders and other experts, shared their rich experiences and expertise about strategies for using assessment and accountability in education to monitor and support learning acceleration.
In the two days of the conference, I never heard any of the almost 100 participants point fingers about decisions regarding remote learning. Rather, they were all looking forward and focusing on how they could improve school and student performance. If you missed the conference, you can access the conference presentations and other materials on this page.
Design Thinking and Innovation
We spend a lot of time every year working with our state and district partners to design or redesign assessment and accountability systems, but we spent even more time on those things in 2022. The first step in this type of work involves clearly defining the issue or problem and then specifying the intended goals of the solution. Chris Domaleski described the multiple steps of this process for large-scale assessment programs and school accountability systems.
While they may sound wonky, theories of action are critical heuristics in these projects, because they require everyone to spell out and regularly re-evaluate the conditions and logical connections that are necessary to move from problems to solutions. We believe so strongly in this strategy—and harp on it so much—that our partners are surprised if someone from the Center doesn’t mention “theory of action” within the first 10 minutes of a conversation about design.
We’re excited that so many of our state and district partners, like Nebraska, Utah, Montana, Hawaii, Chicago, and Boston, are exploring innovations to better support learning for all students. We use careful problem identification and theories of action to help guide this work, and we describe them in these recent reports for Montana and Utah.
On the other hand, it’s no secret that we’ve been raising questions about through-year assessment systems. We’ve been somewhat challenging because we haven’t seen clear problem identification and explicit links between the problems states are trying to solve and through-year models as the proposed solution.
Several of us are continuing to build on the expertise we harnessed in our November 2021 through-year assessment convening to further explore the connections between identified problems and proposed solutions. Nathan Dadey and Aneesha Badrinarayan’s post about the connection between curriculum and assessment is a great example. Similarly, Carla Evans questioned whether state assessments can inform instruction, and Allison Timberlake and I examined how this issue plays out on the ground. We struggled to understand how the proposed solution (through-year assessment) wouldn’t cause more problems than it would solve.
To be clear, we are not reflexively opposed to through-year assessment. In fact, we are working with several state partners to connect the strong assumptions associated with through-year models with the evidence or logic necessary to help these programs meet at least some of the stated goals. But right now, we see far too few of these explicit connections being made.
Communication and Reporting
“Our score reports are the only way we communicate with the public, yet they’re the last thing we attend to in test design.”
I’ll never forget those wise words spoken by Ron Hambleton, one of the true giants in our field, who passed away this past spring. Chris Domaleski paid homage to Ron in this post and recounted many of his lessons on score reporting.
Part of making tight connections between problems and solutions is specifying clear use cases —clearly describing the ways people will use the proposed system they’re designing. That’s an important part of the use case for assessment, but we don’t do it very well. In that spirit, we spent a lot of time in 2022 thinking about improving the way we communicate complex information pertaining to assessment and accountability in education.
Every year, the Center convenes the Brian Gong Colloquium. It was a joy to be in person again for this small meeting, where Center professionals dig into a complex topic with a handful of experts. In May, we gathered in Boulder, Colorado, to delve into ways we could improve our own communication and the communication strategies used by our clients. We learned a lot and summarized those learnings in this post.
Those of you who’ve been following the Center’s work know that we’ve stepped up our communication intensity in recent years, in service of improving policy and practice. We are thrilled that Catherine Gewertz joined our team as our first editorial director in September. Catherine is a nationally recognized journalist—just completing 22 years at Education Week—who supports the Center’s strategic communication efforts. In recognition of her reporting on assessment, the National Council on Measurement in Education named Catherine the winner of its first award for Excellence in Public Communication in 2019. Catherine is already pushing us to better connect, through clear thinking and writing, the issues we’re trying to address with our proposed resolutions.
Wishing You All the Best
We look forward to supporting you in 2023 and beyond as you work to identify problems and identify the best-fitting solutions. For now, I wish you all the best for a terrific holiday season and a happy and healthy 2023.