Learning About Reading Assessment
Our assessments don’t reflect the way children learn to read
There has been a lot of press lately about a learning recession, a term that describes the general decline in academic achievement since about 2013, but the story in reading is different than math. Average reading achievement for elementary, middle, and high schools over the past 35 years, as measured by the National Assessment of Educational Progress (NAEP), looks so flat we could use the trend line to hang a picture. Yes, there have been slight fluctuations over the years, but the overall picture is remarkably flat.
I started wondering about reading scores during and following the COVID-19 pandemic. One common explanation for why math scores crashed more than reading was that parents were more likely to read to their children than to do math with them, which could partly explain why scores dropped much more in math than in reading. However, after students returned to school, math scores in many states approached pre-pandemic levels, while reading scores haven’t returned to 2019 levels. If parent involvement explained the cushion on the way down, why didn’t it seem to help on the way back?
This curiosity prompted me to dive deeper. This fits well with my role as a member of the National Assessment Governing Board (NAGB), where I spend a lot of time examining NAEP score trends. Many have documented the improvement in Grade 4 math scores from the late 1990s through about 2013, generally associated with gains among the lowest-achieving students. Unfortunately, reading score trends have been stubborn.
When I began to puzzle over this stubborn trend, my first thoughts centered on our ability to measure reading comprehension. Are we just bad at it, or are we measuring the wrong things? I also questioned whether we were accounting for shifts in the population of test takers, which might be masking underlying improvements in reading achievement.
I also wondered whether teachers weren’t teaching reading comprehension or were teaching it poorly, but I didn’t know enough about reading instruction to make this argument. Additionally, some states and districts defied this flat-line trend, showing significant gains as measured by both NAEP and state tests. Therefore, it seemed possible to increase reading achievement, but not at a scale that improves the big picture.
Learning to Read, Assessing Reading: Building Insight
These questions led us to focus our annual learning retreat—the Brian Gong Colloquium—on how we assess reading. This annual Center convening allows us to learn about a particular topic from a select group of experts. These learnings have enabled us to better serve our clients and the broader field. Two Center professionals plan the colloquium each year; this year, the job fell to my colleague Laura Pinsonneault and me. We were both interested in this reading dilemma.
Seven national reading experts—literally a who’s who of reading1—joined us for two days in early May to help us unpack what it means to learn to read and comprehend text, and how we should think about assessing reading achievement.
Confronting Our Misunderstandings
It turns out that much of what we thought we knew was wrong, including the common notion that students “learn to read” through 3rd grade, and “read to learn” after that. Children learn long before they can read words. Preschool experiences, including being read to by parents or preschool teachers, provide much of the knowledge that can impact language and reading comprehension.
Additionally, students need to continue learning new reading and comprehension skills and to adapt well-learned ones throughout their school years. Similarly, they need to learn to make meaning from text as soon as they are introduced to reading. This false separation has caused considerable confusion in curriculum, instruction, and assessment over the years.
Models of Reading
Our experts walked us through several models of learning to read, from simple to more complex. I found the model in Direct and Indirect Effects Model of Reading, by Young-Suk Grace Kim (2000), shared with us by Rebecca Silverman, quite compelling. This and other models helped us begin to understand the complexity of reading comprehension. We pressed our experts to share insights about how we can improve reading assessment, given what we were learning about how children learn to read.
They pointed out that we know how to measure the skills in the “word reading” box using assessments such as DIBELS, ROAR, ReadBasix, and others that can be administered quite quickly.
However, our experts shed light on the many factors that contribute to students’ ability to comprehend text. Catts and Kamhi noted that “comprehension is not a skill or set of skills; rather, it is a complex multidimensional ability. In fact, reading comprehension is one of the most complex activities that we engage in on a regular basis, and our ability to do so is dependent upon a wide range of knowledge and skills.” This complexity makes it incredibly challenging to assess students’ reading comprehension, especially within the constraints of large-scale assessment, such as ensuring score comparability, limited testing time, and finances.
Measurement Challenges and the Problem of Transfer
Most of our large-scale reading tests require what learning scientists call “distal,” or “far,” transfer. Transfer, or the ability to generalize what one has learned to new contexts or problems, is an important goal of education, but it’s hard! We start by teaching students to solve problems in familiar contexts (e.g., their neighborhood) or similar to those they have been solving in class. Eventually, we want them to be able to comprehend, in the case of reading, all sorts of familiar and unfamiliar texts.
The ability to make sense of these “drop from the sky” passages calls on students to reason with unfamiliar content in ways that draw on their natural intellectual skills more than on applying what they learned in reading classes. Daniel Willingham, E.D. Hirsch and others have long argued about the importance of building knowledge to develop students’ ability to comprehend texts. When students encounter texts on their state tests that depict unfamiliar subjects or contexts, it becomes what one of our experts termed “uber-transfer.” The hurdles are even higher for students with specific learning challenges and multilingual learners.
Some Potential Ways Forward
We like tackling hard problems at the Center, so we continued pressing for ideas we could try. One of our experts, Hugh Catts, noted that Louisiana’s Innovative Assessment Pilot (IAP) showed some promise. The Center, led by Nathan Dadey, was heavily involved in this project.
Louisiana is one of the few states with a “model” ELA curriculum that about 80% of its schools use. This allowed the state and its partners to design an assessment system that drew on familiar texts as the basis for the reading passages. It also included passages less familiar to students (warm and cold reads), which were intentionally selected to help us understand how well students can engage in transfer. Unfortunately, this pilot was suspended, in part because of the challenges of operating under the federal Innovative Assessment Demonstration Authority.
Conceivably, states could try an approach like this, without having a common curriculum like Louisiana. What other ideas could we pursue? I will dive deeper into additional challenges and potential approaches in subsequent writing, but here is just one idea for now.
Again, learning from Dr. Catts: Reading passages on state tests could be drawn from social studies and science topics that students were exposed to during that school year. These do not have to be highly specific directives or tied directly to a particular curriculum; States would let educators know that the Grade 7 reading test will be based on marine biology and the history of the Industrial Revolution, for example. This would direct teachers to the content that will be tested and help students build the knowledge base and vocabulary needed to understand the reading passages on the state tests. I still have a lot more to learn about this very important topic. I will continue exploring how we might better measure students’ comprehension of texts and other information sources, especially on large-scale assessments.
I am grateful to Hugh Catts for his very helpful comments and suggestions on a previous draft.
- Gina Biancarosa, University of Oregon; Hugh W. Catts, Florida State University; Jeannette Mancilla-Martinez, Vanderbilt University; Jamila Newman, Kaleido Strategies Group; Margaret Osgood Opatz,Capti Assessment; Rebecca Silverman, Stanford University; and Colette Sims Owens, Magpie Literacy.
↩︎