Letting Go of Labels in School Accountability Systems

How the next ESEA Reauthorization Can Improve Federal Accountability Requirements

The Every Student Succeeds Act (ESSA), which is the current reauthorization of the Elementary and Secondary Education Act (ESEA), contains federal accountability requirements wherein all states must create an accountability system for identifying schools for support on the basis of a collection of indicators, with the majority of weight given to statewide achievement tests scores. ESSA refers to this requirement as “Annual Meaningful Differentiation” and prescribes the aggregation of information across accountability indicators. A consequence of this requirement is that many states apply labels to schools on the basis of the annual meaningful differentiation, a practice that has undermined the equity-driven purposes of the law. 

States have come up with a variety of ways to characterize schools as a result of the aggregation requirement, for example, A-F letter grades, star systems, or sets of qualitative descriptors (e.g., “school of excellence”, “meets standards”, “focus”, “priority”). One of the motivations to produce a single, summative label is to satisfy the public transparency purpose of Title I accountability. The idea is to help the public make sense of a large amount of available data related to schools. 

However, because school quality cannot effectively be summarized on the basis of a limited set of unlike and arguably incomparable indicators, the requirement to create an aggregate index or categorization of schools leads to unintended, negative consequences that perpetuate inequity rather than address it. The general public has little reason to doubt the labels ascribed by a state for their local schools, and those parents with resources to select their children’s school on the basis of perceived quality are likely to rely on the publicly-available state ratings, contributing to the increasing school segregation and inequities across districts we have seen over the past 25 years.

The Unintended Negative Consequences of “Annual Meaningful Differentiation”

We argue that the current system is flawed for two distinct but related reasons: 1) federal law insists that the method of “Annual Meaningful Differentiation” is informed in large part by achievement test scores, which are insufficient indicators of school programming quality, and 2) the practice of combining information across accountability indicators is inappropriate for making decisions about school program quality. 

First, as others have been emphatically saying for decades, federal accountability places an overemphasis on measures of student achievement. The relationships between demographic factors (e.g., race/ethnicity, family income, level of parental education, maternal age at birth) and achievement are substantial and enduring. As such, school-level score differences on statewide assessments are better understood as a reflection of systemic racism and school segregation than as differences in program effectiveness. Decreasing the emphasis on student achievement by expanding the definition of school quality within our accountability systems would create opportunities to include measures that are more sensitive to changes in school program quality and more reflective of what local communities value in their schools (e.g., school climate, access to music and arts, equity in access to rigorous courses). 

Second, the federal requirement to create a system for combining information across indicators–often resulting in summative labels for schools–diverts attention from the available information that can provide insight on differences among school program quality. Requiring that states summatively categorize schools under the current constraints lends a false sense of scientific legitimacy to the notion that school programs can be rank-ordered and simplistically compared. The invitation for comparison across schools has not rallied public support for fully funding low-performing schools and closing disparities in outcomes. Instead, evidence suggests that summative school ratings have contributed to increases in school segregation (Davis, Bhatt, Schwarz, 2015). This finding implies that the implicit theory of action driving the current movement of accountability-based school reform is flawed and is serving to perpetuate systemic inequities rather than eradicate them. 

Recommendations to Improve School Accountability in the Next Reauthorization of ESEA

Below we offer suggestions for improving federal accountability policy, which build on suggestions originally proposed by Marion, Domaleski, and Brandt (2020) for the next reauthorization of ESEA. 

1. Expand the definition of a high-quality school.

Encourage states to develop a more complete picture of school quality through their accountability systems. We suggest expanding the use of indicators that are directly within a school or district’s control and are sensitive to change in school programming quality. These may include program-related input measures (e.g., quality curricula, access to rigorous coursework, parent perceptions) as well as a broader set of student data that provide insight into students’ experiences (e.g., quality of student-teacher relationships, student sense of belonging). Incorporating these types of direct and sensitive measures of school conditions empowers the school to demonstrate meaningful changes and improvements in learner experiences within their programs. 

The next authorization of ESEA should significantly lower the restrictions around the types of indicators that can be included in the accountability system. For example, we should eliminate the restriction that the School Quality/Student Success (SQSS) indicator, or set of SQSS indicators, must be disaggregated by subgroup. This requirement, while well-intentioned, restricts valuable indicators related to school quality because it requires that the indicator be collected at the student level. It’s possible to imagine relevant school-level indicators, currently precluded from accountability systems, that may be paramount for understanding the degree to which schools have adopted equitable learning practices (e.g., the degree to which schools are stratifying students into tracks).   

2. Remove the “Annual Meaningful Differentiation” requirement.

Remove the constraint that states must attempt to combine information across accountability indicators. A school is more complex than any single measure or index can imply. The practice of summarizing school performance across indicators to create a single index or label is not helpful for improving the school or for providing an accurate picture of school quality. 

Instead, encourage states to design school reporting dashboards that better communicate quality relative to a range of indicators. Many states, for example, California and West Virginia, are already making good use of technology to provide more accurate and actionable pictures of school quality.

Under ESSA, states are required to use the “Annual Meaningful Differentiation” for identifying a set of schools for support and improvement efforts. Removing this requirement will mean that states could re-imagine the mechanisms by which they deliver supports to schools. For example, states could empower all schools with the resources and tools to develop their own theories of change and track progress towards their own improvement goals. In the case of limited state funding and capacity, the state could ensure that the distribution of financial supports is focused on those schools with the greatest need for supplemental resources and those communities that have been historically marginalized. 

3. Include Accountability in the Innovative Assessment Demonstration Authority (IADA).

As Marion (2021) suggests, encourage states to innovate in both assessment and accountability together as a system within the IADA federal pilot program. This freedom will advance our understanding related to the options for how assessment and accountability could be differently conceptualized and implemented at the federal, state, and local levels to improve equity. For example, there is considerable promise in allowing schools and states to work together to develop localized goals, theories of action, and mechanisms for progress monitoring through an empowerment evaluation model (Fetterman, 2001). This type of localized improvement system has the potential to create meaningful and sustaining school improvement. Opening up the opportunity to innovate in accountability as part of the IADA would create the context for developing and testing new theories of action and models for accountability. 

None of these suggestions are silver bullet solutions. Rather, if enacted thoughtfully, we see them as small changes that have the potential to interrupt long-standing patterns of privilege and oppression. 

ESSA marked an important shift toward a more holistic approach to accountability by encouraging multiple measures of school and student success. We hope the next reauthorization of ESEA will offer states and continue the trend toward opportunities to more fully capture elements of school quality. 

Susan Lyons, Ph.D. (@susanclyons) is an independent consultant. Please submit any comments to susan@womeninmeasurement.org.