When it Comes to School Ratings, Meaning Matters
What’s in a Grade, Anyway?
Letter grades are a popular way to describe performance. I’m referring to those same letter grades you received in school - A to F. We all know that the coveted A is “superb,” and an F warns that performance is completely deficient. What’s a C? Perhaps it is used to communicate “good enough” (but not great), or possibly it means “average.” Should we worry that those are often two different things?
And of course, grades aren’t just used in schools. Conduct a quick search of things that receive letter grades and one is likely to find examples that are both consequential and frivolous. For example:
- Credit rating agencies use letter grades to signal the risk of default
- Health departments issue letter grades to restaurants to communicate compliance with food safety regulations
- Sports writers often rate team performance using letter grades
- Movie reviewers use letter grades to communicate if a film is worth the price of admission
Given their widespread appeal, it’s worth asking: does everyone really know what the grades mean?
What’s in an “A”?
If an A is superb, then AA+ should be something like “stratospheric.” But some readers may recall that in August of 2011, the influential credit rating agency Standard and Poor (S&P) temporarily downgraded the United States credit rating to AA+. In the midst of a serious recession, S&P explained this downgrade reflected a heightened concern about stability and predictability of U.S. markets. This downgrade had substantial negative impact on global financial markets. That’s not stratospheric. In fact, in this case, AA+ was not good at all.
What about health inspections? How certain can one be that it is safe to dine at an A-graded restaurant? The New York Department of Health explains that restaurants receiving an A have not received more than 13 points in violations. This explanation leaves room for the highest rating to include restaurants that have been cited for failing to keep food at the proper temperature, serving raw food that has not been washed, or failing to properly sanitize cooking utensils. That kind of activity is far from flawless.
The point is not that these rating systems are inaccurate or unfair, only that they may not be as simple and clear-cut as they appear. All rating systems, including those that use monikers other than letter grades (e.g. stars or performance levels) must be built on a set of criteria or decisive rules that define each category if they have any promise of being useful. Is the best hotel the one with the most lavish rooms, the closest to the beach, the most attentive service, the cheapest rates, or some combination of these factors? The reasoning behind the rating matters.
So, What Does All This Have to do with School Accountability Systems?
This year, states have developed school accountability systems in response to the Every Student Succeeds Act (ESSA). While these systems include multiple indicators, to many stakeholders, the outcome of central interest is the overall rating or classification that is produced for each school. These ratings are often used to identify schools that merit reward or require support, and to evaluate the efficacy of educational programs and policies.
In many cases, however, accountability ratings are set normatively (e.g. the top 10% of schools receive an A or five stars). Using this approach alone fails to ensure that the system reflects the policy values and prioritized outcomes that have been established by state leaders.
So, what’s the alternative? My colleagues and I have developed a framework for establishing more clear and meaningful performance standards for school accountability systems. This framework draws on the principles used in setting standards for assessment. The guiding idea is to explicitly connect accountability performance standards or ratings to the state’s policy goals. This action requires 1) developing and vetting clear expectations for school performance and 2) using both data and expert judgment to determine which school profiles match those expectations.
There are various approaches that can accomplish this outcome, just as there are many ways to set standards for assessments. In our paper, we illustrate the approach in some detail and describe its implementation in two states.
Any strong approach should reflect the following principles, which are taken from the Standards for Educational and Psychological Testing (2014) adapted to apply to school accountability systems.
- Document Rationale, Procedures and Results
Provide an explanation of the reasons for selecting the approach, including justification for how the method fits the context and supports the purposes and uses of the system.
Further, the procedures and results should be described in detail, including development of performance expectations, selection and qualification of participants, and variability in judgments.
- Ensure the Process Allows Appropriate Participants to Apply Their Knowledge and Experience
The individuals selected to provide judgments should be well-qualified and should represent the range of perspectives and interests that would be considered when establishing school performance expectations (e.g. teachers, administrators, representatives from key interest groups).
Moreover, the procedures and expectations of the judges should be clear and straightforward and all participants should be well-trained.
- Include Information Associated with Relevant Consequences and Criteria
Judges should have access to appropriate data to inform their decisions (e.g. data for each accountability indicator for a range of schools and by subgroup).
Additionally, to the extent that data associated with key criteria that are not otherwise available in the accountability system are accessible, these data should be included in the process to inform decision making (e.g. college-going rates might be included in a system that privileges college readiness).
Finally, participants should have access to information about the meaning and consequences associated with each performance classification.
For too long, school accountability systems have produced outcomes that were arbitrary and unclear. The methods used to determine these ratings are often obscure and the connection to the overall policy goals left unaddressed.
In order for school accountability systems to serve as a useful component of an overall strategy for improving student outcomes, we must start with clear, well-defined performance standards that are demonstrably connected to policy goals and established using a trustworthy and transparent process.
It’s time to develop systems worthy of a higher grade.