The Next Generation of State Assessment and Accountability

Part 3: Changes to the Current Accountability Model That May Fundamentally Reshape Educational Assessment and Accountability

This is the final installment in a three-part series on the future of large-scale state assessment and accountability. Of course, it is impossible to know the future, but forecasts for educational assessment can be informed by examining what has shaped state assessment and accountability in the past. 

In my first two posts I examined two past shifts: the shift from norm-referenced testing to standards-based state assessment and accountability, and the shift from paper-based to computer-based testing. The first shift exemplified the roles played by educational theory, public values, and political policy in shaping state assessment and accountability; the second shift showed the influence of new technologies and what I characterized as the valuing of efficiency.

In this third and final post I focus on changes that may fundamentally reshape the next generation of state assessment and accountability. There are many possible changes that may occur. For this post, I discuss four changes that strike at the essential aspects of the current assessment and accountability model. That model, or theory of action, holds that the main considerations in educational reform are standards, assessments, and accountability.  

Standards-Based Theory of Action

Standards, assessments, and accountability are organized into a coherent system to enable a theory of action. A simplified theory of action for standards-based assessment and accountability might be stated as follows:

Assessments aligned to standards and goals will provide clear and credible information about performance outcomes (1), especially when encapsulated as accountability designations for schools based on students and student groups (2). Accountability consequences – including designations and support – will motivate, inform, and enable (4) local educators to improve their performance in student/school contexts, particularly inputs and processes (3), which will lead to increased performance toward the goals (1), which in turn will be reflected in assessment scores. 

Four Possible Foundational Changes

It is certainly important to consider how particular aspects of this general theory of action might be improved. But what are possible values and perspectives that might foundationally challenge the current system? Among the many candidates, I briefly discuss four:

1. Different goals: Individualization rather than commonality
2. Differentiated and different accountability: Less focus on school, more attention to state and district responsibility to address structural issues; less external, more internal
3. Different model or theory of action for improving school processes: Performance improvement rather than accountability
4. Different assumptions and values: Includes conceptions and criteria for technical quality

1. Individual goals and/or evidence – A view is arising that the current model of common content standards assigned by age/grade and common assessments (administered at the same time to all students) is not appropriate, and is less useful than a model of individual learning goals and paths for individual students with individualized assessments matching those goals and paths.

“Individualization” of goals and assessment can vary in the following ways:

• The same goals and evidence but at different times (“move on when ready”)
•  The same goals but different evidence (“common standards, different applications”)
• Different goals and different evidence.  

We can also observe examples of individualization in the following ways:

• Most states have differentiated “pathways” for Career Technical Education (CTE) students and dual credit options for all students
• Several states have enacted laws supporting competency-based or mastery learning
• Some Innovative Assessment Demonstration Authority (IADA) states have instituted individual “capstone” or other projects as part of the assessment evidence of students meeting their goals  

While theoretically it is possible to embed high individualization within the current standards-based accountability model, the fundamental differences result in ugly contortions. A true focus on individualization would have profound implications for standards, assessments, and accountability.

2. Differentiated and Different Responsibility/Accountability – The current model of accountability focuses on schools. In this approach, two assumptions of school-based accountability are being challenged: a) schools are responsible and should be held accountable for addressing all the conditions that create barriers to student learning, and b) the same general accountability model should apply to all schools.  

“Differentiated accountability” recognizes that different entities have responsibilities for various aspects of the educational endeavor, and each of these entities should be accountable, not just schools. One example is that some high-performing nations and states, and certainly many districts, claim the responsibility to address many “inputs” issues – for example, schools are not in the position to control or sometimes cost-effectively address issues such as teacher contracts, development of curriculum, community-based health issues, and a host of other things that impact school resources and school functioning.

Advocates of differentiated accountability may argue that just as it isn’t reasonable to hold students accountable for meeting standards until they have been given a fair opportunity to learn, so it is reasonable to hold schools accountable until schools have been given a fair opportunity to provide the opportunity to learn. Various specific models of differentiated accountability have been proposed under titles such as “reciprocal” or “balanced” accountability. Enacting differentiated accountability may have a major impact on improving student learning that could not otherwise be possible with a focus exclusively on schools.

Context is a second way accountability may be differentiated. Just as differentiated instruction may be appropriate, different accountability systems might be appropriate for schools, districts, or other educational units depending on the mission, performance history, or context. Examples include different accountability systems for alternative schools, schools with consistently high performance, or schools with unusual circumstances (e.g., rapidly declining enrollment, dealing with natural disasters).

Finally, some have also advocated different models of accountability: less “hierarchical” (external and top-down) and “bureaucratic” (embodied as rules largely applied mechanically). Contrastingly, “professional accountability” is more collegial and requires professional judgment. Some have argued that professional accountability is more appropriate and effective than the current school accountability model when:

• the ultimate goal of improved student learning or equity depends on increased professionalization of educators, or 
• when the system is so complex and/or contextual that it cannot be adequately captured in a mathematical formula. 

Examples of professional and administrative accountability models rather than “bureaucratic” models include site-based self-evaluation by educators, the use of inspectorates, accreditation models, and “ministry of education” models.

3. Performance Improvement Systems – Some have observed that current accountability systems are intended to spur improvement, but are not designed to inform improvement, except at a very high level. One reason that current accountability systems are not designed to inform improvement is that they are outcomes-based systems – they focus on distal performances, and do not include information about the key processes that are supposed to lead to the outcomes. Thus, although the state is holding schools accountable for their performance, the typical state has almost no information from its accountability system as to who did what, why, when, and how that action was related to the outcomes (e.g., student test results). Some say that educational systems would not need the current accountability systems if they had performance improvement systems that actively connect inputs and processes with outcomes. Some examples of performance improvement approaches are “improvement science” that has led to “early warning systems” evaluations of effective “coupled support systems” at the district, school, and class levels; the systems for monitoring the processes are at least as important as the specific programs they lead to, because situations always change.

Similarly, states should have systematic monitoring and improvement processes in place of their own and others’ support efforts. States should have empirically informed information about what it takes for specific supports to be effective within certain contexts, and possible alternatives. 

Instead of a theory of action that focuses on end-of-year assessment results that then trigger student and school supports the following year, a different theory of action might focus on within-year monitoring and support of student learning through interim and formative assessment —at least curriculum-sensitive if not embedded—and an ongoing program evaluation focus on school functioning and resources.  

In a radical flip, a matching accountability approach would have schools focused on what happens during the year—responsible for the nature and quality of school-managed inputs, with the summative assessments used as program evaluation tools to help inform improvement efforts.  Accountability for inputs might reciprocally expand beyond schools to include those who impact inputs, including state, district, parents, and students themselves.

Such a focus on performance improvement would move away from segmented assessment and accountability systems to more integrated systems that would focus on curriculum and instruction, with the assessment’s primary role being that of informing formative evaluation rather than external accountability. 

This approach has always been the focus of textbook/classroom materials publishers, but there are many new players. Some are going beyond materials to include other services focused on professional development, accreditation, and targeted improvement support; some of the assessment services include systems of formative, interim, and summative assessment supports, and innovative measurement models. Many current examples do not function at the state level. It will require willingness on the part of states to bridge the divide between accountability identification and support, and often for the state to be less agnostic about curriculum and instruction.  

4. Different Assumptions and Values, Including Conceptions and Criteria for Technical Quality – As new values, uses, and applications for assessments arise – such as those described in these possible future scenarios – how assessments are designed and evaluated for quality may also change. Some examples of current questions include: 

• For “through year” assessments that involve multiple tests, how should a validation argument be constructed so that it takes into account time and tests that may not follow traditional summative state assessments’ assumptions (e.g., may be “off-grade-level”)?
• For complex performance assessments with only one task, how should validity and reliability be conceived and evaluated?  
• For students who are individualizing their instructional content, timing, and experiences, how can assessments provide useful information when traditional results depend upon group results?  
• Will machine learning supplant traditional psychometrics? Will “big data” supplant data analysis for program evaluation? And if so, what are the criteria of technical quality that will replace traditional validity and reliability?

What will the future be in state assessments and accountability? I’d welcome hearing your views about the exciting future we are already living in.

I gratefully acknowledge those who have helped shape my thinking on this topic, chiefly my colleagues at the Center for Assessment, especially Charlie DePascale, with whom I’ve shared many hours of engaging conversation. However, the opinions expressed are mine and are not intended to represent the Center or my associates.