A Tricky Balance
The Challenges and Opportunities of Balanced Systems of Assessment
The seminal publication, Knowing What Students Know: The Science and Design of Educational Assessment (NRC, 2001), crystalized the call for balanced systems of assessment. Yet almost 20 years have passed and there are very few examples of well-functioning systems, particularly systems that incorporate state summative tests. Why? In spite of recent efforts to articulate principles of assessment systems, creating balanced assessment systems is really hard!
The Center for Assessment’s 20th Annual Reidy Interactive Lecture Series (RILS) offers us an opportunity to take a look back and a look ahead. The Center for Assessment has learned a lot about designing and implementing high-quality assessment systems over the past 20 years. One of our major goals for RILS 2018 is to leverage the lessons of the past to forge an ambitious agenda for ways in which balanced systems of assessment can enhance equitable and excellent learning and life opportunities for all students.
The call for balanced systems of assessment in Knowing What Students Know (NRC, 2001) and many subsequent writings (e.g., Chattergoon & Marion, 2016; Gong, 2010, Marion, 2018, Shepard, 2001, Shepard, Penuel, & Pellegrino, 2018) were born from a recognition that most assessments were doing a poor job of serving what should be a primary purpose—improving learning and instruction. Educators understand that large-scale summative tests are far too distal from instruction, at the wrong grain size, and administered at the wrong time of year to make a difference in their daily practice. Therefore, the calls to balance—actually rebalance—assessment systems were motivated by desires to enhance the utility of the system for improving learning and teaching as other stated purposes such as accountability and evaluation.
Assessment systems are balanced when the various assessments in the system are coherently linked, often through a clear specification of the learning targets, comprehensively support multiple purposes and uses, and continuously document student progress over time. These properties of coherence, continuity, and comprehensiveness originally described in Knowing What Students Know (NRC, 2001) help create a powerful image of a high-quality system of assessments. Building from NRC, 2001, we have found that coherence, utility, and efficiency are a bit more practitioner-oriented when working with district and state leaders (Chattergoon, 2016; Chattergoon & Marion, 2016).
Utility is the degree to which the system provides the information necessary to support the intended aims of an assessment program. Importantly, utility is not a general property of assessments but, like validity, utility must be designed and evaluated for each purpose and intended use such as monitoring, program evaluation, and accountability. We need to do much better at maximizing the assessment utility for each of the stakeholders in the educational system efficiently as possible. Balanced (and efficient) systems of assessment have been posited as a way to serve the needs of the various stakeholders in the educational system, but we argue that utility has not been addressed carefully enough in the design of assessment systems. We expand our conceptualization of utility and other criteria for evaluating assessment systems in upcoming papers and it will be a through-line for our discussions at RILS.
As noted earlier, balanced systems have proved elusive – like assessment unicorns. We think we should see them, but we don’t! We have identified many factors that have likely prevented assessment system implementation. These barriers might act alone or in concert with other factors to prevent high-fidelity implementation of balanced assessment systems. There are more potential barriers than we can discuss at RILS, but based on research and our extensive experience we have identified the following critical factors that hinder assessment system design and implementation:
- Assessment literacy at multiple levels of the system
- Politics, policy, and political boundaries
- Lack of attention to learning and curriculum
- Commercialization, proliferation, and incoherence of assessments
Given these seemingly insurmountable challenges, we humbly identify some high-leverage strategies that can increase the likelihood of seeing high-quality balanced assessment systems implemented in practice. We acknowledge that “solving” any one of these challenges will be difficult at best, but we are poised to work with colleagues in research and practice to make as much headway as possible.
We are producing a series of papers related to the challenges and opportunities associated with balanced assessment systems and we have organized several sets of concurrent sessions at RILS to allow participants the opportunities to engage with colleagues to better understand how to overcome the barriers in their contexts. We look forward to lively, engaging, and productive discussions when we gather together in Portsmouth, New Hampshire on September 27th and 28th.
For more information and to register for RILS 2018, please visit 2018 Conference page on the Center website.