Testing experts know a lot about how to conduct scoring of students’ written responses to assessment items. Raters are trained under strict protocols to follow scoring rules accurately and consistently. To verify that raters did their job well, we use a few basic score quality measures that center on how well two or more raters agree. These measures of agreement are called inter-rater reliability (IRR) statistics, and they are widely used, perhaps in part because they are easy to understand and apply. 

As 2019 drew to a close I had the chance to reflect on the conversations I've had with many of my colleagues throughout the year, and one topic of conversation that sticks out to me is frustration about the minimal value-add of work focused on large scale assessment and state-level accountability systems to the student experience. 

Developing a Better Understanding the Role of Assessment and Accountability in Improving Student Outcomes

In his CenterLine post, Can Educational Assessment Improve Teaching?, Executive Director Scott Marion invited readers to share their thoughts on the complex, but critical issue of identifying ways that assessments can be used to improve teaching quality. In this guest post, we share Kadie Wilson’s response to Scott’s invitation. Kadie Wilson is Assistant Superintendent in New Hampshire School Administrative Unit #9.

The end of the decade signified the end of an era here at the Center for Assessment; Charlie DePascale officially retired on December 31, 2019, after more than 17 years as a Senior Associate. 

In considering a decade of large-scale state assessment, I’m reminded of a moment in June 2010. while attending the Council of Chief State School Officers (CCSSO) National Conference on Student Assessment (NCSA) in Detroit, Michigan, I was part of a crowd gathered in the hotel lounge watching the U.S. Men’s National  Team (USMNT) play a World Cup soccer match against Algeria.  

Depending on your slant, you probably have a favorite among the many sayings around the kinds of things that come in threes. Some people focus on the belief that tragedies occur in threes. I prefer to focus on how some of the best things come in threes:

1. Freud’s id, ego, and superego
2. The three books in Tolkien’s Lord of the Rings series
3. The 90s hit teen pop sensation Hanson 

Okay, some might argue with the three Hanson brothers – how about The Jimi Hendrix Experience instead? 

How would I design an assessment or assessment system if my goal was to improve teaching? My colleagues, Chris Domaleski and Leslie Keng, and I met recently with state assessment leaders who are thinking about reforming their assessment system. 

Technology-Enhanced Items (TEIs) are a kind of test question or task. A characteristic feature of TEIs is that, in contrast to traditional multiple-choice (MC) items, which require the selection or “bubbling” of a single option, TEIs generally require test-takers to make more than one interaction with the item.

The most interesting TEIs are simulations with game-like contexts. Picture a virtual laboratory where the goal is to isolate a specific compound, or a simulated garden where the test-taker can conduct an experiment to learn about (or be tested on) a concept in genetics. 

If you've ever worked with someone from the Center, been in a Center staff meeting, or even had dinner with someone from the Center, you know that we refer to Theories of Action incessantly. It may sound wonky and weedy (and it is), but there's a reason why we value it so much. That's because a theory of action (TOA) can help us clarify what we truly believe should happen if a program or system is implemented. 

Defining a Theory of Action to Help Guide Longer-Term Goals

I am encouraged that so many educational leaders are wrestling with systematically bringing educational reforms to scale. Unfortunately, as these leaders have come to realize, achieving widespread implementation of meaningful reforms is really hard – especially when pursuing a goal of increasing assessment literacy.