This is the first in a series of three posts on planning for the examination of the validity of scores collected through remote test administration. Next up is a discussion of a framework for the types of analyses that will be useful for understanding the degree to which scores are comparable between remotely tested students and students tested in the classroom, what might be done to adjust them if they are not, and the conditions under which data can be collected to support those analyses.

What is fairness in educational testing? 

We cannot know what classrooms and teaching will look like in the coming school year, but that doesn’t need to prevent states from planning ahead for accurately equating spring 2021 summative assessments. In making that statement we assume that those tests will happen and their blueprints and administration will be unaltered while acknowledging that the conditions of learning may be very different than in the past.

Although so much about the future seems uncertain, we are excited this month to bring a little normalcy into our world by addressing key questions and challenges in educational assessment and accountability through our 2020 summer internship program. This summer, the Center welcomes four advanced doctoral students who will work with the Center’s professionals on projects that will have direct implications for state and national educational policy. Each intern will work with a Center mentor on one major project throughout the summer.

According to Education Week, “as of March 23, 2020, 7:31 p.m. ET: 46 states have decided to close schools. Combined with district closures in other states, at least 123,000 U.S. public and private schools are closed, are scheduled to close, or were closed and later reopened, affecting at least 54.8 million students”.

Testing experts know a lot about how to conduct scoring of students’ written responses to assessment items. Raters are trained under strict protocols to follow scoring rules accurately and consistently. To verify that raters did their job well, we use a few basic score quality measures that center on how well two or more raters agree. These measures of agreement are called inter-rater reliability (IRR) statistics, and they are widely used, perhaps in part because they are easy to understand and apply. 

Without a doubt, the public is skeptical about using machine scoring for examinees’ written responses. This skepticism makes sense because we know that machines do not score all elements of writing equally well. Machines do not “understand” creativity, irony, humor, allegory, and other literary techniques, opening them to criticism for their insufficiency in evaluating some of these more subtle qualities of writing. 

Earlier this month, the Center for Assessment held its 15th annual colloquium, the first named in honor of Center co-founder, Brian Gong. The Brian Gong Colloquium is a two-day meeting arranged by the Center for Assessment to discuss select topics of importance in testing and accountability with recognized experts. 

The 2019 meeting focused on the present and future use of learning analytics, machine learning, and artificial intelligence in educational assessment, and highlighted topics including: