Ready or Not: AI is Changing Assessment and Accountability

The Opportunities and Risks of Artificial Intelligence in K-12 Assessment and Accountability

The immense power of artificial intelligence (AI) is transforming society and the economy. K-12 education is no exception. In this post, and its accompanying table, we identify some of the opportunities for innovation that advances in AI might facilitate in K-12 assessment and accountability, and the potential risks they might present.

Advances in AI are beginning to change the roles that human experts play in design and implementation processes. These roles shift us from being the de-facto direct creators of end products to the supervisors of AI algorithms and the evaluators, synthesizers, and refiners of the resulting products created by these algorithms.

Even as those shifts happen, people are debating how to think about them. Should we consider the performance of these algorithms a human-like intelligence? Or simply highly sophisticated automation based on large volumes of input? No matter what view we take, it is clear that advances in AI shape the way knowledge work is being conducted.

Implications of Artificial Intelligence for Educational Assessment

AI advancements necessitate changes across curricular design, teacher professionalization, and approaches to assessment. For example, how advances in AI shape our understanding of which competencies matter for a future-oriented workforce is relevant to how teachers teach these competencies through enacted curricula, and how they are assessed and valued in assessment and accountability systems.

Yet the remarkable speed at which AI and other elements within the Fourth Industrial Revolution are advancing outpaces the rate at which many educational policies, agencies, and initiatives move at scale. The good news is that not all foundational educational principles need to be thrown out and replaced with seemingly “new” initiatives, at least not wholesale. Early literacy and numeracy, for example, will remain critical in pre-K through middle school. Still, if the educational impact of discussions about coding skills and computer science more generally is any guide, we will see new competencies emerge due to AI.

AI is affecting work in assessment and accountability more narrowly as well. A recent call for a special issue of the Journal of Educational Measurement mentions the role of ChatGPT and “ultra large” numbers of automatically generated items that leverage certain AI capabilities. It builds on a past special issue of considerations around validity and similar virtual convenings. 

To accompany this blog post, we have created a table in which we walk through 25 implications from 10 areas of work that affect the way we design and implement assessment and accountability systems. For each, we call out illustrative opportunities and risks.

• Competency selection
• Competency development
• Task development
• Test assembly and delivery
• Reporting and data interactions
• Data engineering
• Scoring design
• Validation approaches
• Accountability approaches
• Ethical considerations and data privacy

Many of the opportunities we describe represent current-day pushes for educational innovations that have been argued for before but are inspired this time by advances in AI rather than by other kinds of policy reforms or scientific developments. 

Most of the risks in this work are likely to stem from insufficient oversight, quality control, or expertise, or over-reliance on existing frameworks and methods that are poorly matched to emerging problems. These are also well-known issues affecting various areas of work in assessment and accountability when stakeholders desire innovation.

Importantly, the ways these issues will be worked through in a given context are affected not just by rational, objective, and scientific considerations. Rather, they will be driven very strongly by morals, values, and principles held by professionals who shape, either directly or indirectly, key aspects of strategic priorities for the adoption of curricula, teacher training, assessment systems, accountability systems, and state policies. 

AI and Measurement: Embracing Opportunity, Minimizing Risk

So what should professionals working in areas related to assessment and accountability do? The critical question is, as always, how to maximize opportunities and minimize risks while the issues continue to evolve at a rapid pace. 

The worst thing we can do is to be afraid and revert back to dated approaches, such as testing with paper-and-pencil or in-person only, just because they seem to be “safe” from advances that we do not yet understand. 

We do not pretend to have definitive answers at this moment in time. Still, many issues are extensions of existing challenges for which systematic frameworks, standards, and professional practices exist that can be adapted or built upon.

However, our profession will need to update its best-practice standards much more frequently if it wants to remain relevant. A once-in-a-decade cycle, as reflected in the Joint Standards, will likely be insufficient. A good example of an updated set of standards is the 2022 release of the ITC Guidelines for Technology-Based Assessment. 

Testing professionals will also need to stay abreast of developments in technology-supported instruction and associated standards since innovative learning activities in classrooms will drive perceptions of relevance and utility for assessment and accountability systems. 

In some areas and domains, especially those closely associated with postsecondary experiences and pathways, the consideration and utilization of AI will be unavoidable. Courses, apprenticeships, and self-directed virtual learning opportunities will be shaped by advances in AI as professions evolve. 

In turn, these experiences will set the expectations for what powerful assessments and accountability mechanisms should look like and be able to do. It is up to our profession to translate and adapt standards and best practices accordingly if it is to remain relevant and play a powerful, constructive role in the ways AI shapes assessment and accountability.