Making Validity Work in Practice

A clearer process for identifying, evaluating, and acting on validity evidence

This blog is based on ideas presented in the session From Validity Arguments to Validity Synthesis and Judgments at the 2026 annual meeting of the National Council on Measurement in Education (NCME). A forthcoming paper by Derek Briggs, Chris Domaleski and Mike Russell will provide a fuller description of the framework.

We spend a lot of time in educational assessment talking about validity, but considerably less time figuring out how to make validation work in practice. That practical application is important, though. Validity—the degree to which evidence supports the intended interpretations and uses of an assessment—is the cornerstone of sound assessment practice.

We haven’t overlooked it for a lack of scholarship. The Standards for Educational and Psychological Testing provide a well-established foundation, and argument-based approaches to validation have substantially influenced how the field thinks about the evidence needed to support test score interpretations and uses. These contributions are indispensable to contemporary validity theory.

In practice, however, validity frameworks can be dense and difficult to translate into action. Assessment programs may assemble an impressive collection of technical studies and still struggle to answer some basic questions: What are the most important claims we need to support? What evidence matters most? Is it sufficient? Where are the gaps? And what should we do next?

Just as important, who is responsible for answering these questions? Validation requires more than collecting evidence. Someone needs to identify the key questions, determine what evidence is needed, synthesize what that evidence tells us, and ultimately reach a judgment about whether it is sufficient.

Too often, neither the process nor the responsibilities are clearly defined. Validity instead gets relegated to a dense technical manual that few people read, much less use to guide improvement. 

A More Practical Approach

Over the past couple of years, I have worked with Derek Briggs and Mike Russell to explore a more practical approach to this problem. The effort grew out of our service on the Technical Advisory Committee (TAC) for the Multi-State Alternate Assessment (MSAA) and reflects Derek and Mike’s thought leadership, as well as the collaboration of the many partners who lead and support the MSAA program.

The basic idea is straightforward: Start with the most important claim an assessment program wants to make about its scores, identify what must be true for that claim to be credible, assemble the evidence, and then actually make a judgment about whether the evidence is sufficient. Once evidence for the primary claim is deemed to be sufficient, a program is better positioned to repeat the process for additional claims.

For MSAA, the primary claim was that its scores provide trustworthy information about the grade-level knowledge and skills of students with significant cognitive disabilities. We then organized the assumptions necessary to support that claim into five categories listed below.  

1. Construct and Design: The assessment contains high-quality items aligned to the intended construct and appropriately targeted to students.
2. Fairness: The assessment minimizes factors unrelated to the knowledge and skills it is intended to measure.
3. Administration and Scoring: The assessment is administered and scored as intended so students can accurately demonstrate what they know and can do.
4. Precision and Generalization: Assessment scores are sufficiently reliable and comparable to support the intended interpretations and uses.
5. Score Reporting: Assessment results are based on appropriate performance standards and reported clearly and in a timely manner.

These categories are supported by individual assumptions as shown in the figure below. 

This framework is not intended to introduce a new theory of validity. In fact, an important feature of the approach is its consistency with established validity principles and professional standards, including federal peer review. The goal was much more practical: to make the questions that matter explicit and manageable. In developing the framework, we challenged ourselves to avoid unnecessary jargon and use clear, accessible language so that the approach would be understandable and useful to a broad range of constituents. 

Equally as Important: Define the Process

An essential feature of the MSAA work was that we explicitly addressed who does what.

We developed a six-step process:

1. Identify the primary score interpretation and intended uses.
2. Identify the assumptions that must hold for those claims to be supported.
3. Plan the validity synthesis, including the evidence needed for each assumption.
4. Have the assessment contractor assemble and synthesize the evidence.
5. Have independent evaluators judge the sufficiency of that evidence.
6. Have the TAC review the credibility of the process and results and recommend priorities for improvement. 

MSAA leaders, Cognia (the assessment contractor), and the TAC worked together to implement this process during 2025. My colleague Cara Laitusis and I served as independent evaluators, reviewing the evidence for each assumption and rating the degree of support from insufficient to strong. We then presented our findings to the TAC, who provided an additional layer of review and judgment to help reach an overall conclusion and identify key next steps. The findings were documented in a memorandum to the MSAA partners. 

The result was far more useful than another collection of evidence in a technical report. We had a synthesis that addressed the more important questions: Taken together, what does the evidence tell us? And where do we need to improve?

That last question is especially important. Rather than concluding vaguely that “more validity evidence is needed,” the process helped identify where attention was most needed and why. It gave the states, the contractor, and TAC a shared understanding of the strengths and gaps in the evidence and a clearer basis for setting priorities and working together to strengthen the program.

From Documentation to Improvement

Validity should not be an inscrutable academic exercise, or relegated to a chapter in a technical manual that few people will ever read. It should be a disciplined process for asking the most important questions about an assessment, examining the evidence, making reasoned judgments, and identifying what needs to improve.

That is ultimately what I find most promising about the MSAA experience. It changed the conversation from “Do we have validity evidence?” to “What does the evidence actually tell us, and what should we do about it?”

If we can make that shift more broadly, validity can become not just something assessment programs document, but something they actively use to get better.