Claims and Evidence for Through Year Assessments: What We Know and What We Need to Know

November 15-16, 2021
Online

Agenda-at-a-Glance

This Center for Assessment virtual convening will occur over four virtual sessions on November 15th and 16th, 2021. Attendance is free, but pre-registration is required (please click on the session titles below to register for each session). All sessions will be facilitated by Center for Assessment experts and will feature perspectives from leading professionals at state and district departments of education, assessment companies, and researchers.

        Date

      Time                (EST)

     Session Title &               Registration

    Nov. 15

   1-2:30PM

Definitions, Aims, and Use-Cases

    Nov. 15

   3-5PM

Claims, Designs, and Evidence

    Nov. 16

   1-2:30PM

Technical and Logistical Issues

    Nov. 16

   3-5PM

Threading the Needle

More information about the conceptual foundations of the convening and details about the agenda follow. 

Agenda At-a-Glance

What is a Through Year Assessment?

Nathan Dadey and Brian Gong (in press) defined through year assessment as follows:

  1. Administered through multiple, distinct administrations across a school year, and
  2. Meant to support both (a) the production and use of a summative determination, and (b) one additional goal.

In other words, a through year assessment system involves a distributed design that is meant to support some specific goal(s), in addition to the creation of a summative determination, as required by current federal law. We must understand the goal(s) behind a through year system in order to fairly understand and evaluate the design. We have observed several common goals expressed by state and district leaders for pursuing through year assessment.

  1. The summative assessment can be spread out so the end-of-year test may not have to be as long as it would be if everything was assessed at once. 
  2. State leaders hope to provide “instructionally useful” information to educators and students throughout the year to enhance the value proposition of state tests beyond accountability. This could include flexible administration of system components to better align with local scope and sequences.
  3. State leaders are hoping to build more coherent systems compared with the current constellation of a wide variety of interim assessments used in most states. Through year systems represent an attempt, in a “loose-coupling” sense, to help districts create more balanced systems of assessments than is currently the case.

Dadey and Gong suggest these common goals fall into three broad categories - logistical, instructional, and administrative - and rarely is there a single goal held by all stakeholders. A successful through year assessment system design involves determining what goals are prioritized and acknowledging that not all goals can be met within a single system.

What is a Through Year Assessment?

Current Examples of Through Year Assessment Systems

Many might picture three or four assessments per year as the prototypical through year design, such as those that might follow an assessment schedule of typical commercial interim assessments. However there are many variations including approaches that rely on almost weekly tests as part of a diagnostic classification modeling (DCM) system, “delayed multi-stage adaptive” systems, curriculum-embedded assessments, and progress to proficiency models. No doubt there will be additional designs emerging in the near future. 

Current Examples of Through Year Assessments

The Convening

We think our understanding of this rapidly evolving topic will benefit from presentations and facilitated conversations among diverse stakeholders, including, but not limited to, state assessment and policy leaders, commercial assessment providers, academic researchers, district leaders, school leaders, and teachers. Our goals for the convening are to:

  • Develop a shared understanding of common terminology and other conceptual underpinnings of the through year model, 
  • Identify and clarify the specific claims associated with various through year assessment designs, and
  • Determine the research and evaluation questions that should be addressed to provide evidence related to the claims associated with specific through year designs, including carefully attending to intended positive and unintended negative consequences associated with the system. 

We intend to follow up the convening with tools, syntheses, and other documents to support state leaders and assessment providers as they wrestle with if or how to move forward on through year assessment design, implementation, and validation.

The convening will occur over four virtual sessions on November 15th and 16th, 2021. Attendance is free, but pre-registration is required. All sessions will be facilitated by Center for Assessment experts and will feature perspectives from leading professionals at state and district departments of education, assessment companies, and researchers. An initial agenda is below; more details will follow in early September.

Convening Details

Agenda

Monday, November 15th
1-2:30PM
Definitions, Aims, and Use-Cases

Definitions, Aims, and Use Cases

 

 

 

 

 

 

 

 

 

 

 

Given the considerable variety of designs using the terms “through course,” “through year,” “distributed assessment,” or others, we start the convening with identifying the goals behind many through year proposals. In other words, we want to understand what problems/issues education leaders and assessment companies are trying to address. This session also involves providing grounding definitions and key characteristics of the various terms so that we can be working from a common language for the rest of the meeting.

To register for this session, Click here.

3-5PM
Claims, Designs, and Evidence

Claims, Designs, and Evidence

 

 

 

 

 

 

 

 

 

 

 

Claims are the foundation for design and validation. This session will include representatives from states and vendors to address some or all of the following: 

  1. Recapping problems states are trying to solve.
  2. What are the key features of the particular design in relation to the specific claims?
    • What is the rationale for various designs?
    • How are these designs intended to address the stated problems?
    • What assumptions do the various designs make about how students learn such as described by the cognition vertex of the assessment triangle (NRC, 2001). Additionally, what assumptions do the various models hold about how learning grows and develops through the year?
    • How are the various designs intended to interact with curriculum and instruction? For example, does the through year model assume that certain topics will have been taught in certain ways by specific points in the year? 
  3. What are the main summative claims being made for the through course system? What are additional claims other than summative claims?
    • How do these claims overlap with claims from a more standard end of year assessment system?
    • What are the claims being made for the scores from the “through year” components?
    • What are the claims being made at the end of the year in a through year system?
    • What is the theoretical and/or empirical evidence in support of the various claims?

To register for this session, click here.

Tuesday, November 16th
1-2:30PM
Technical and Logistical Issues

Technical and Logistical Issues

 

 

 

 

 

 

 

 

 

 

 

Technical and logistical issues associated with through year designs are not for the faint of heart. What sounds like a simple decision—computing a “final score” or “annual determination”—involves many tradeoffs. Several state and assessment company researchers will be asked to address the following big picture technical issues:

  1. Scoring and creating an annual summative score (determination),
  2. Creating an interpretation and use argument (IUA) for through year models,
  3. Addressing such issues as establishing the sequence and timing of administrations, dealing with missing data, balancing test security concerns, protecting student privacy, determining the degree of standardization required for each event, and ensuring accessibility/ accommodations are provided.

To register for this session, click here.

3-5PM
Threading the Needle

Threading the Needle

 

 

 

 

 

 

 

 

 

 

 

The final session will involve turning the tables. Center for Assessment professionals and some external experts will be invited to do short presentations on the following questions:

  1. Can convening participants envision a through year design (broadly conceived) that can meet both accountability (e.g., ESSA) and instructional claims? What are some promising design ideas that might thread this needle?
  2. What is the best way to advise states and other clients purportedly interested in pursuing a through year design? Can we make this work? Do we have to temper our claims?
  3. What is the level of ongoing tailoring and development that needs to be done?
  4. What are any additional assessment literacy needs to be associated with through year designs compared to the normal challenges with increasing assessment literacy among key stakeholders? 
  5. If not through year designs, what are some other possible designs that might be able to meet some of the stated goals discussed in earlier sessions?

To register for this session, click here.