2019 Reidy Interactive Lecture Series

Improving the Selection, Use and Evaluation of Interim Assessments

 

September 26th – 27th, 2019
Portsmouth Harbor Events and Conference Center
100 Deer Street, Portsmouth, NH 03801

 

Agenda

Interim Toolkit Phase 1

Interim Toolkit Phase 2

Conference Overview

State and local education leaders are constantly working to develop assessment systems that provide schools and educators with an accurate, comprehensive profile of student performance. In most cases these systems include a combination of state summative assessment administered at the end of the year and locally developed or selected formative assessment tools used during instruction. These assessments are intended to work together in a coherent, strategic manner to inform instruction and improve educational decision making.

Some states, alone as a part of a consortium, have tried to develop assessments to fill the information gap between state-developed summative assessments of learning and classroom-directed formative assessment for learning, but several districts and states have turned to the multitude of commercially-developed off-the-shelf products broadly referred to as interim assessments. While many of these tools are thoughtfully designed to provide on-demand, user-friendly information about student and aggregate performance they are only useful to the extent that they meet a district’s specific assessment needs. Different tools are designed to support different purposes and uses, and no tool will support all uses equally well. For this reason, state- and commercially-developed interim assessments must be selected and evaluated in light of a prioritized set of goals and a clear vision of the role the assessment is intended to play within the existing assessment system.

The goal of RILS 2019 is to help states and districts make better informed decisions about the appropriateness and utility of different interim assessments options. During the day and a half meeting we will hear from assessment experts, state and district representatives, and practitioners with experience selecting, using and evaluating interim assessments. Conference participants will also work in small groups to review and extend a tool developed specifically to help state and district leaders engage in a thoughtful, comprehensive process focused on identifying assessments that will meet their specific needs.

Through presentations, invited panels and small work groups we will address a series of questions that are foundational to the selection, use, and evaluation of interim assessments:

  • What factors should a state or district consider prior to considering different interim assessment options?
  • What questions should be addressed to clarify the desired characteristics and features of an interim assessment solution (e.g., content representation, frequency and timing of administration, reporting details)?
  • How does the intended use of an interim assessment influence decisions about necessary design, administration and reporting characteristics?
  • What type of evidence should be prioritized when evaluating the technical quality of an interim assessment for a particular purpose and use?
  • How do you evaluate whether an interim assessment, once in place, is being used as intended and supporting the attainment of the goals for which it was selected?

Through these discussions we will determine what, if any modifications or additional resources are necessary to ensure the tool will be clear and useful when posted for broader distribution.

For an overview of the structure of RILS 2019 please review the draft agenda.

For additional information on improving the selection, use, and evaluation of interim assessments you may visit our recent blog post.

Agenda

#RILS2019 @NCIEA1

Thursday, September 26th

7:30 AM

Registration and Breakfast

8:30 AM

Welcome to RILS
The Center for Assessment will welcome participants to the 21st annual Reidy Interactive Lecture Series and provide a brief overview of the conference.

PRESENTER: Scott Marion (Center for Assessment)

8:45 AM

Setting the Stage
During this session, the Center will describe the landscape of interim assessments, provide a working definition of interim assessments for the conference, describe some common barriers to selecting and evaluating interim assessments, and describe the primary outcomes for RILS 2019.

PRESENTERS: Nathan Dadey (Center for Assessment), Juan D'Brot (Center for Assessment), Erika Landl (Center for Assessment), Scott Marion (Center for Assessment)

PRESENTATIONS:
Interim Lay of the Land: Preliminary Findings from Two High Level Scans

RESOURCES:
Designing Balanced Assessment Systems (Chatergoon and Marion, 2016)
A Tricky Balance (Marion, et. al, 2019)
Interim Assessment Use (Dadey and Diggs, 2019)
Scan of Interim Assessments Spreadsheet (Dadey and Diggs, 2019)
Defining Interim Assessment (Perie, Marion, Gong, 2007)
Defining Interim Assessment (Perie, Marion, Gong, 2009)

9:45 AM

Purposes of Interim Assessments: Small Group Feedback and Discussion
Working in small groups, participants will discuss issues faced by states, districts, and schools related to how interim assessments are selected and used. The purpose of this discussion is to identify common challenges associated with selecting assessments (e.g., coherence), identifying desired assessment characteristics, and evaluating the impact and utility of interim assessment results.

PRESENTERS: Juan D'Brot (Center for Assessment), Erika Landl (Center for Assessment)

PRESENTATION:
Morning_Day1_Slides

10:45 AM

Break

11:00 AM

State and District Case Study Discussion
During this session state and district leaders from Wyoming will share their perspectives related to the role and purpose of interim assessments and discuss what they see as the greatest challenges that need to be addressed in order to ensure they add value. An expert discussant will provide comments and reactions.

PRESENTERS: Leslie Keng (Center for Assessment), Laurie Hernandez & Jessica Steinbrenner (Wyoming Department of Education), Sonya Tysdal (Weston County School District 1, WY), Scott Marion (Center for Assessment)

PRESENTATIONS:
WY_Intro_Slides
WY_State&District_Slides

12:00 PM

Lunch

1:00 PM

The Importance of Identifying Assessment Gaps and Needs
During this brief session, the Center will highlight the need for a thoughtful process prior to making important decisions about the need for an interim assessment and its intended role, if any, in an existing assessment system. We will discuss the first phase of a proposed process, Identifying Gaps and Needs, which requires education leaders to articulate their vision for teaching and learning and their assessment theory of action prior to conducting an assessment search, review, or evaluation.

PRESENTER: Juan D'Brot (Center for Assessment)

PRESENTATION:
Gaps&Needs_Intro_Slides

1:15 PM

District Perspective: School Administrative Unit #9, New Hampshire
This session will highlight how a district considers the design and selection of interim assessments based on its unique context and identified information needs. The conversation will address the district’s vision, how they intend to use assessment information, and how they specified high priority assessment selection criteria.

PRESENTERS: Carla Evans (Center for Assessment), Kadie Wilson (School Administrative Unit 9, NH)

1:45 PM

Discussion and Refinement of Phase 1 of the Interim Assessment Identification and Evaluation Process: Identifying Assessment Gaps and Needs
During this session we will break into groups to explore a sample district profile that illustrates the types of information a district might compile based on responses to Phase 1 of the Interim Assessment Identification and Evaluation Process. After reviewing the profile, groups will be asked to review the questions and structure of the Phase 1 tool to determine whether there is sufficient guidance and scaffolding for someone to respond to each set of questions focused on identifying assessment gaps and needs. Each group will be partnered with 1-2 Center staff who will help facilitate the discussion and take notes. An electronic template will be established so that feedback can be captured during the discussion and shared with the large group tomorrow.

PRESENTERS: Juan D'Brot (Center for Assessment), Erika Landl (Center for Assessment)

3:00 PM

Break

3:15 PM

Introduction to Phase 2: Identifying and Prioritizing Assessment Characteristics & Evidence of Assessment Quality
During this brief session, the Center will discuss Phase 2 of the process, introduce the Assessment Characteristics Tool and discuss how it is intended to be used to support assessment selection and evaluation. The intent of this tool is to help state and district leaders clarify and articulate the specific design, administration and reporting characteristics an interim assessment must demonstrate to meet the district’s information needs in a manner that is consistent with its vision for teaching and learning.

PRESENTER: Erika Landl (Center for Assessment)

PRESENTATION:
Assess_Char_Intro_Slides

3:35 PM

Discussion and Refinement of the Assessment Characteristics Tool
During this session, participants will again break into groups. Working at tables, participants will be asked to use the same scenarios considered during the previous activity and the Phase 2 tool to determine the primary characteristics and features that an assessment should demonstrate to meet the district’s goals. Each table will be asked to develop a broad profile of the type of assessment they are looking for and provide their thoughts regarding resources, guidance and business rules that would need to be developed or provided to support implementation in schools. Each group will be partnered with 1-2 Center staff who will help facilitate the discussion and take notes. An electronic template will be established so that feedback can be captured during the discussion and shared with the large group tomorrow.

PRESENTERS: Juan D'Brot (Center for Assessment), Erika Landl (Center for Assessment)

4:40 PM

Closing Comments and Day 2 Preview

PRESENTERS: Juan D'Brot (Center for Assessment), Erika Landl (Center for Assessment)

4:45 PM

Adjourn for Day 1

5:15 PM

Welcome Reception

Friday, September 27th

7:30 AM

Breakfast

8:30 AM

Welcome Back and a Brief Overview of Evidence for Evaluating Technical Quality
During this session, the Center will welcome the group and discuss the interplay between assessment characteristics and appropriate evidence of technical quality. We also introduce a resource developed to help districts identify and prioritize evidence of assessment quality given information collected with the Assessment Characteristics tool.

PRESENTER: Brian Gong (Center for Assessment)

PRESENTATION:
Gong_Eval_Slides

RESOURCES:
Matching Instructional Uses with Design (Gong, 2019)
Evaluating test validity (Shepard, 2016)

8:45 AM

Ongoing Evaluation – Discussing the Need for a Use and Utility Evaluation Tool
During this session we will make a case for the ongoing evaluation of interim assessments to ensure they are adding value and being used and interpreted as intended. We will discuss the relevance of program evaluation in the context of information use, how evaluative practices could help address implementation and unintended consequences, and the need to collect evidence that goes beyond the evaluation of the assessment itself.

PRESENTERS: Juan D'Brot (Center for Assessment), Erika Landl (Center for Assessment)

PRESENTATION:
Evaluation_Intro_Slides

9:00 AM

State and District Perspectives: Panel and Group Discussion on the Use and Utility of Interim Assessments
This session will address the challenges associated with consequences and use of interim assessment information. Experts, states, and districts will be asked to describe their perspective on evaluating the implementation, utility, and impact of interim assessments and how operational constraints play a role. The Center will conclude this session by describing opportunities for developing a tool or framework that can help practitioners support the implementation of interim assessments.

PRESENTERS: Scott Marion (Center for Assessment), Marianne Perie (Measurement in Practice), Margaret Goertz (Consortium for Policy Research in Education, University of Pennsylvania), Charity Flores & Mary Williams (Indiana Department of Education), Craig Walker (Oklahoma Department of Education), Nick Meyer (Indianapolis Public Schools)

PRESENTATIONS:
Use_Panel_Slides

10:30 AM

Break

10:45 AM

Developing Specifications for a Use and Utility Tool
During this session, participants will break into groups to discuss the highest priority categories that could be used to evaluate the use of interim assessments and their associated information. Participants will be asked to consider their own experiences to help address evaluating the capacity and behaviors of people who are using interim assessment information to help avoid unintended consequences of misuse. The Center will take this feedback to develop a Use and Utility Tool, which will be shared with participants and others freely on our website. Each group will be partnered with 1-2 Center staff who will help facilitate the discussion and take notes. An electronic template will be established so that feedback can be captured during the discussion.

PRESENTERS: Juan D’Brot (Center for Assessment), Erika Landl (Center for Assessment)

PRESENTATIONS:
Eval_Considerations_Slides

11:45 AM

Next Steps and Closing Comments

PRESENTER: Chris Domaleski (Center for Assessment)

12:00 PM

Conclude for Lunch