RILS Conferences

RILS Conferences | Past Conferences

13th Program: Edward F. Reidy, Jr.
Interactive Lecture Series

Using Multiple Measures to Enhance Validity and Reliability: Applications to school accountability, teacher evaluation, and instructional improvement

Sponsored by
Center for Assessment

September 22-23, 2011
Location: Omni Parker House
Boston, MA

Conference Overview

The call for “multiple measures” in the design of assessment and accountability systems has taken on a mantra‐like refrain. Universally endorsed, federally mandated, scarcely used, and widely interpreted, multiple measures and their appropriate use are at the heart
of design decisions for any assessment or accountability system and will be the topic of the 2011 Reidy Interactive Lecture Series (RILS).
Teachers have been using multiple measures to award grades forever and measurement researchers have been dealing with this issue ever since the creation of the first composite score. Many technical efforts have focused on maximizing the reliability of the combined score, but there has been much less attention on maximizing the valid use of multiple measures. However, current assessment and accountability initiatives require that we design and implement the most thoughtful approaches possible for using and combining multiple measures
of complex student performance to inform instruction and to make overall judgments of
“college readiness” or “teacher effectiveness” to name a few current examples. Like all RILS,
this conference will focus on bridging the conceptual and practical aspects of this issue
through a focus on the following three broad, but related themes:

» Exploring the conceptual underpinnings of multiple measures by addressing questions
such as what are the various purposes of using multiple measures (validity, reliability,
fairness) and what questions are we able to ask and answer through specific
uses of multiple measures.
» The use of multiple measures in assessment systems to obtain an overall judgment
of student proficiency, including the implications of various approaches to combining
information from multiple item formats, assessment instruments, domains, and levels
of cognitive complexity. Specific approaches to combining information (or not combining
information) from multiple components (e.g., summative, interim, formative)
to best understand students’ performance will be considered.
» The use of multiple measures in accountability systems will focus on the complex
task of expanding measures to develop a more complete picture of complex school
and teacher performance and then compressing those multiple and diverse sources
of information to a single accountability rating or decision.
In keeping with the mission of RILS, these discussions will be engaging and interactive,
enabling participants to come away with concrete guidance about how to approach some of
the complex challenges inherent in the use of multiple measures for assessment and accountability.

Conference Agenda
Thursday, September 22, 2011
7:30		Registration and Breakfast
8:30		Welcome and introduction to the 2011 Reidy Interactive Lecture Series
			Brian Gong, Center for Assessment
9:00-12:00		Session 1: Why Multiple Measures?
	9:00	What evaluation questions are we trying to answer with multiple measures?
			Peg Goertz, CPRE, University of Pennsylvania
10:00		Break
	10:15	What measurement questions are we trying to answer with multiple measures?
			Charles DePascale, Center for Assessment
	11:15	Moderated discussion among speakers and audience members
			Marianne Perie, Center for Assessment
12:00		Lunch
1:00-4:00		Session 2: Implications of Multiple Measures for Assessment Systems
	1:00	Issues of multiple modalities and multiple formats when combining information for English Language Proficiency Assessments
			Gary Cook, WCER, University of Wisconsin
2:00		Break
	2:20	Considerations when combining information from multiple and innovative item formats for inferences about students' understanding
			Kathleen Scalise, University of Oregon
	3:20	Moderated discussion among speakers and audience members
			Paul Nichols, Center for Assessment
4:15		Wrap-up of Day 1 and looking ahead to Day 2
			Scott Marion, Center for Assessment
4:30		Recess until Friday morning
Friday, September 23, 2011
7:30		Breakfast and Breakfast table discussions
			Educator evaluation: Scott Marion & Damian Betebenner
			NCLB Waivers: Marianne Perie & Chris Domaleski
			Thoughts about reauthorization: Brian Gong
			Replacement Units for transitioning to CCSS: Karin Hess & Jeri Thompson
8:30		Report out from table discussions
9:00-12:30		Session 3: Implications of Multiple Measures for Accountability Systems
	9:00	Expanding measures of school outcomes
			Brian Stecher, RAND Corporation
10:00		Break
	10:15	Moderated discussion on multiple measures for accountability purposes (including educator accountability)
			Brian Stecher, RAND Corporation
			Scott Marion, Center for Assessment
			Peg Goertz, CPRE, University of Pennsylvania
			Rich Hill, Center for Assessment
			Facilitated by Chris Domaleski, Center for Assessment
11:30		Closing thoughts and continuing issues
			Brian Gong, Center for Assessment
12:15		Lunch
1:15		Adjourn RILS 2011

Check out our past conferences.