Considering Coherence in the Collection and Use of OTL Data

Ensuring Equitable Learning Opportunities for all Students Requires Coordination Across the Education System

This is the second post by one of our 2021 summer interns based on their project and the assessment and accountability issues they are addressing this summer with their Center mentors. Dan Silver, from the University of Southern California, is working with Scott Marion on using opportunity-to-learn data (OTL data) to enhance the utility of state test scores and better understand schooling conditions during and after the pandemic.

The COVID-19 pandemic has exacerbated existing inequities in the distribution of learning opportunities and amplified the need for coherence in the collection and use of opportunity-to-learn (OTL) data. Many have argued that OTL data adds important context to assessment results, fills gaps where achievement data are missing, and supports a more complete picture of the functioning of schools and the systems in which they operate than achievement data alone can.

The recent influx of federal education spending as part of the American Rescue Plan provides states and districts with the means to direct additional funding to schools and classrooms where the negative effects of the pandemic have been felt most acutely. OTL information can help guide some of these decisions this year and next. However, we maintain that measuring OTL should not just occur in times of crises; it is essential in “normal” times as well. Therefore, we created a framework to guide conceptualization, measurement, action, and communication of an OTL indicator data collection and reporting system.

A Coherent OTL Framework

This framework extends the work of the National Academies of Science, Engineering, and Medicine (2019) on Monitoring Educational Equity in two key ways: 

• It was designed with an explicit social justice orientation.
• It acknowledges the central role that coherence must play in meaningful, justice-oriented reform. 

My colleague, Thao Vo, recently described what it means for a framework to adopt a social justice orientation. I describe below how we operationalize coherence and provide examples to explain why coherence must be a central part of the OTL framework.

The literature on balanced assessment systems offers powerful arguments that assessment systems function best when they are coherent, such that “the entire system is working toward a common set of learning goals.” When all aspects of a system are working toward agreed-upon goals, the achievement data produced in an assessment system are likely to provide useful information for actors across the system. 

Similarly, we are concerned that OTL data collection and use should facilitate productive action – but doing so requires coordination across the education system. Specifically, OTL data should be collected and acted on coherently across:

• system levels (e.g., state, district, school), 
• units within the same system level (e.g., across schools within a district), 
• developmental levels (e.g., early childhood, elementary, middle, high), and
• time (data should be collected regularly enough to facilitate continuous improvement and evaluation, although it is not necessarily the case that identical data needs to be collected each year; see Figure 1). 

These dimensions of coherence correspond roughly to the concepts of vertical coherence, horizontal coherence, developmental coherence, and continuity, respectively, as described in the assessment systems literature.

Acting Coherently

The framework includes indicators and data collection opportunities at the state, district, school, classroom, and student/family levels. Actors at different levels of the education system have different policy and practical tools available to them. This approach allows for each level of the education system to address the problems it is best set up to solve, resulting in coordinated action across system levels. For example, evidence of extreme between-district racial segregation could not easily be acted on at the school level, but state policymakers could enact a housing policy to reduce that segregation. Conversely, the state could not easily act on the information, for example, that a student, Cesar J., is exhausted in school because he needs to care for his siblings at home, but trusted teachers and school leaders could recommend appropriate community-based wraparound services to help with childcare for Cesar’s siblings.

In some cases, different levels of the system have such different strengths that we recommend collecting and analyzing entirely different streams of data. In other cases, collecting and analyzing data on similar indicators at multiple levels of the education system can lend important perspective to those data. 

• Data on the availability of advanced high school courses is readily collected at the district level from master schedules, and district-level actors have the ability to build capacity in schools to improve the advanced course offerings. 
• Data on exactly who is enrolled in these advanced courses might be better acted on at multiple system levels. District leaders could reexamine tracking policies that make advanced course taking more difficult for some students, while elementary and middle school leaders could encourage and build capacity among their teachers to prepare all students to excel in advanced high school coursework. 
• Finally, students could report on their perceptions regarding encouragement/ discouragement from participating in higher level and/or advanced course work. 

This type of coherent approach across developmental levels can support more productive action to bridge opportunity gaps. For example, classroom-level OTL surveys might indicate that certain schools had a culture among staff that discouraged students from pursuing advanced coursework. School and district administrators could work to build a culture in such schools where taking advanced classes is seen as a worthy pursuit, to encourage and support historically underserved students to take rigorous courses. Because this reform only would be enacted in schools where climate data indicated it was needed, this approach represents a response to OTL data that is coherent across units within the same system level: only those schools that need this support receive it. 

Regardless of the exact support rendered, we recommend that users of our framework plan collect similar data at regular intervals, allowing for continuous improvement and constituting a response to OTL data coherent across time.

A Call to Action

Leaders at multiple levels of the education system must collect and act on OTL data since policies and norms from the state level down to the classroom level and from high school down to kindergarten all have profound power to bolster or hinder K-12 students’ OTL. We argue that collecting and acting on OTL data in ways that are coherent across system levels, units within system levels, developmental levels, and across time provides the likeliest route to improving educational conditions for underserved students and their families. Long after the COVID-19 crisis is over, this outcome will remain an essential goal.