Through-Year Assessment: The Importance of Score Reports

Research on Montana’s New System Explores Users’ Feedback

This is the fourth in a series of posts by our 2024 summer interns, based on the assessment and accountability projects they designed with their Center mentors. Mi Jin Park, a doctoral student at the Graduate Center, City University of New York, worked with the Center’s executive director, Scott Marion.

Montana’s through-year assessment program, the Montana Aligned to Standards Through-Year (MAST), marks a shift in how assessments are approached.MAST’s primary goals are to deliver meaningful data throughout the school year, rather than only at the end, and to provide federally required summative data.The Montana Office of Public Instruction (OPI) has partnered with the Center for Assessment to support the transition.

What is the MAST Program?

Generally, statewide assessments are given at the end of the year, limiting timely feedback. MAST addresses this by aligning assessments with local curriculum and allowing flexible test administration. (Through-year assessment systems are not without trade-offs, however.)

In 2023-2024, Montana piloted MAST in grades 3-8 for math and English/language arts, using two formats: multiple testing windows or one end-of-year window for field-testing and psychometric analyses.

During the pilot, MAST used only item-level score reports. Looking ahead, for the 2024-2025 school year MAST will be administered in grades 3-8 in math and ELA with full student reporting at the testlet and summative levels. Refer to this report for more details on Montana’s framework and transition to the assessment system.

The Critical Role of Score Reports

Score reports are crucial to the MAST program’s success, as education leaders use them to share assessment information with students, parents, teachers, and school and district leaders. MAST’s effectiveness relies on educators accurately interpreting and using these reports to support instruction and student learning. 

For my research, I conducted think-aloud interviews with educators and leaders to document how they interpreted and planned to use score report information. I relied on Perie et al. (2009)’s framework, focusing on the instructional and evaluative purposes of assessment, to design the protocol and analyze the findings. 

The score reports used were early mock-ups, not part of the pilot, and the educators saw them for the first time during the interview, without context for the test content or administration. The operational reports will be available when the first testing window opens this fall, and future research will focus on the interpretation and use cases from educators across the state. 

My interviews provided feedback on the strengths and challenges of the score reporting system. Key insights are summarized below.

How Users Interpreted the Score Reports

A teacher I interviewed interpreted the scale scores on the score reports as scores on individual test questions; others felt they contributed little to instruction in middle school. The school psychologist and counselor I interviewed preferred percentiles over scale scores, finding them more familiar and easier to interpret, especially for comparing student performance. 

Participants also inconsistently interpreted achievement levels; some saw them as grade-level expectations, while others saw them as intervention levels or proficiency levels. It’s important to note that scale scores will no longer be included in testlet reports. 

A couple of participants said that reporting clusters on the ELA score reports was too broad to inform instruction. 

Lastly, the credit-earned section, along with the specific corresponding standards and questions, prompted a school leader and several teachers to plan interventions, such as reteaching. Immediate interventions can also be planned based on score report indications of earned credit and incorrect responses, with students who show no credit or partial credit needing targeted support, particularly in subjects where concepts build on previous material.  

Score Reports for Instructional Support

Score reports can be instrumental in informing instruction, helping teachers refine practices based on student learning strengths and needs. 

Based on my interviews, several participants saw the level of detail in the data provided by the score reports as beneficial for identifying and addressing student needs in a timely manner. 

For example, as one ELA teacher explained, having a classroom-level perspective at the start of the academic year through score reports is more beneficial for identifying and addressing student needs than relying on general statements from previous teachers about students’ abilities. The teacher added that it’s helpful to have classroom-level insights to build an understanding of students’ broad instructional needs as a group before transitioning to individual student-level reports. 

An elementary school teacher and a middle school math teacher noted the potential use of the common “misconceptions” section in their students’ math score reports (shown above in the first image). These misconceptions, along with the corresponding standards, can guide teachers in providing targeted instruction and interventions. Both school leaders and teachers I interviewed mentioned that integrating score report data with other available assessment data can help identify students for different levels of intervention. 

The school leaders reported that detailed item descriptions from score reports can help staff members develop specialized goals for special education students.

Effective use of score reports often requires collaboration among educators. As a school leader described, the reports can also facilitate ongoing discussions between school leaders and teachers about classroom needs and school-wide interventions. Integrating data from various assessments along with the MAST score reports can provide a more comprehensive view of student performance, a school psychologist explained, further informing decision-making processes for Response to Intervention (RTI) and Multi-Tiered System of Supports (MTSS). 

Score Reports for Evaluation

Generally, score reports can be used to evaluate the efficacy of programs, teachers, and instructional strategies to help educators, school administrators, curriculum supervisors, and district policymakers make informed decisions to enhance educational programs. 

Based on my interviews, school leaders mentioned identifying and analyzing potential factors contributing to lower achievement in specific classes using school-level score reports from MAST, which provide classroom-level details. For instance, a current school leader and former teacher noted that looking at why a class has a significant number of students at lower levels can lead to collaborations with teachers regarding classroom data, observations, along with other benchmarking data, ultimately guiding support strategies for both teachers and students.

The district-level score reports that MAST produces provide school-level details. A school leader with a background in special education and student services explained that these reports can identify which schools in a district need support in specific areas aligned with their unique challenges. Comparing performance across schools provides valuable insights for instructional coaches and administrators to target support more effectively. However, the leader added that this can be difficult in a district with only one elementary school and one middle school.

Recommendations for MAST Score Report Enhancement

Based on the feedback from educators, I offer several recommendations that could enhance the MAST score reports:

1.      Common language and clarity:

• Incorporate a common language, as well as clear definitions and explanations, for terms used in score reports (e.g., scale scores, levels, average performance, misconceptions).
• Align this terminology with familiar district benchmark assessments, to the extent it is accurate and makes sense.

2.      Tighter subgroupings within ELA clusters:

• Break down broad content clusters into more specific sub-clusters to provide more meaningful and actionable guidance on subsequent instructional approaches.

The MAST program represents a shift towards more modular assessments aligned with Montana’s content standards. Through its design, it aims to provide timely instructionally useful data to advance continuous improvement in teaching and learning. While there are challenges to address, particularly in how data is interpreted and potentially used, the feedback from educators offers several helpful suggestions for moving forward.