Keeping up the PACE
Evaluating Grade 8 Student Achievement Outcomes for the NH PACE Program, an Innovative Assessment System
This is the third in a series of posts by our 2020 summer interns and their mentors based on their project and the assessment and accountability issues they addressed this summer. Alexandra Stone, from the Neag School of Education at the University of Connecticut, worked with Carla Evans on analyses to evaluate the effects on student achievement in the NH PACE program.
There is a national interest in innovative assessment systems, including those that rely on performance assessments, to document student achievement on state content standards. However, there is not a lot of research to date on the effects of large-scale K-12 performance assessment programs on student achievement outcomes.
This summer, I worked with Carla Evans to evaluate student outcomes with respect to New Hampshire’s innovative assessment program called PACE (Performance Assessment of Competency Education).
We used student-level data from the first five years of implementation of PACE (the 2014-15 school year through the 2018-19 school year) to compare the performance on the grade 8 New Hampshire state assessments in English language arts (ELA) and mathematics of students in PACE districts to students in non-PACE districts with similar demographics.
We describe our analytic process and overall findings below. We also interpret those findings in light of a priori expectations and possible alternative findings.
Background and Purpose of PACE
New Hampshire’s PACE innovative assessment and accountability system is “grounded in a competency-based educational approach” and “an accountability strategy that offers a reduced level of standardized testing together with locally developed common performance assessments.” The state of New Hampshire was granted authority to administer PACE and its traditional state assessment concurrently in select districts starting in the 2014-15 school year.
The PACE system minimizes the use of the traditional state assessment in ELA and mathematics to only once per grade span. In the other grades where annual assessment is required by federal law, there are three main pieces of information used to produce student proficiency designations:
- Student performance on classroom tests and performance assessments unique to each district;
- Student performance on common performance assessments across all PACE districts; and
- Teacher judgments of student achievement.
According to the PACE theory of action, deeper learning should occur in districts that implement PACE due to changes in instruction and assessment (Marion & Leather, 2015). The implementation of the innovative assessment system is intended to support ambitious teaching practices, a deepening of the level of content that students are taught, and student demonstration of complex thinking and reasoning skills.
One criterion for the success of the PACE innovative system is more performance-based instruction and assessment practices alongside no appreciable differences with respect to student performance on the state standards as measured by the state assessment. Previous research conducted by Evans (2019) found there were relatively small positive effects on state achievement tests in ELA and mathematics for students in grades 8 and 11 who were in PACE districts for the first three years of implementation compared to those who attended districts that did not implement PACE.
Purpose & Rationale of Current Study
Now that the PACE innovative assessment system has been implemented for five years and includes additional districts beyond the initial early adopters of competency-based education, it is important to evaluate the extent to which one key assumption of the program still holds—that there is no appreciable difference in performance on the state test between students who attend PACE districts and those who do not. Therefore, the current study:
- Examined two additional years of PACE implementation than in the previous study,
- Included the additional districts that implemented PACE over that time, and
- Expanded on the quantitative methodologies applied to examine the program effects of PACE.
The purpose of the research we worked on this summer was to examine the effect of PACE on student performance on the grade 8 New Hampshire state assessments in ELA and mathematics. We used students’ fourth-grade state test scores as a baseline to compare the performance of students in PACE districts to demographically similar students who were not in districts that implemented PACE. It was expected that the implementation of the PACE innovative assessment system did not have any negative effect on students’ statewide standardized test scores. To measure the effect of PACE on student state test performance, we used hierarchical linear modeling to account for students being nested within the districts.
Findings & Implications for Future Innovative Assessment Programs
Findings from our study suggest that there is no difference in grade 8 ELA or mathematics outcomes between PACE students and non-PACE comparison students during the first five years of implementation.
Additionally, there is little to no evidence to suggest that there is any individual PACE district or any single school year driving positive or negative effects. Also, subgroup analyses did not reveal any significant difference in PACE effects based on gender, student with disability status, or free/reduced-price lunch status.
Though findings of no significant differences between groups are often ignored or not reported at all, we believe it is important for the field to have additional research accumulate on the effects of large-scale innovative performance assessment programs on student achievement outcomes.
Such results can help bolster claims that the positive benefits of performance-based instruction and assessment outweigh the risks involved in not testing every student, every year. Studies such as this could even support other innovative performance assessment system designs and pilot programs that require the state achievement test to be administered only once per grade span rather than every year without concerns that such a reduction in statewide testing leads to or promotes inequitable opportunities to learn.