Name of Partners: SRI International (SRI), Montana Office of Public Instruction (MT-OPI)

Purpose: In this project, the researchers employed a continuous improvement process to support the literacy skills of middle and high school students. Although literacy programs such as the Montana Striving Readers Program (MSRP) had made progress toward improving the literacy skills of elementary and middle school students, the MT-OPI remained concerned regarding the large numbers of Montana economically disadvantaged students and American Indian students entering secondary schools with insufficient reading skills. This project sought to intervene with students at the beginning of middle and high school so that they could build the literacy skills needed to either successfully transition to high school or complete high school.

Project Activities: The partnership was a collaboration between SRI researchers, MT-OPI instructional leaders, and educators in two middle schools and two high schools in Montana. MT-OPI nominated these schools for participation because they had a large percentage of economically disadvantaged students and/or American Indian students who did not meet state literacy standards. Within each participating school, a team of 8–12school administrators, instructional coaches, reading intervention teachers, and general education teachers collaborated with staff from SRI and MT-OPI on a continuous improvement process to design and implement a set of quick-turnaround, tiered literacy interventions using a Plan Do Study Act (PDSA) process. SRI and MT-OPI staff delivered training, coaching, and resources to support the school teams. Project activities focused on building the capacity of partner middle and high schools to adapt literacy interventions, implement them consistently, and use data to inform continuous improvement.

These school-level PDSA teams worked with SRI and MT-OPI to design and implement six PDSA cycles across the 4-year project. During PDSA cycle 1, each school identified a specific change idea using the school's instructional framework. PDSA strategies included implementing formative assessments in each class period; focusing a vocabulary strategy on prefixes and suffixes; and using ability groups during in-class writing activities. PDSA cycles 2 and 3 focused on the use of Course and Unit Organizer Routines, the Strategic Instruction Model, the Question Exploration Routine, and the Advancement Via Individual Determination strategy. During PDSA cycle 4, school teams implemented school-specific instructional strategies for supporting vocabulary skill development. PDSA cycle 5 then focused on the simultaneous integration of these vocabulary strategies with the strategies from PDSA cycles 2 and 3. PDSA 6 took place over 12 weeks and involved school-specific strategies such as one-sentence summaries, growth mindsets, checks for understanding, and weekly student-teacher conferences.

In the final year of the project, the SRI and MT-OPI teams collaborated on the development and dissemination of an MT PDSA Implementation Guidebook. This guidebook focused on providing lessons learned for implementing improvement science practices in Montana schools. The guidebook describes how the focal schools used improvement science methods, what they were attempting to accomplish, and what changes they implemented as part of the project. The guidebook also includes information on issues teams identified as important for success and points to resources others can use when implementing the PDSA process.

Key Findings: The team examined the relationship between participation in the PDSA intervention and the literacy skills of students and instructional practices of teachers in participating schools. The analyses were conducted using multi-level linear regression analyses and compared participating schools to a matched set of non-participating schools that were similar on observable characteristics. According to the report published by the project team (Grindal et al., 2020), the research team  did not find evidence of a statistically significant association between a school's participation in the intervention and students' literacy skills.

Structured Abstract

Setting: This project took place at four Montana middle and high schools in three rural settings located across the state (North, Central, and Southeast).

Sample: Study participants included students and teachers located in four middle and high schools. Each of the schools had large percentages of struggling readers, American Indian students, English learners, and low-income students. Comparison schools were selected from the 11 other Montana middle and high schools that had participated in the Montana Striving Readers Program, a previous literacy focused school improvement initiative.

Approach: The study team worked with teams of 8–12school administrators, instructional coaches, reading intervention teachers, and general education teachers to implement a set of quick turnaround interventions using a Plan Do Study Act (PDSA) process. These PDSA teams worked with SRI and MT-OPI to design and implement two 6-week PDSA cycles in each school year. During each PDSA cycle, team members implemented small alterations in their school practices or processes, referred to as "change ideas," which were designed to advance student literacy skills and focused on students who struggled with reading. PDSA team members were encouraged to select change ideas that would support struggling readers in particular, with the idea that these supports would help all learners access core classroom instruction and content. Teams met weekly to review data on relevant teacher practices and make changes to implementation. At the end of the cycle, the teams reviewed the data on student outcomes and developed a plan for the next PDSA cycle. School teams engaged in 6 PDSA cycles across the 4-year project.

Research Design and Methods: The study team used propensity scores to test the effect of participation in the PDSA cycles on student literacy outcomes. The propensity score is the predicted probability of participating in a treatment based on a set of potentially confounding covariates (i.e., student demographic and disability characteristics, baseline score) using logistic regression. Propensity score analysis attempts to equalize the mean values of potentially confounding observed covariates in the treatment and comparison groups, assuring that differences in outcomes between the treatment and covariate effect are not the result of differences in mean values of those covariates.

Control Condition: Comparison schools were selected from the 11 other Montana middle and high schools that had participated in the Montana Striving Readers Program, a previous literacy focused school improvement initiative. Students in the comparison schools received literacy instruction through the Montana Striving Readers Program, without the additional supports and strategies provided through the partnership's PDSA approach.

Key Measures: Student outcome data from the treatment schools and comparison schools were used to answer research questions regarding differences in student literacy/reading achievement. Data on the overall reading ability of middle school students were collected via fall and spring Istation Indicators of Progress Advanced Reading assessments as well as the spring administration of the Smarter Balanced Assessment Consortium English language arts assessment. The study team used data from the ACT Reading assessment conducted in the spring of grades 10 and 11 to measure the reading skills of high school students.

Data Analytic Strategy: Researchers used a two-level model with students at level one and classrooms at level two to compare outcomes in literacy and achievement of students in treatment and comparison schools. To match improvement and comparison schools, the study used nearest neighbor criteria based on school and average student characteristics, and propensity score matching to arrive at a comparison group of students from the comparison schools that was matched to students in the four improvement schools.

Products and Publications

ERIC Citations: Find available citations in ERIC for this award here.

Reports:

Grindal, T., Nunn, S., Wei, X., Boyce, J., & Chow, K. (2020). The Implementation and Effects of a Multi-Year Continuous Improvement Research Project in Rural Secondary Schools: Montana Partnership Comparison Study 2.