|Title:||An Evaluation of Turning Around North Carolina's Lowest Performing Schools: A State Partnership Evaluation|
|Principal Investigator:||Henry, Gary||Awardee:||University of Delaware|
|Program:||Evaluation of State and Local Education Programs and Policies [Program Details]|
|Award Period:||5 years (8/1/2015-7/31/2020)||Award Amount:||$4,944,014|
Previous Award Number: R305E150017
Co-Principal Investigators: Herman, Rebecca; Marks, Julie; Townsend-Smith, Deanna
Partner Institutions:North Carolina State Board of Education and Department of Public Instruction
Purpose: The research team investigated the implementation and impact of a school turnaround plan for low-performing schools in North Carolina. The plan, called the North Carolina Transformation (NCT) program, identified and targeted increasing each school's leadership and instructional capacities. Rather than the expected positive effects of NCT on student and teacher outcomes, their regression discontinuity study found that student achievement gains were lowered and that teacher turnover increased in the schools. The lowered NCT effectiveness appears to have been related to the timing of implementing the comprehensive needs assessment, reductions in services due to state budget cuts, and a theory of change that omitted several evidence-based practices.
Project Activities: Researchers examined implementation of NCT and subsequent state services to support low-performing schools over a 3-year period and analyzed the impact of NCT during that time. The research team analyzed administrative data to determine the effects of NCT on student, teacher, and principal outcomes. They used teacher and principal surveys and qualitative data collected from site visits along with reports by the coaches assigned to each NCT school to determine the quality and fidelity of implementation of the improvement process in NCT schools. Researchers also explored the underlying processes that mediated or suppressed overall effects and the effects on the lowest achieving students. In addition, the research team engaged in a formal process for partners to communicate and exchange timely feedback about program implementation and effectiveness. Finally, in an effort to increase the capacity of the North Carolina State Board of Education to assess plans and implementation of plan for Continuous School Improvement under the Every Student Succeeds Act, the research team analyzed barriers to improving the lowest performing schools in NCT, including assessing the NCT's theory of change, factors contributing to the instability of school processes and operations, and reactions to the scope and scale of supports received by the schools. The results of these activities were communicated with the State Board of Education through regular briefings, policy and research briefs, and Board planning meetings and have been incorporated into the Board's NC Strategic Dashboard Monitoring Tool. Lessons learned and key takeaways from the research were presented and discussed in three hybrid Town Halls, held in NC during February 2022 and including NCT researchers, other school improvement researchers, and principals who served on the Expert Advisory Committee for the evaluation of NCT.
Setting: Approximately 80 percent of the public schools eligible to participate in NCT were located in rural areas across the state of North Carolina. North Carolina operated two prior rounds of turnaround programs before the mandate to turnaround the new group of low-performing schools evaluated in this study.
Sample: The population eligible for NCT consisted of students, teachers, and leaders in schools that did not meet the state's performance standards. Seventy-eight of the state's lowest performing schools based on the 2015 proficiency rate were assigned to receive treatment under NCT. In year 1 of NCT implementation, the analytic sample included 87 of the 331 eligible schools; 50,731 of the 195,437 students; and 2,658 of the 10,770 teachers due to optimal bandwidth parameters. Sample sizes for the second year of implementation were similar. On average, treatment schools had larger percentages of minority and low-income students, larger percentages of novice teachers, higher per pupil spending, and lower enrollment than other eligible schools.
Intervention: In 2015–16, North Carolina began implementing a new school turnaround approach, NCT, that included (1) a comprehensive needs assessment and support for using the assessment findings to identify school needs and develop a school improvement plan, (2) school transformation coaching for school leaders, (3) instructional coaching for teachers identified to need support, and (4) principal participation in statewide professional development. The average treatment school received 45 instructional coach visits and 25 school transformation coach visits over the first 2 years of implementation, but number of visits varied considerably between schools. After the first school year in which NCT was implemented, the number of coaching visits declined significantly.
Research Design and Methods: The research team used two quantitative designs in this study: (1) researchers estimated the effects of NCT on student and teacher outcomes using both intent to treat and treatment on the treated regression discontinuity designs that relied on the assignment of eligible schools not meeting growth to the NCT based on the 2014–15 schoolwide proficiency rate, and (2) researchers used an interrupted time series design with a non-equivalent comparison group (difference-in-differences) to test moderated effects for the lowest achieving students and examine heterogeneity of effects. In addition, researchers explored potential mediators and suppressors in both the RD and DiD frameworks. The research team evaluated implementation fidelity by collecting surveys from teachers and principals at both treatment and comparison schools and assessing implementation quality using coaches' documentation of activities, on-site observations, and interviews at the NCT schools.
Control Condition: In this study, all eligible schools that missed the proficiency rate cut-off for NCT served as the comparison group (n=253). In the main effect estimates for student outcomes, 56 control schools were included in the analytical sample in the first year due to bandwidth parameters and 37 in year 2. In the main effect estimates for teacher outcomes, 51 control schools were included in the analytical sample in the first year due to bandwidth parameters and 37 in year 2.
Key Measures: Outcome measures obtained from administrative data for this study included student achievement (end of grade and end of course test scores), proficiency, graduation rate, grade retention, and chronic absenteeism. Also from administrative data, the teacher and principal outcome measures included principal and teacher turnover and, from statewide biennial survey data, teacher and principal working conditions. Implementation quality measures from interviews with teachers and principals included attitudes about and experiences with NCT processes. Implementation activity and intensity measures were drawn from coaching reports filed by instructional and transformation coaches for each visit to NCT schools in each of the 3 study years. Implementation fidelity measures were collected from a variety of sources including teacher and principal surveys developed and administered by the study team, coaching reports, and interviews.
Data Analytic Strategy: To estimate the main effects, parametric and non-parametric local average treatment effect estimates followed the What Works Clearinghouse standards for regression discontinuity designs. Both intent-to-treat and treatment on the treated were examined due to some non-compliance with assignment on both sides of the cut-off. Researchers estimated difference-in-differences and school fixed effects estimators to examine moderation and explore mediation.
Additional online resources and information:
Harbatkin, E. (2022). Staffing for school turnaround in rural settings. Leadership and Policy in Schools, 1-23.
Henry, G. T., & Harbatkin, E. (2020). The next generation of state reforms to improve their lowest performing schools: An evaluation of North Carolina's school transformation intervention. Journal of Research on Educational Effectiveness, 13(4), 702–730.
Henry, G. T., McNeill, S., & Harbatkin, E. (2022). Accountability-driven school reform: Are there unintended effects on younger children in untested grades? Early Childhood Research Quarterly, 61(4), 190–208.
* This grant was funded under the FY 2015 Evaluation of State Education Programs and Policies competition.