REL Midwest Ask A REL Response

Educator Effectiveness

July 2017

Center for Prevention Research and Development, University of Illinois. (2015). Schools to Watch: School Transformation Network, a U.S. Department of Education Investing in Innovation (i3) development grant. (Final Evaluation Report). Champaign, IL: Author. Retrieved from https://eric.ed.gov/?id=ED564016

From the abstract: "The Schools to Watch: School Transformation Network Project is a whole school reform model designed to improve the educational practices, experiences, and outcomes of low-performing middle-grades schools. Developed by the National Forum to Accelerate Middle-Grades Reform, the four-year project was funded in 2010 by a U.S. Department of Education Investing in Innovation (i3) development grant. The purpose of the study was to examine the impact of the project on intermediate outcomes such as culture, collaboration, and instructional practices as well as the long term outcome of student achievement. The study employed a quasi-experimental design where two student cohorts were tracked over four years at 34 schools (17 intervention and 17 comparison) in three states. The intervention schools were comprised of persistently low-performing middle-grades schools serving high need students. Comparison schools were selected using key demographics to match to intervention schools. Several process and measurement tools for assessing implementation and intermediate outcomes were used, including surveys, the STW criteria rating rubric, coach's logs, and focus groups. The long term outcome data for the impact study included student English and math achievement scores on annual standardized state assessments. To examine achievement scores between intervention and comparison students, a series of 2-level models (students within schools) were run to assess 8th grade achievement (i.e., after students received all three years of the intervention). Results showed that i3 STW Project schools improved their culture and climate, collaboration practices, leadership practices, STW criteria implementation, and classroom instructional practices. There was no overall intervention effect on either English or math student achievement, however, significant results were found for the highest implemented schools, those project schools that achieved STW designation during the project. The results of the study provide unique insight into the reform process for i3 STW Project schools as well as other middle-grades schools that are struggling to improve. The multiple supports provided by the project combined with the guiding vision of the STW criteria and rubric supported these high need schools to improve contextual factors (i.e., culture, collaboration, leadership, teaching and learning practices), and for a subsample of schools, student achievement. Districts and schools embarking on reform need to focus on collaborative leadership, have a guiding vision, use a continuous improvement model for instructional improvements, and value networking with other schools to gain knowledge."

Note: The School Transformation Network Project includes components to change school culture.

Christofferson, R. D., & Callahan, K. (2015). Positive Behavior Support in Schools (PBSIS): An administrative perspective on the implementation of a comprehensive school-wide intervention in an urban charter school. Education Leadership Review of Doctoral Research, 2(2), 35-49. Retrieved from http://eric.ed.gov/?id=EJ1105721

From the abstract: "This research explores the implementation of a school-wide intervention program that was designed to foster and instill intrinsic values based on an external reward system. The Positive Behavior Support in Schools (PBSIS) is an intervention intended to improve the climate of schools using system-wide positive behavioral interventions to discourage disruptive behaviors. The charter school that was the focus of this research experienced high staff turnover, negative school climate and student suspension rates that exceeded the state average. A mixed methods research design included de-identified data that were retrieved from 200 students in grades kindergarten through two, 205 parents and 54 staff members. The data sources included data from the School-wide Evaluation Tool (SET), Climate Survey and Office Discipline Referrals (ODR). Results indicated the implementation of Positive Behavior Support in Schools had a positive and significant impact on improving student behaviors and school climate. Results indicated that the implementation of the program significantly reduced the number of office discipline referrals and in-school suspension rates, and improved perceptions of students, staff and parents regarding the school climate. However, the results also indicated that there was no significant difference in the out-of-school suspension rates during the two-year implementation of PBSIS. This study provided administrators and staff with a comprehensive understanding of the implementation challenges associated with a school-wide intervention, as well as evidence to support practices that were effective."

Faria, A.-M., Sorensen, N., Heppen, J., Bowdon, J., Taylor, S., Eisner, R., & Foster, S. (2017). Getting students on track for graduation: Impact of the Early Warning Intervention and Monitoring System after one year. Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Midwest. Retrieved from https://eric.ed.gov/?id=ED573814

From the abstract: "Although high school graduation rates are rising—the national rate was 82 percent during the 2013/14 school year (U.S. Department of Education, 2015) - dropping out remains a persistent problem in the Midwest and nationally. Many schools now use early warning systems to identify students who are at risk of not graduating, with the goal of intervening early to help students get back on track for on-time graduation. Although research has guided decisions about the types of data and indicators used to flag students as being at risk, little is known about the impact of early warning systems on students and schools-and in particular, whether these systems do help get students back on track. This study, designed in collaboration with the REL Midwest Dropout Prevention Research Alliance, examined the impact and implementation of one early warning system-the Early Warning Intervention and Monitoring System (EWIMS)-on student and school outcomes. To assess the impact of EWIMS on student and school outcomes, 73 high schools in three Midwest Region states were randomly assigned to implement EWIMS during the 2014/15 school year (37 EWIMS schools) or to continue their usual practices for identifying and supporting students at risk of not graduating on time and to delay implementation of EWIMS until the following school year (36 control schools). The study included 37,671 students in their first or second year of high school, with 18,634 students in EWIMS schools and 19,037 students in control schools. EWIMS and control schools and students were similar on all background characteristics prior to random assignment. The study examined the impacts of EWIMS on indicators of student risk and on student progress in school after the first year of EWIMS adoption. The study found that EWIMS reduced the percentage of students with risk indicators related to chronic absence and course failure but not related to low GPAs or suspension: (1) The percentage of students who were chronically absent (missed 10 percent or more of instructional time) was lower in EWIMS schools (10 percent) than in control schools (14 percent); this 4 percentage point difference was statistically significant; and (2) The percentage of students who failed one or more courses was lower in EWIMS schools (21 percent) than in control schools (26 percent); this 5 percentage point difference was statistically significant; (3) The percentage of students who had a low GPA (2.0 or lower) was 17 percent in EWIMS schools and 19 percent in control schools; this difference was not statistically significant. However, sensitivity analyses that used continuous GPA data instead of the binary risk indicator showed that, on average, GPAs were higher in EWIMS schools (2.98) than in control schools (2.87); this difference was statistically significant; and (4) The percentage of students who were suspended once or more was 9 percent in both EWIMS and control schools; there was no statistically significant difference. EWIMS did not have an impact on student progress in school. That is, there was not a statistically significant difference between EWIMS and control schools in the percentage of students who earned insufficient credits to be on track to graduate within four years (14 percent in both). At the school level, EWIMS did not have a detectable impact on school data culture, that is, the ways in which schools use data to make decisions and identify students in need of additional support. In nearly all participating schools, overall implementation of the EWIMS seven-step process was low, and implementation was challenging. Nevertheless, EWIMS schools were more likely than control schools to report using an early warning system and having a dedicated team to identify and support at-risk students, but EWIMS schools did not differ from control schools in the frequency of data review or the number and type of interventions offered. This report provides rigorous initial evidence that even with limited implementation during the first year of adoption, using a comprehensive early warning system can reduce the percentage of students who are chronically absent or who fail one or more courses. These short-term results are promising because chronic absence and course failure in grades 9 and 10 are two key indicators that students are off track for on-time graduation. However, because the past research linking indicators to on-time graduation is correlational, it is not yet known if improving these indicators leads to improving on-time graduation rates. Also, EWIMS did not have a detectable impact on other measured indicators that are related to students' likelihood of on-time graduation, including low GPAs, suspensions, and earning insufficient credits."

Note: Early Warning Intervention and Monitoring System (EWIMS) includes components to change school data culture.

Gandhi, A. G., Slama, R., & Park, S. J. (2016). Focusing on the whole student: An evaluation of Massachusetts' Wraparound Zones Initiative. Paper presented at the Society for Research on Educational Effectiveness Conference, Washington, DC. Retrieved from https://eric.ed.gov/?id=ED567241

From the abstract: "Over the past twenty years, efforts to turn around low-performing schools have increasingly become a central component of federal and state education policy agendas. The purpose of the study was to evaluate the impact of the Wraparound Zones Initiative (WAZ), a program supported by the Massachusetts Department of Elementary and Secondary Education (ESE), on student outcomes including achievement, attendance, retention, and suspension. The study was conducted as part of a multi-year mixed-methods, formative and summative evaluation of WAZ that ESE commissioned from American Institutes for Research (AIR). The setting consisted of districts and schools in Massachusetts that had been identified by the state as chronically underperforming and in need of state intervention. The sample for this study was drawn from students in Cohort 1 and Cohort 2 WAZ schools serving elementary and/or middle grades, plus students in a set of matched non-WAZ comparison schools. The WAZ Initiative is designed to create coordinated district systems that allow schools to proactively and systematically address students' nonacademic needs. Comparative interrupted time series (CITS) design was used to measure the impact of receiving a WAZ grant on student outcomes, including student achievement, attendance, retention, and suspension. Overall, students in WAZ schools performed better on the Massachusetts Comprehensive Assessment System (MCAS) English language arts (ELA) and mathematics assessments as compared with students in comparison schools, when considering prior achievement trends. Effects were statistically significant after the second and third years of WAZ implementation for ELA, and after the second year for mathematics. This research study is significant in that it demonstrates that a program focused on student support and social-emotional learning can have an impact on student achievement, and can be an integral component of overall school turnaround strategy."

Hinojosa, T., Bos, J., O'Brien, B., Park, S., Liu, F., & Jerabek, A. (2016). Starting strong: A randomized controlled trial of the Building Assets Reducing Risks (BARR) model in 9th grade. Paper presented at the Society for Research on Educational Effectiveness Conference, Washington, DC. Retrieved from https://eric.ed.gov/?id=ED567027

From the abstract: "Students beginning high school commonly experience increased stress and behavior problems alongside declines in grades, attendance, interest in school, and perceptions of academic competence and self-esteem (Alvidrez & Weinstein, 1993; Reyes et al., 2000). Moreover, research indicates that, relative to students who graduate from high school, those who leave school prematurely are more likely to have experienced deeper 9th grade declines (Reyes et al., 2000; Roderick, 1995). Research also demonstrates positive school climates, positive relationships between students and staff, and among staff, are essential ingredients for turning around low performing schools (Gordon, 2006; National Research Council, 2004; Cohen, 2006; Jerald, 2006; De La Ossa, 2005). More specifically, research is growing on the effectiveness of student-teacher relationships in producing increased attendance and academic performance, and decreased behavior problems. For example, Allen, Pianta, Gregory, Mikami, and Lun (2011) conducted a randomized controlled trial (RCT) in which secondary school teachers were given a year of coaching on effective teaching and student-teacher interactions. After a year of training, students with teachers in the experimental group scored significantly higher on year-end achievement tests than did students in the control group. Quality of student-teacher interaction was a significant mediator of student achievement. With core components derived from research, the Building Assets Reducing Risks (BARR) model© targets students at this critical juncture in their academic career-9th grade. BARR addresses developmental, academic, and structural challenges in the 9th grade by combining student asset building, teachers- real-time analysis of student data, and intensive teacher collaboration to prevent course failure. BARR develops positive student-teacher relationships and integrates student supports into a school's existing model for addressing nonacademic barriers to learning. With funding from the Investing in Innovation program, BARR has been rigorously studied using an RCT and is now under scientific investigation in schools of varying locales and geographic regions to better understand the broader context under which BARR impacts students transitioning to the 9th grade. This current study is a multisite randomized controlled trial that explores replication of BARR across three cohorts of schools. Results from the first RCT demonstrated positive impact on 9th grade students' standardized test scores, credits earned, and overall failure rate."

Jones, A., & Shindler, J. (2016). Exploring the school climate—student achievement connection: Making sense of why the first precedes the second. Educational Leadership and Administration: Teaching and Program Development, 27, 35–51. Retrieved from http://eric.ed.gov/?id=EJ1094419

From the abstract: "Many educators view school climate and student achievement as separate considerations. For some, the idea of promoting a high quality climate can seem like a luxury in the face of the current high stakes assessment climate in which student achievement gains are the paramount consideration. However, the results of this study suggest that climate and student achievement are related. In fact, the quality of the climate appears to be the single most predictive factor in any school's capacity to promote student achievement. The purpose of this study was to explore the relationship between student academic achievement and various elements within the domain of school climate, and to examine the nature and potential causality of that relationship. The paper also seeks to derive implications for practice including a possible fundamental conceptual framework for climate quality and function and an operational roadmap for moving from a less functional to more functional climate. The study examined school climate and achievement at 30 urban public schools. The sample of schools was drawn from a large geographical area and reflected schools from diverse ethnic and socio-economic communities. Each school assessment team administered the Alliance for the Study of School Climate (ASSC) School Climate Assessment Instrument (SCAI). The team at each school incorporated a standard protocol and surveyed a minimum number of participants (N = 30+ students, 10+ teachers as well as 10+ staff and parents, with most sample sizes being larger). Focus group data were also collected. California State Academic Performance Index (API) and Similar School Rating (SIM) scores (published by the state), were used to measure student achievement at each school. The results of the study confirmed a strong relationship between the quality of school climate and academic achievement levels. While the direction of the causality between the two variables is not entirely indicated by the data, the substantial relationship between climate and SIM rating suggest that a conclusion can be drawn that, to a good degree, better climates led to achievement, and were not simply a byproduct."

Klute, M., Cherasaro, T., & Apthorp, H. (2016). Summary of research on the association between state interventions in chronically low-performing schools and student achievement. (REL 2016–138). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Central. Retrieved from http://eric.ed.gov/?id=ED565613

From the abstract: "This report summarizes the research on the association between state interventions in chronically low-performing schools and student achievement. Most of the research focused on one type of state intervention: working with a turnaround partner. Few studies were identified that examined other types of interventions, such as school closure, charter conversion, and school redesign. Most studies were descriptive, which limits the conclusions that can be drawn about the effectiveness of the interventions. Results of studies of turnaround partner interventions were mixed and suggested that student achievement was more likely to improve when particular factors-such as strong leadership, use of data to guide instruction, and a positive school culture characterized by trust and increased expectations for students—were in place in schools. Studies in California examined the Immediate Intervention/Underperforming Schools Program or its successor, the High Priority School Grant Program. Ten studies examined interventions in states other than California. Studies varied somewhat in the details of the interventions studied, including whether additional funding was provided to support implementation of reforms. Unlike interventions in California, studies in other states did not describe school participation in interventions as voluntary."

Kwong, D., & Davis, J. R. (2015). School climate for academic success: A multilevel analysis of school climate and student outcomes. Journal of Research in Education, 25(2), 68-81. Retrieved from http://eric.ed.gov/?id=EJ1098022

From the abstract: "This multilevel study examined the relationship between school climate and academic achievement. Using the Educational Longitudinal Survey (ELS, 2002), and a sample of 16,258 students and 1954 schools nationwide, we found that student';level perception of school climate-especially the student learning environment-was highly predictive of academic success in math and reading standardized test scores. This study confirmed that, among school climate variables at the school level, worse institutional facilities have a negative impact on student achievement and higher levels of institutional surveillance negatively affected the positive effects that student perceptions of safety and their learning environments had on student success. Finally, reducing high levels of institutional surveillance was found to mitigate socioeconomic inequalities."

Moller, S., Mickelson, R. A., Stearns, E., Bottia, M., & Banerjee, N. (2011). The benefits of collective pedagogical teacher culture for diverse students' mathematics achievement by academic engagement. Paper presented at the Society for Research on Educational Effectiveness Conference, Washington, DC. Retrieved from https://eric.ed.gov/?id=ED528806

From the abstract: "Most studies of educational organizations have focused on structural features of schools, such as size, resources, and infrastructure. Research on schools' organizational culture is more sparse. Yet, these studies have suggested that the organizational culture of schools can have important implications for teaching practices and student outcomes. Schools' organizational cultures are critical because they define how teachers interact with each other and students. To better understand the impact of organizational culture on students' achievement, the authors will assess how components of Collective Pedagogical Teacher Culture interact with students' race and SES to shape mathematics achievement trajectories. They will also assess how components of culture moderate the relationship between academic engagement and achievement."

Ohlson, M. (2009). Examining instructional leadership: A study of school culture and teacher quality characteristics influencing student outcomes. Florida Journal of Educational Administration & Policy, 2(2), 102–124. Retrieved from http://eric.ed.gov/?id=EJ930117

From the abstract: "First, to examine the influence of teacher input characteristics and teacher perceptions of school culture on student absences. Second, to examine the influence of teacher input characteristics and teacher perceptions of school culture on out-of-school suspensions. Data was obtained for the 2006-2007 school year from 23 urban public elementary schools in Florida. Using the school as the unit of analysis, data was collected examining student absences and suspensions during the 2006-2007 school year reported by the Florida Department of Education's School Indicators Report. Surveys were administered to examine collaborative leadership, teacher collaboration, unity of purpose, professional development, collegial support, and learning partnership, identified by Gruenert and Valentine (1998) as the six components of the collaborative culture of a school. Surveys also documented teacher input characteristics such as years teaching, percent out of field, and highest degree obtained. Correlation using multiple regression was used to analyze the data. As the Unity of Purpose factor increased, the model predicted that student absences would decrease by 22.56%. In addition, the model predicted that when either the average years of experience for teachers within a school increased or when the Collaborative Leadership factor increased, student suspensions would decrease by 0.413%. and 4.81% respectively."

Ohlson, M., Swanson, A., Adams-Manning, A., & Byrd, A. (2016). A culture of success—Examining school culture and student outcomes via a performance framework. Journal of Education and Learning, 5(1), 114–127. Retrieved from http://eric.ed.gov/?id=EJ1097804

From the abstract: "This study is a report of the relationship between a collaborative school culture, teacher quality and the influence these variables have upon student attendance and suspensions. The research is based upon data gathered from 50 public schools throughout the southeastern United States. Surveys were administered to examine teacher quality characteristics, elements of educational leadership, and components of a collaborative school culture. Data were analyzed in relation to teacher input characteristics such as certification, years teaching, percentage teaching out of field, and highest degree obtained. The findings revealed that as teacher collaboration increased, the model predicted that student suspensions would decrease by 6.709%. In addition, the model predicted that when the percentage of out-of-field teachers within a school increased, student suspensions would decrease by 0.16%. Finally, as the percentage of non-certified teachers within a school increased, the student suspension percentage increased by 0.22%. The findings offer valuable insight into the characteristics of quality teaching and school culture that demonstrate the greatest impact on student attendance and suspensions and may influence educational policy, teacher training, educational leadership, and school reform initiatives."

Reedy, K., & Lacireno-Paquet, N. (2015). Evaluation Brief: Implementation and outcomes of Kansas multi-tier system of supports. San Francisco, CA: WestEd. Retrieved from http://eric.ed.gov/?id=ED559728

From the abstract: "Implementation of multi-tier system of supports (MTSS) has grown rapidly in Kansas and is a key strategy for turning around low-performing schools in the state. MTSS is designed to improve outcomes for all students by instituting system-level change across the classroom, school, district, and state. Such systemic change is accomplished by developing a coherent continuum of evidence-based, systemwide practices to support a rapid response to each student's academic and behavioral needs. MTSS features frequent data-based monitoring for instructional decision making. MTSS core components include evidence-based curricula, high-quality instruction, a comprehensive assessment system, data-based decision making, effective intervention, fidelity of implementation, ongoing professional development, and leadership within an empowering school and district culture. This final summative evaluation report: (1) Describes the current status of MTSS implementation in Kansas; (2) Provides insights as to what it takes to implement MTSS with fidelity Offers recommendations for next steps; and (3) Offers recommendations for next steps. The evaluation team, led by WestEd's Kristin Reedy and Natalie Lacireno-Paquet, found that Kansas MTSS is substantially contributing to improved student outcomes at the local level, as well as benefitting teachers, improving instruction, and supporting better school functioning."

Note: Multi-tier system of supports (MTSS) includes components to change school culture.

Robertson, J. S., Smith, R. W., & Rinka, J. (2016). How did successful high schools improve their graduation rates? Journal of At-Risk Issues, 19(1), 10–18. Retrieved from http://eric.ed.gov/?id=EJ1104424

From the abstract: "The researchers surveyed 23 North Carolina high schools that had markedly improved their graduation rates over the past five years. The administrators reported on the dropout prevention practices and programs to which they attributed their improved graduation rates. The majority of schools reported policy changes, especially with suspension. The main interventions that showed positive impact were improvements in academic support, school/classroom climate, and transition from middle to high school. School districts did support their schools, but only 61% gave additional financial support. Several school administrators reported success of specific programs, teachers having engaging lessons and high expectations, close monitoring of students, giving students more chances to succeed, and improved individual/family support as contributors to their improved graduation rates."

Rutledge, S. A., & Cannata, M. (2015). Identifying and understanding effective high schools: Personalization for academic and social learning & student ownership and responsibility. Nashville, TN: National Center on Scaling Up Effective Schools. Retrieved from https://eric.ed.gov/?id=ED561272

From the abstract: "What are the policies, programs and practices that make some high schools in the same state and district context more effective than others? Motivated to understand the differences between schools with similar size and demographics yet different attendance, graduation and levels of student academic growth, the National Center for Scaling Up Effective Schools (NCSU)-a federally-funded project aimed at identifying, developing and implementing processes to scale up effective practices in urban high schools-embarked on year-long initiative to identify the major differences between two high and two low performing high schools in two districts—Broward County, Florida and Fort Worth, Texas. The study and findings are discussed in this report. In both districts it was found that the more effective high schools successfully mobilized both the academic and social emotional systems at their schools in the service of students. Administrators, guidance counselors and teachers at the effective schools worked together to bridge the academic and social emotional elements of schooling, seeing them as interwoven. They implemented teaching strategies, cultural habits, and organizational routines that promoted interconnections between the classroom and the social emotional lives of students. As such, students in the higher performing schools were much more likely than those in the lower performing schools to say that adults in the school supported them in developing both cognitive and non-cognitive skills necessary for their academic success and social wellbeing. Further, the high performing schools in the districts mobilized these academic and social emotional practices in different ways, particular to their local context and needs."

Voight, A., & Hanson, T. (2017). How are middle school climate and academic performance related across schools and over time? Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory West. Retrieved from http://eric.ed.gov/?id=ED572366

From the abstract: "A growing number of educators concur that, in order to improve student academic performance, schools need to focus not only on students' academic needs but also on their social, emotional, and material needs (Piscatelli & Lee, 2011). As a result, school climate-the social, emotional, and physical characteristics of a school community (Cohen, McCabe, Michelli, & Pickeral, 2009)-is gaining more attention as a lever to improve student academic performance. Most studies on the relationship between school climate and academic performance assert that a more positive school climate promotes higher academic performance. But evidence of a relationship between the two is weak. These studies generally are based on data collected at a single point in time and compare academic performance across schools with different school climates. They show that academic performance is higher in schools with a more positive school climate at single points in time. However, little evidence exists that changes in school climate over time are associated with changes in academic performance. This study used grade 7 student data from the California Healthy Kids Survey and administrative data for approximately 1,000 middle schools in California for 2004/05-2010/11 to measure students' perceptions about six domains of school climate. Schools with a positive school climate were those in which students reported high levels of safety/connectedness, caring relationships with adults, and meaningful student participation and low rates of substance use at school, bullying/discrimination, and student delinquency. School-level academic performance was measured using grade 7 California Standards Test scores in English language arts and math. The study team examined the relationship between school climate and academic performance across schools to determine whether in a given year California middle schools with a more positive school climate had higher academic performance. The study team also sought to determine how academic performance for a given school improved as school climate improved by examining how changes in school climate over two-year intervals were related to changes in average academic performance. Key findings include: (1) Schools with a more positive student-reported school climate had higher academic performance in English language arts and math; (2) Changes in a school's student-reported school climate over time were associated with changes in academic performance at that school; and (3) The changes in academic performance within a school that were associated with changes in student-reported school climate over time were substantially smaller than the differences in academic performance across schools with different school climate values in a given year. For example, in a given year schools at the 50th percentile on school climate were at the 48th percentile on math performance, on average, while schools at the 60th percentile on school climate were at the 51st percentile on math performance. This finding suggests that an improvement of 10 percentile points in school climate would be associated with an average 3 percentile point increase in academic performance. However, when followed over time, schools with a 10 percentile point increase in student perceptions of school climate averaged a less than 1 percentile point increase in academic performance."