Purpose: This study examines how school leaders respond to the designation of not meeting AYP, what reforms (if any) are introduced into schools as a result, and the relationship of student performance to the designation and subsequent reforms.

Project Activities: The research team will use mixed-methods and a regression-discontinuity design to estimate the effect on achievement outcomes of being labeled "in need of improvement" at the whole school level, as well as for various subgroups. Schools are assigned to the treatment group (that is, classification as in need of improvement) based on their performance on the 2008 Pennsylvania state assessment.

Products: Products from this project include published reports on the impact on student achievement of school classification as "in need of improvement."

Setting: The study will be conducted in Pennsylvania due to a change in required proficiency levels occurring in 2008.

Population: The study will gather achievement and survey data from all 3,104 public schools in Pennsylvania.

Intervention: The intervention being tested in this research is classification as a school in need of improvement based on the performance of one student subgroup and the subsequent reforms that are adopted. In 2008, schools in Pennsylvania will be required to meet minimum proficiency levels that shift from 45% to 56% in mathematics and from 54% to 63% in reading. More than 2,000 public schools will need to increase achievement in order to meet these new goals and avoid classification as a school in need of improvement. Some percentage of these schools (forecasts for the number are difficult to make at this time but range from 200-500 schools) will fail to meet AYP for the first time and will do so because of the performance of one student subgroup in the schools.

Research Designs and Methods: The research team will use mixed-methods and a regression-discontinuity design to estimate the effect on achievement outcomes of being labeled "in need of improvement" at the whole school level, as well as for various subgroups. Schools are assigned to the treatment group (that is, classification as in need of improvement) based on their performance on the 2008 Pennsylvania state assessment. For example, schools that score at or above 56% proficient on the state math assessment will not receive the "treatment" of being labeled as "in need of improvement." Schools that score below 56% proficient may be designated as in need of improvement, and for the purposes of this analysis these schools are considered to be the treatment group. If the classification as "in need of improvement" leads to the use of improvement strategies, and those strategies successfully lead to improved achievement, then we can expect to see an impact of that classification.

Control Condition: The research team will compare the academic achievement of those schools that receive the designation of being "in need of improvement" with those schools that do not receive such a designation.

Key Measures: Outcomes in this research will include data about school improvement efforts following a new classification and an analysis of changes in student achievement that follow classification.

Data Analytic Strategy: The research team will first use qualitative and survey data to describe and categorize the improvement strategies that are chosen by schools following AYP classification. In order to examine the relationships between school classification, improvement strategy, and student performance, the research team will use a regression-discontinuity analysis drawing on school and subgroup achievement data from multiple years. They will also use a multivariate regression that accounts for a number of school variables, including school classification and improvement strategy, and identifies the contribution of these factors to change in student achievement. In combination, these analyses should illustrate the relationships, if any, between school classification, improvement strategies, and student achievement. Changes in school program and student achievement will be analyzed for the particular student subgroup that has failed to meet AYP as well as for the remainder of the student body.

Products and Publications

Beaver, J.K., and Weinbaum, E.H. (2015). State Test Data and School Improvement Efforts. Educational Policy, 29 (3), 478–503.

Nongovernment report, issue brief, or practice guide

Beaver, J.K., and Weinbaum, E.H. (2012). Measuring School Capacity, Maximizing School Improvement. Philadelphia: Consortium for Policy Research in Education.

Weinbaum, E.H., Weiss, M., and Beaver, J.K. (2012). Learning From NCLB: School Responses to Accountability Pressure and Student Subgroup Performance. Philadelphia: Consortium for Policy Research in Education.

Weiss, M.J., and Weinbaum, E.H. (under review). Multiple Rating Score Regression Discontinuity Design: Lessons From Attempting to Estimate the Effect of Adequate Yearly Progress (AYP) Labels. Philadelphia: Consortium for Policy Research in Education.

** This project was submitted to and funded under Education Policy, Finance, and Systems in FY 2008.