Skip Navigation
Funding Opportunities | Search Funded Research Grants and Contracts

IES Grant

Title: Modeling Longitudinal Effects of New York City's 5th Grade Promotion Policy on Student Achievement through a Regression Discontinuity Design
Center: NCER Year: 2009
Principal Investigator: Mariano, Louis Awardee: RAND Corporation
Program: Improving Education Systems      [Program Details]
Award Period: 4 years Award Amount: $244,251
Type: Efficacy and Replication Award Number: R305A090039
Description:

Co-Principal Investigator: Sheila Kirby

Purpose: This project will examine the effects of grade retention and supportive interventions under the promotion policy currently in place for 5th graders in New York City. This project is taking place within an already existing larger evaluation of the New York City promotion policy and is focusing specifically on the subset of at-risk students who qualify for support in both English and math. These students have been left out of similar past studies but are to be included in this study through the methodological extension of defining a unidimensional treatment assignment variable for the regression discontinuity design being used in the main evaluation.

Project Activities: The project will analyze longitudinal data on two cohorts of New York City 5th graders after the policy was implemented. This analysis will estimate the impacts of the city's grade retention policy on this subset of students' achievement in English and mathematics. The work will also include an estimation of the impact of the intensity of program participation on student outcomes.

Products: The main product of the study will be a test of the efficacy of grade retention policies, including supportive interventions, on student achievement for the subset of students eligible for support in both English and math. Additional products will include articles published in peer-reviewed journals.

Structured Abstract

Setting: This project will take place in New York City.

Population: New York City fifth-graders subject to the grade retention policy will be the target population of this study. The sample will be drawn from two such cohorts of fifth- grade students (2004–05 and 2005–06 5th graders) and these include about 120,000 students. Of these, about 3,200–22,000 fall within the neighborhood around the threshold value of the assignment variable depending on which of the three assessments is considered and how tightly the range is constructed around the threshold value.

Intervention: Under New York City's promotion policy: 1) students scoring below a threshold on a 4th grade spring assessment have an increased probability of attending a Saturday Preparatory Academy (SPA) for additional instructional support; 2) students scoring below a threshold on a 5th grade spring assessment are classified as at risk for retention and have an increased probability of attending a Summer Success Academy (SSA) for additional summer instruction; and 3) students scoring below a threshold on the 5th grade summer assessment have a sharp increase in the probability of retention. The SPA provides 3 hours of English and math instruction for 17 to 24 Saturday sessions. The Summer Success Academy offers 4.5 hours of instruction four days a week for six weeks during the summer, including 1.5 hours of English, 1.5 hours of mathematics, and 1 hour of academic intervention services. Grade retention provides one more year of 5th grade instruction.

Research Design and Methods: The study uses a regression discontinuity (RD) design to compare the relationship between a treatment assignment variable and an outcome variable for subjects above and below an assignment threshold (cut-off point) that determines "treatment" status. The relevant assignment variables are assessment scale scores in English and mathematics in the spring of 4th grade, spring of 5th grade, and summer of 5th grade for the SPA, SSA, and retention treatments, respectively. The outcome variables include the score on future assessments (from grade 5 up to grades 6 to 8 depending on the cohort and the student's promotion or retention). In practice, students below the threshold have an increased probability of treatment but not all students below the threshold receive the intervention. For example, a student may avoid being retained by demonstrating proficiency via a portfolio. As a result, a fuzzy RD design is used.

Control Condition: The control condition includes students who were near the threshold in terms of the score cutoff for each intervention (the Saturday Preparatory Academy, the Summer Success Academy, or grade retention), but who were not subjected to the intervention.

Key Measures: The outcomes of interest include ELA and mathematics scale scores on the 4th and 5th grade standardized assessments and also the scores on the 6th through 8th grade standardized assessments if completed.

Data Analytic Strategy: The problem of having two threshold scores (scoring below the threshold in either English or math raises the probability of the same intervention) is reduced to a single dimension by concentrating on the minimum score which is standardized by placing both scores on a common scale with respect to their thresholds.

The treatment effect will be identified by dividing the effect of treatment eligibility on the outcome at the threshold value by the effect of treatment eligibility on the probability of receiving the treatment at the threshold value. These two values will be estimated separately using local linear regression techniques and then the ratio of these two estimates will be taken as the estimate of the treatment effect on the treated.

Complementing the RD analyses, a generalized additive mixed model framework (GAMM) will be used to investigate how the intensity of SSA and SPA participation (using attendance data) affects student outcomes. The GAMM approach does not assume a constant linear relationship throughout the range of possible values, thereby allowing examination of the relationship between intensity of program participation and the outcome for change-points in the relationship (e.g., changes in slope) that would indicate critical levels of exposure.

** This project was submitted to and funded under Education Policy, Finance, and Systems in FY 2009.


Back