Description:

Co-Principal Investigator: Hirokazu Yoshikawa (New York University) and Rebecca Unterman (MDRC)

Purpose: Rigorous evaluations of public prekindergarten programs have confirmed that children enrolled in these programs have higher language, literacy, and mathematics outcomes at kindergarten entry, on average. A prior IES-funded study of the Boston Public Schools (BPS) prekindergarten program extended this evidence base. In a regression discontinuity (RD) study of over 2,000 children, the researchers found that the BPS program had moderate-to-large impacts on children’s language, literacy, and mathematics skills, and small impacts on children’s emotional development outcomes. Although there have been evaluations of the impacts of public prekindergarten programs on children’s school readiness skills, there is a lack of evidence about the longer-term impacts of those programs. The purpose of this project is to conduct a retrospective efficacy study of the short- and medium term impacts of the Boston public prekindergarten program on key child academic and school progress outcomes, measured at kindergarten entry, kindergarten exit, and at the end of first and third grade.

Project Activities: The researchers will estimate causal impacts by taking advantage of the yearly BPS school assignment lottery that randomizes some children into the program and denies access to others. Over a three year period, secondary data from four cohorts of students (2007-2008; 2008-2009; 2009-2010; and 2010-2011) will be obtained and used to investigate impacts on student outcomes and variations in those impacts.  Secondary data sources will be used to examine cross-school variation in the persistence of child-level impacts and predictors of variation in these impacts. Researchers will address the following research questions: (1) Does the Boston prekindergarten program have positive short-term impacts on children’s beginning-of-kindergarten literacy skills? (2) Does the Boston prekindergarten program have positive medium-term impacts on children’s end-of-year kindergarten and first-grade literacy skills and on their third-grade state standardized mathematics and reading test scores? (3) Does the Boston prekindergarten program reduce children’s medium-term risk of being retained in grade or of being classified as special-needs? (4) Do student characteristics (cohort, race/ethnicity, home language, free-reduced-lunch status, and gender) moderate the short-term and medium-term impacts of the Boston prekindergarten program? and (5) Is there statistically significant variation across schools in short- and medium-term impacts for prekindergarten participants and in the sustainability of impacts? If so, what are the school-level mediators of this variation?

Products: The products of this project will be evidence of the longer-term effects of participating in the BPS prekindergarten program on student achievement and school progress in elementary school.  Study results will have high practical value in informing public prekindergarten programmatic decisions and public policy proposals to expand access to public prekindergarten programs nationally, as well as in identifying the conditions under which prekindergarten impacts do and do not persist. Peer reviewed publications will also be produced.

Structured Abstract

Setting: This study will take place in Boston, Massachusetts.

Sample: The sample for this study will consist of children whose families applied for a BPS prekindergarten slot between 2007 and 2010 in at least one school that had more applicants than seats. The sample includes children from a range of socioeconomic backgrounds and racial/ethnic groups.  The sample also includes English language learners.

Intervention: The BPS intervention consists of a year of full-day prekindergarten in a classroom staffed by a teacher with at least a Bachelor’s degree and a teacher’s assistant. In the 2007-2008 and 2008-2009 school years, teachers received training in two early childhood curricula, one which targeted children’s language and literacy skills (Opening the World of Learning, OWL) and one which targeted children’s mathematics skills (Building Blocks). Teachers also received bi-weekly visits from an early childhood coach. The coaches were master teachers who helped the prekindergarten teachers implement the curricula and address challenges in their classroom, such as classroom management. In the 2009-2010 school year, the district made changes in the supports provided to teachers. In that year, the district provided training and biweekly coaching on the curricula to new prekindergarten teachers only (e.g. new teachers were trained and supported via the model used with the 2007-2008 and 2008-2009 cohorts of teachers). Most veteran teachers received only training (and no coaching) from the district in 2009-2010 and 2010-2011.    

Research Design and Methods: In this study, researchers will use secondary data sources from four cohorts of children, two of which experienced an enriched version of the intervention and two of which experienced a streamlined and less expensive version of the intervention, to explore potential differences in the program impacts over time. Key outcomes in the proposed study are children’s kindergarten, first grade, and third grade literacy skills, children’s third-grade mathematics skills, children’s third-grade grade retention, and children’s third grade-special education receipt. Using a multi-level lottery fixed effects approach, the researchers will use the exogenous variation in program receipt created by the lottery assignment system to estimate causal program impacts on substantively important child outcomes in kindergarten, first, and third grade. The presence (or absence) of significant variation in impacts and sustainability of impacts across schools, as well as classroom-level and school-level predictors of variation, will be determined.  Researchers will conduct analyses to evaluate the casual impacts of those participating in the BPS program.  Retrospective data will be used to identify the type of care experienced by children randomized out of the program (e.g. the counterfactual). In year 1, the researchers will obtain secondary data sources from the Boston Public Schools and the Massachusetts Department of Education (MCAS), clean data, and conduct preliminary analyses.  In year 2, the researchers will obtain additional data from BPS and MCAS, conduct final analyses, and write papers summarizing the main and subgroup findings from their intent to treat and local average treatment effect analyses. In year 3, the researchers will write a paper on variation in impacts and impact sustainability and on predictors of variation.

Control Condition: The control condition consists of children who did not participate in the Boston Public School prekindergarten program. These children attended a range of standard early childhood or child care programs, including Head Start, public or private child care centers, non-relative home-based care, and relative care.

Key Measures: To assess the impacts of the BPS prekindergarten program on students’ kindergarten and first grade literacy skills, the researchers will use teacher-collected data from the Dynamic Indicators of Basic Literacy Skills (DIBELS).  Administered subtests measured children’s letter knowledge (Letter Naming Fluency subtest), oral reading fluency (Oral Reading Fluency subtest), phonological awareness (Initial Sound Fluency and Phoneme Segmentation subtests), and alphabetic principle (Nonsense Word Fluency subtest). For third grade outcomes, scaled scores from the statewide mathematics and reading standardized tests, the Massachusetts Comprehensive Assessment System (MCAS), will be used to evaluate longer-term impacts on students’ reading and mathematics achievement. In addition to the reading and mathematics outcomes, the researchers will also examine impacts of prekindergarten participation on retention rates and special education placement at the end of third grade.

Data Analytic Strategy: To address research questions about the impacts for BPS participants, researchers will conduct regression analyses for each outcome. To estimate impacts for children who enroll in BPS prekindergarten, two-stage least squares analyses will be conducted. Researchers will conduct additional analyses to address variation in impacts for each cohort of BPS participants.

Products and Publications

ERIC Citations: Find available citations in ERIC for this award here.

Journal articles

Kabay, S., Weiland, C., & Yoshikawa, H. (2020).Costs of the Boston public prekindergarten program. Journal of Research on Educational Effectiveness, 13, 574–600. Full text

Rochester, S., Weiland, C., Unterman, R., & McCormick, M. (2019). The little kids down

Shapiro, A., Martin, E., Weiland, C., & Unterman, R. (2019). If you offer it, will they come? Patterns of application and enrollment behavior in a universal prekindergarten context. AERA Open, 5(2), 1–22. 2332858419848442. Full text

Weiland, C., Unterman, R., & Shapiro, A. (2021). The kindergarten hotspot: Literacy skill convergence between Boston prekindergarten enrollees and non-enrollees. Child Development, 92(2), 600–608.

Weiland, C., Unterman, R., Shapiro, A., Staszak, S., Rochester, S., & Martin, E. (2020). The effects of enrolling in oversubscribed prekindergarten programs through third grade. Child Development, 5, 1401–1422.Full text