Study Design

The study uses a regression-discontinuity (RD) design to assess the impact of ERF funding and program support for preschools on the language and literacy preparedness of preschool children. This study design takes advantage of the fact that the U.S. Department of Education (ED) is required to follow a formal, structured process for selecting grantees to receive ERF funding. In its published announcement of the availability of ERF grants for FY 2003 (Federal Register of March 11, 2003), ED established criteria for scoring each application received. Independent reviewers used these criteria to review and score applications. ED then awarded ERF grants to the grant applicants with the highest application scores, progressing down the score distribution until all funding available for the fiscal year had been allocated. In this way, 30 grants were awarded to the grant applicants with scores of at least 74; applicants with scores below 74 were not awarded grants. Impact estimates were obtained by comparing child outcomes and teacher practices in funded sites to those in unfunded sites, controlling for a smooth function of the application score.

The final evaluation sample was composed of a treatment group, which consisted of 4-year-olds attending preschool in 28 of 30 ERF grantee sites, whereas the comparison group consisted of children attending preschool in 37 of the 67 unfunded applicant sites that had the highest application scores and that agreed to participate in the study. Approximately three classrooms were selected from each participating site with probabilities proportional to the number of eligible students in each class (see Table 1). The study team randomly selected approximately 11 4-year-old students per classroom whose parents had provided written consent for participation in the study.

The study team collected data for the evaluation from several sources. Trained staff directly assessed the language and literacy skills of children participating in the study. Trained observers measured classroom practice in a subsample of study classrooms. The teachers of all children in the sample and the director or principal of each preschool participating in the study completed a self-administered questionnaire. Teachers of the sampled children were also asked to rate each child's social-emotional behavior. The study team also obtained data from the preschools about children's school attendance for the 2004–2005 year. Finally, parents of the sampled children were interviewed by telephone.

Data were collected at two times: fall 2004 and spring 2005. The same data-collection instruments and procedures were used in the funded and unfunded sites.

Child Assessments. Table 2 shows the instruments that were used to measure children's language and literacy skills in three domains (print and letter knowledge, phonological awareness, and oral language) and their social-emotional behavior.

Classroom observations and surveys. Classroom practice and overall quality of the preschool classrooms were measured by two observation instruments—the Teacher Behavior Rating Scale (TBRS)² and 11 items from the Early Childhood Environment Rating Scale-Revised (ECERS-R) that form the Teaching and Interactions Subscale.³ Trained members of the study team conducted the classroom observations.

The evaluation team also developed self-administered surveys that the teachers and preschool principals or directors completed in the fall of 2004 and spring 2005. Parents of children in the study were interviewed through computer-assisted telephone interviewing. The team conducted in-depth telephone interviews with grantee directors for each of the 28 funded grantees to learn about their use of ERF funds, including challenges encountered and notable successes.

Impact estimation and hypothesis testing. Impact estimates were obtained by comparing child outcomes and teacher practices in funded sites to those in unfunded sites, controlling for a smooth function of the application score. If the application score fully reflects the selection rule used to award ERF grants and we control for the correct function of the score, this approach produces unbiased estimates of the effect of ERF.

We adopted a 2-tailed hypothesis test because it was unclear before the evaluation whether ERF funding would improve all outcomes. For each outcome, the findings indicate the statistical significance of the impact estimates at the 5-percent level. The analysis methods accounted for the fact that some outcome domains contained multiple measures. The tables presented include checkmarks for domains in which impacts are jointly statistically significant once the adjustment for multiple comparisons is made. The tables also include p-values for tests of statistical significance of individual outcomes that do not reflect adjustments for multiple comparisons. The conclusions are unaffected when adjustments for multiple comparisons are applied.

The following sections contain findings about

• characteristics of ERF children and preschools
• ERF impacts on teachers and classroom practices
• ERF impacts on children's language and literacy skills and social-emotional outcomes

The evaluation also estimated ERF impacts for several subgroups defined by key characteristics of children, preschools, and teachers.

Top