Effects of Preschool Curriculum Programs on School Readiness. Report from the Preschool Curriculum Evaluation Research Initiative. NCER 2008-2009 [Doors to Discovery vs. business as usual]

Findings

General Mathematics Achievement outcomes—Indeterminate effect found for the domain

Outcome measure	Comparison	Period	Sample	Intervention mean	Comparison mean	Significant?	Improvement     index	Evidence tier
Child Math Assessment-Abbreviated (CMA-A) Composite score	Doors to Discovery vs. Business as Usual	Posttest	Preschool children; 183 students	N/A	N/A	No	--
Woodcock-Johnson III (WJ-III): Applied Problems subtest	Doors to Discovery vs. Business as Usual	Posttest	Preschool children; 183 students	N/A	N/A	No	--
Building Blocks Shape Composition Task	Doors to Discovery vs. Business as Usual	Posttest	Preschool children; 183 students	N/A	N/A	No	--

Phonological processing outcomes—Indeterminate effect found for the domain

Outcome measure	Comparison	Period	Sample	Intervention mean	Comparison mean	Significant?	Improvement     index	Evidence tier
Preschool Comprehensive Test of Phonological and Print Processing (Pre-CTOPPP) Elision subtest	Doors to Discovery vs. Business as Usual	Posttest	Preschool children; 182 students	N/A	N/A	No	--

Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.

Study Details

Setting

The study was conducted with children from 29 full-day preschool classrooms (14 Doors to Discovery™ and 15 comparison) selected from Head Start and public preschool programs in the greater Houston, Texas area.

Study sample

This randomized controlled study, conducted during the 2003–04 and 2004–05 school years, included three groups: Doors to Discovery™, Let’s Begin with the Letter People®, and a comparison group. Study authors recruited 32 Title I and non-Title I (universal) preschools and Head Start centers that included a total of 79 classrooms. Within each of the three school types (Title I, non-Title I, and Head Start), schools were randomly assigned to either the Doors to Discovery™ intervention group, the Let’s Begin with the Letter People® intervention group, or a comparison group, with all classrooms within a preschool being assigned to the same intervention condition. Fourteen schools (27 classrooms) in the study were assigned to implement the Doors to Discovery™ curriculum, 12 schools (25 classrooms) implemented Let’s Begin with the Letter People®, and six schools (27 classrooms) were assigned to the comparison condition. Subsequent to randomization, teachers were provided with a description of the national PCER study; of those teachers that opted to participate in the national PCER study during the 2003–04 school year, 45 were randomly selected (15 from each group). All 79 classrooms participated in the local investigator’s pilot study during the first year. Following the pilot year, and prior to starting the national PCER study, one teacher (and her classroom) dropped out of the study, leaving 14 Doors to Discovery™ classrooms,15 Let’s Begin with the Letter People® classrooms, and 15 comparison classrooms. The evaluation of Doors to Discovery™ included 29 of the 44 classrooms (14 Doors to Discovery™ classrooms and 15 comparison classrooms, while the remaining 15 classrooms were assigned to Let’s Begin with the Letter People®). Seven children (whose parents had provided consent to participate in the study) were randomly selected from each classroom at baseline for inclusion in the study. The number of children participating in the study at baseline was 196 (100 Doors to Discovery™ and 96 comparison). The parental consent rate was 65% for the intervention group and 55% for the comparison group. At baseline, children in the study averaged 4.6 years of age; 55% were male; 43% were Hispanic, 30% were Caucasian, and 13% were African American. The analysis sample for the Doors to Discovery™ study included 183 children (94 Doors to Discovery™ and 89 comparison).For the PCER Consortium (2008, Chapter 6) study, the Doors to Discovery™ intervention had been in place for a full (pilot) year when the evaluation year started. Although the PCER Consortium (2008, Chapter 6) study used a randomized controlled trial design to assign schools to intervention or comparison conditions in the pilot year—with all classrooms in a school assigned to the same condition—the study analyzed data from the second year of implementation, when children who had been in the classrooms at random assignment had moved to kindergarten and a new class of children had replaced them. Thus, the study had high attrition at the child level and must demonstrate baseline equivalence between the intervention and comparison group sample of children used in the analyses of outcomes. An author query was conducted to obtain the study data necessary to establish equivalence at baseline for one outcome measure in each domain (i.e., unadjusted means and standard deviations of the outcome measures for the intervention and comparison groups). The pretest data provided for each domain were used to establish baseline equivalence for the domain. Baseline equivalence was established from the data provided by the study authors. Baseline equivalence of the analytic sample of children in the two groups at the end of kindergarten was not available, so findings from the kindergarten follow-up are not reported.

Intervention Group

Intervention group teachers implemented Doors to Discovery™. For this study, each classroom’s fidelity to the curriculum was rated on a 4-point scale, ranging from “not at all” (0) to “high” (3). The average score for the Doors to Discovery™ classrooms was 2.13 on this measure. A second intervention group was assigned to the Let’s Begin with the Letter People® curriculum; the effects of this intervention on the study sample are not discussed in this report.

Comparison Group

Comparison teachers used teacher-developed nonspecific curricula. Their classrooms were rated with the same fidelity measure used in the Doors to Discovery™ classrooms, which ranged from 0 to 3. The average score for the comparison classrooms was 1.0.

Outcome descriptions

The outcome domains assessed were children’s oral language, print knowledge, phonological processing, and math. Only outcomes in the phonological processing and math domains met evidence standards with reservations. Phonological processing was assessed with the Pre- CTOPPP Elision subtest. Math was assessed with the WJ-III Applied Problems subtest, the CMA-A, and the Building Blocks Shape Composition task. For a more detailed description of these outcome measures, see Appendix B. Oral language was assessed with the PPVT-III and the Test of Language Development–Primary III (TOLD-P:3) Grammatic Understanding subtest. Print knowledge was assessed with the Test of Early Reading Ability-III (TERA-3), the WJ-III Letter-Word Identification subtest, and the WJ-III Spelling subtest. Baseline equivalence was not established for outcomes in the oral language and print knowledge domains, and therefore, these findings are not reported.

Support for implementation

Teachers received curriculum training prior to the start of the 2003–04 school year. This was the second year of implementation of the intervention, and most of the teachers had been trained prior to the start of the 2002–03 school year. New teachers each received 12 hours of training, and returning teachers each received 6 hours of training. The research team collected site-specific curriculum fidelity data three times during the preschool year. All classrooms were observed using the Teacher Behavior Rating Scale in fall and spring of the preschool year.

Findings

Oral language outcomes—Indeterminate effect found for the domain

Outcome measure	Comparison	Period	Sample	Intervention mean	Comparison mean	Significant?	Improvement     index	Evidence tier
Test of Language Development - Primary III (TOLD-PIII): Grammatic Understanding subtest	Let's Begin with the Letter People® vs. Business as usual	Posttest	preschool children; 184 students	N/A	N/A	No	--
Peabody Picture Vocabulary Test III (PPVT-III)	Let's Begin with the Letter People® vs. Business as usual	Posttest	preschool children; 184 students	N/A	N/A	No	--

Phonological processing outcomes—Indeterminate effect found for the domain

Outcome measure	Comparison	Period	Sample	Intervention mean	Comparison mean	Significant?	Improvement     index	Evidence tier
Preschool Comprehensive Test of Phonological and Print Processing (Pre-CTOPPP) Elision subtest	Let's Begin with the Letter People® vs. Business as usual	Posttest	preschool children; 184 students	N/A	N/A	No	--

Print knowledge outcomes—Indeterminate effect found for the domain

Outcome measure	Comparison	Period	Sample	Intervention mean	Comparison mean	Significant?	Improvement     index	Evidence tier
Woodcock-Johnson III (WJ-III): Spelling subtest	Let's Begin with the Letter People® vs. Business as usual	Posttest	preschool children; 184 students	101.34	97.37	No	--
Woodcock-Johnson III (WJ-III): Letter-Word Identification subtest	Let's Begin with the Letter People® vs. Business as usual	Posttest	preschool children; 184 students	108.72	106.04	No	--
Test of Early Reading Ability III (TERA-III)	Let's Begin with the Letter People® vs. Business as usual	Posttest	preschool children; 183 students	92.94	92.76	No	--

Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.

Study Details

Setting

The Let’s Begin with the Letter People® study was conducted with children from 30 full-day preschool classrooms (15 Let’s Begin with the Letter People® and 15 comparison) selected from Head Start and public preschool programs in the greater Houston, Texas area.

Study sample

This randomized controlled study, conducted during the 2003–04 and 2004–05 school years, included three groups: Let’s Begin with the Letter People®, Doors to Discovery™, and a comparison group. Study authors recruited 32 Title I and non-Title I preschools and Head Start centers that included a total of 79 classrooms. Within each of the three school types (Title I, non-Title I, and Head Start), schools were randomly assigned to either the Let’s Begin with the Letter People® intervention group, the Doors to Discovery™ intervention group, or a comparison group, with all classrooms within a preschool being assigned to the same intervention condition. Twelve schools in the study (25 classrooms) were assigned to implement Let’s Begin with the Letter People®, 14 schools (27 classrooms) implemented the Doors to Discovery ™ curriculum, and six schools (27 classrooms) were assigned to the comparison condition. Subsequent to randomization, teachers were provided with a description of the national PCER study; of those that opted to participate in the national PCER study during the 2003–04 school year, 45 were randomly selected (15 from each group). All 79 classrooms participated in the local investigator’s pilot study during the first year. Following the pilot year, and prior to starting the national PCER study, one teacher (and her classroom) dropped out of the study, leaving 15 Let’s Begin with the Letter People®, 14 Doors to Discovery™, and 15 comparison classrooms. The evaluation of Let’s Begin with the Letter People® included 30 of the 44 classrooms (15 Let’s Begin with the Letter People® and 15 comparison, while the remaining 14 were assigned to Doors to Discovery™). Seven children (whose parents had provided consent to participate in the study) were randomly selected from each classroom at baseline for inclusion in the study. The number of children participating in the study at baseline was 196 (100 Let’s Begin with the Letter People® and 96 comparison). The parental consent rate was 65% for the intervention group and 55% for the comparison group.16 At baseline, children in the study averaged 4.6 years of age; 55% were male; 43% were Hispanic, 30% were White, and 13% were African American. The analysis sample for the Let’s Begin with the Letter People® study included 184 children (95 Let’s Begin with the Letter People® and 89 comparison). For the PCER Consortium (2008, Chapter 6) study, the Let’s Begin with the Letter People® intervention had been in place for a full (pilot) year when the evaluation year started. Although the PCER Consortium (2008, Chapter 6) study used a randomized controlled trial design to assign classrooms to intervention or comparison conditions in the pilot year, the study analyzed data from the second year of implementation, when children who had been in the classrooms at random assignment had moved to kindergarten and a new class of children had replaced them. Thus, the study had high attrition at the child level and must demonstrate baseline equivalence between the intervention and comparison group sample of children used in the analyses of outcomes. An author query was conducted to obtain the study data necessary to establish equivalence at baseline for one outcome measure in each domain (i.e., unadjusted means and standard deviations of the outcome measures for the intervention and the comparison groups). The pretest data provided for each domain were used to establish baseline equivalence for the domain. Baseline equivalence was established from the data provided by the study authors. Baseline equivalence of the analytic sample of children in the two groups at the end of kindergarten was not available, so findings from the kindergarten followup are not reported.

Intervention Group

Intervention group teachers implemented Let’s Begin with the Letter People®. Each classroom’s fidelity to the curriculum was rated on a four-point scale ranging from “not at all” (0) to “high” (3). The average score for Let’s Begin with the Letter People® classrooms was 1.86 on this measure. A second intervention group was assigned to the Doors to Discovery™ curriculum; the effects of this intervention on the study sample are not discussed in this report.

Comparison Group

Comparison group classrooms used a variety of teacher-developed, nonspecific curricula reflecting the business-as-usual curricula for those classrooms. Comparison teachers’ classrooms were rated with the same fidelity measure used in the Let’s Begin with the Letter People® classrooms, which ranged from 0 to 3. The average score for the comparison classrooms using this measure was 1.0.

Outcome descriptions

The outcome domains assessed were children’s oral language, print knowledge, phonological processing, and math. Oral language was assessed with the PPVT-III and the TOLD-P:3 Grammatic Understanding subtest. Print knowledge was assessed with the TERA-3 and the WJ-III Letter-Word Identification and Spelling subtests. Phonological processing was assessed with the Pre-CTOPPP Elision subtest. For a more detailed description of these outcome measures, see Appendix B. Math was assessed with the WJ-III Applied Problems subtest, the Child Math Assessment–Abbreviated (CMA-A), and the Building Blocks Shape Composition task. The math outcomes are not reported because the findings for this domain do not meet WWC evidence standards due to a lack of baseline equivalence of the intervention and comparison children in the analytic sample.

Support for implementation

Teachers received curriculum training prior to the start of the 2003–04 school year. This was the second year of implementation of the intervention, and most of the teachers had been trained prior to the start of the 2002–03 school year. New teachers each received 12 hours of training, and returning teachers each received 6 hours of training. The research team collected site-specific curriculum fidelity data three times during the preschool year. All classrooms were observed using the Teacher Behavior Rating Scale in fall and spring of the preschool year.