An Evaluation of Curriculum, Setting, and Mentoring on the Performance of Children Enrolled in Pre-Kindergarten [Let's Begin with the Letter People vs. business as usual]

Findings

Print knowledge outcomes—Indeterminate effect found for the domain

Outcome measure	Comparison	Period	Sample	Intervention mean	Comparison mean	Significant?	Improvement     index	Evidence tier
Woodcock-Johnson III (WJ-III): Letter-Word Identification subtest	Doors to Discovery vs. Business as Usual	Posttest	Preschool children; 365 students	11.01	9.99	No	--

Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.

Study Details

Setting

The study was conducted with children from 20 full-day preschools (54 classrooms: 27 Doors to Discovery™ and 27 comparison) selected from Head Start and public preschool programs in the greater Houston, Texas area.

Study sample

This randomized controlled study, conducted during the 2003–04 school year, included three groups: Doors to Discovery™, Let’s Begin with the Letter People®, and a comparison group. Study authors recruited 32 Title I and non-Title I (universal) preschools and Head Start centers that included a total of 79 classrooms. Within each of the three school types (Title I, non-Title I, and Head Start), schools were randomly assigned to either the Doors to Discovery™ intervention group, the Let’s Begin with the Letter People® intervention group, or a comparison group, with all classrooms within a preschool being assigned to the same intervention condition. Fourteen schools (27 classrooms) in the study were assigned to implement the Doors to Discovery™ curriculum, 12 schools (25 classrooms) implemented Let’s Begin with the Letter People®, and six schools (27 classrooms) were assigned to the comparison condition. In a second stage of random assignment, those schools assigned to Doors to Discovery™ and Let’s Begin with the Letter People® were randomly assigned (within each curriculum) to two groups: one that received mentoring, and one that did not receive mentoring. The number of children at random assignment is not provided in the article or in author queries. However, the authors indicated that the consent rate was 65% in the intervention classrooms and 55% in the comparison classrooms. The authors also noted that at random assignment, 215 study children were in intervention group classrooms, and 203 study children were in comparison classrooms. Using the consent rate and the number of study children, we estimate that the sample of children totaled 729: 324 in intervention classrooms and 405 in comparison classrooms. At baseline, 51% of children in the Head Start classrooms were Hispanic, and 38% were African American; in the Title I classrooms, 53% of children were Hispanic, and 27% were Caucasian; and in the non-Title I programs, 71% of children were Caucasian, and 15% were categorized as Other. Pretest data were collected prior to the implementation of the curriculum, and posttest data were collected at the end of the school year. The analysis sample for the Doors to Discovery™ study included 365 children (183 Doors to Discovery™ and 182 comparison).Although the Assel et al. (2007) study used a randomized controlled trial design to assign schools to intervention or comparison conditions, the study had high attrition at the child level and must demonstrate baseline equivalence between the intervention and comparison group sample of children used in the analyses of outcomes. An author query was conducted to obtain the study data necessary to establish equivalence at baseline (i.e., unadjusted means and standard deviations of the outcome measures for the intervention and comparison groups). Baseline equivalence was established for outcomes in the print knowledge domain but not for the oral language or phonological processing domains. The study also discusses differences in child outcomes for the groups that received mentoring compared with those that did not, but since the estimated differences are not presented in the paper, we do not present these analyses in this intervention report.

Intervention Group

Intervention group teachers implemented Doors to Discovery™. Eight thematic units cover topics such as nature, friendship, communities, society, and health and present rich information. The teacher guide offers open-ended statements and questions to promote discussion. Fidelity to the curriculum was measured three times during the school year. At the first evaluation, 29% of teachers scored at high levels for curriculum fidelity (4 or 5 on a 5-point scale). By mid-year, 57% of teachers received high scores for curriculum fidelity. A second intervention group was assigned to the Let’s Begin with the Letter People® curriculum; the effects of this intervention on the study sample are not discussed in this report.

Comparison Group

Comparison group teachers used nonspecific curricula, which included a variety of curriculum materials that followed state guidelines for public preschool programs. Head Start comparison classrooms did not use a curriculum with a specified scope or sequence.

Outcome descriptions

Print knowledge was assessed with the WJ-III Letter-Word Identification subtest. For a more detailed description of this outcome measure, see Appendix B. In addition, the study authors assessed children in the oral language and phonological processing domains. Oral language was assessed with the Preschool Language Scale, Fourth Edition (PLS-4) Auditory Comprehension Subscale and the Expressive Vocabulary Test (EVT). Phonological processing was assessed with the Developing Skill Checklist (DSC) and the Rhyming section from the WJ-III Sound Awareness subtest. The authors did not establish baseline equivalence on the outcomes in these domains.

Support for implementation

Intervention teachers were trained on Doors to Discovery™ by the curriculum’s publishing company during a 4-day workshop. Training took place in small groups and included instruction in all content areas. The mentors were senior-level trainers of the curriculum. Teachers in the mentoring condition received help from one of three senior-level trainers of the curriculum who served as mentors. Mentors met with teachers two times a month for about one and a half hours, providing assistance in areas of lesson planning, curriculum components, and fidelity, among other topics. Mentors also identified and discussed areas of improvement for individual teachers. All teachers, regardless of mentoring condition, received three feedback sessions over the course of the school year surrounding their implementation of the intervention.

Findings

Oral language outcomes—Indeterminate effect found for the domain

Outcome measure	Comparison	Period	Sample	Intervention mean	Comparison mean	Significant?	Improvement     index	Evidence tier
Preschool Language Scale Fourth Edition (PLS-4): Auditory Comprehension subscale	Let's Begin with the Letter People® vs. Business as usual	Posttest	preschool children; 366 students	N/A	N/A	No	--
Expressive Vocabulary Test (EVT)	Let's Begin with the Letter People® vs. Business as usual	Posttest	preschool children; 366 students	N/A	N/A	No	--

Print knowledge outcomes—Indeterminate effect found for the domain

Outcome measure	Comparison	Period	Sample	Intervention mean	Comparison mean	Significant?	Improvement     index	Evidence tier
Woodcock-Johnson III (WJ-III): Letter-Word Identification subtest	Let's Begin with the Letter People® vs. Business as usual	Posttest	preschool children; 362 students	N/A	N/A	No	--

Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.

Study Details

Setting

The Let’s Begin with the Letter People® study was conducted with children from 18 fullday preschools (52 classrooms: 25 Let’s Begin with the Letter People® and 27 comparison) selected from Head Start and public preschool programs in the greater Houston, Texas area.

Study sample

This randomized controlled study, conducted during the 2003–04 school year, included three groups: Let’s Begin with the Letter People®, Doors to Discovery™, and a comparison group. Study authors recruited 32 Title I and non-Title I (universal) preschools and Head Start centers that included a total of 79 classrooms. Within each of the three school types (Title I, non-Title I, and Head Start), schools were randomly assigned to either the Let’s Begin with the Letter People ® intervention group, the Doors to Discovery™ intervention group, or a comparison group, with all classrooms within a preschool being assigned to the same intervention condition. Twelve schools in the study (25 classrooms) implemented Let’s Begin with the Letter People®, 14 schools (27 classrooms) were assigned to implement the Doors to Discovery™ curriculum, and six schools (27 classrooms) were assigned to the comparison condition. In a second stage of random assignment, those schools assigned to Let’s Begin with the Letter People® and Doors to Discovery™ were randomly assigned (within each curriculum) to two groups: one that received mentoring and one that did not receive mentoring. The number of children in the study classrooms at random assignment was not provided in the article or in author queries. However, the authors indicated that the consent rate was 65% in the intervention classrooms and 55% in the comparison classrooms. The authors also noted that at random assignment, 215 study children were in intervention group classrooms and 203 study children were in comparison classrooms. Using the consent rate and the number of study children, the WWC estimates that the total sample of children included 705, with 300 in intervention classrooms and 405 in comparison classrooms. At baseline, 51% of children in the Head Start classrooms were Hispanic and 38% were African American; in the Title I classrooms, 53% of the children were Hispanic and 27% were Caucasian; and in the non-Title I classrooms, 71% of the children were Caucasian and 15% were classified as Other. Pretest data were collected prior to the implementation of the curriculum, and posttest data were collected at the end of the school year. The analysis sample for the Let’s Begin with the Letter People® study included 366 children (182 Let’s Begin with the Letter People® and 184 comparison). Although the Assel et al. (2007) study used a randomized controlled trial design to assign schools to intervention or comparison conditions, the study had high attrition at the child level and must demonstrate baseline equivalence between the intervention and comparison group samples of children used in the analyses of outcomes. An author query was conducted to obtain the study data necessary to establish equivalence at baseline (i.e., unadjusted means and standard deviations of the outcome measures for the intervention and comparison groups). Baseline equivalence was established for outcomes in the oral language and print knowledge domains but not for the phonological processing domain. The study also discusses differences in child outcomes for the groups that received mentoring compared with those that did not, but since the estimated differences are not presented in the paper, we do not present these analyses in this intervention report.

Intervention Group

Intervention group teachers implemented Let’s Begin with the Letter People®. The curriculum is organized into 26 thematic units, with teachers typically devoting at least one week for each unit. Fidelity to the curriculum was measured three times during the school year. At the first evaluation, 30% of teachers scored at high levels for curriculum fidelity (4 or 5 on a 5-point scale). By mid-year, 72% of teachers received high scores for curriculum fidelity. A second intervention group was assigned to the Doors to Discovery™ curriculum; the effects of this intervention on the study sample are not discussed in this report.

Comparison Group

Comparison group teachers used nonspecific curricula, which included a variety of curriculum materials that followed state guidelines for public preschool programs. Head Start comparison classrooms did not use a curriculum with a specified scope or sequence.

Outcome descriptions

Outcome domains that met WWC evidence standards with reservations include oral language and print knowledge. Oral language was assessed with the PLS-4 Auditory Comprehension Subscale and EVT. Print knowledge was assessed with the WJ-III Letter-Word Identification subtest. For a more detailed description of the outcome measures presented in this intervention report, see Appendix B. In addition, the study authors assessed children in the phonological processing domain. Phonological processing was assessed with the Developing Skill Checklist (DSC) and the Rhyming section from the WJ-III Sound Awareness subtest. Children were assessed at the beginning and end of the school year.

Support for implementation

Intervention teachers were trained on Let’s Begin with the Letter People® by the curriculum’s publishing company during a 4-day workshop. Training took place in small groups and included instruction in all content areas. The mentors were senior-level trainers of the curriculum. Teachers in the mentoring condition received help from one of three senior-level trainers of the curriculum who served as mentors. Mentors met with teachers two times a month for about 1.5 hours, providing assistance in areas of lesson planning, curriculum components, and fidelity, among other topics. Mentors also identified and discussed areas of improvement for individual teachers. All teachers, regardless of mentoring condition, received three feedback sessions over the course of the school year, surrounding their implementation of the intervention.