Appendix A1.1 Study characteristics: Starkey & Klein, 2005 (randomized controlled trial)
| Characteristic | Description |
|---|---|
| Study citation1 | Starkey, P., & Klein, A. (2005). A longitudinal study of the effects of a pre-kindergarten mathematics curriculum on low-income children’s mathematical knowledge (From PCER 2002: Grantee Annual Progress Report (2005), IES Grant No. R305J020026). Berkeley: University of California. |
| Participants | The study was implemented with two cohorts of children from two states across a two-year time period. This WWC review focuses on cohort one combined across states.2 During the first year of implementation, 20 Head Start and 20 state-funded preschool classrooms were randomly assigned to the Pre-K Mathematics intervention or business-as-usual comparison groups within program type. The classroom assignment to conditions was maintained for the second year of implementation. Four intervention and two comparison classrooms from the first year of implementation were unable to participate and were replaced by randomly selecting classrooms from a list of volunteers in the second year.2 The study began with 316 low-income children in cohort one combined across states. During year one of the study, 38 children left resulting in a final cohort one sample of 278 children. The mean age of the children in cohort one was 4.4 years. Fifty-three percent of the children in cohort one were African-American; 22% were Hispanic; 22% were Caucasian; 4% were Asian-American; and 4% were interracial or another ethnicity. Forty-eight percent of the children were female. |
| Setting | The study took place in New York and California in 40 Head Start and state-funded classrooms from 37 schools in cohort one and 40 Head Start and state-funded classrooms from 36 schools in cohort two. |
| Intervention3 | Teachers implemented the Pre-K Mathematics curriculum with small groups of 4–6 children in twice weekly 20-minute sessions for one school year. Each week involved the introduction of a new math activity. The sessions included activities in seven units: counting and number; understanding arithmetic operations (fall and spring units); spatial sense and geometry; patterns; measurement and data; and logical reasoning. In addition, teachers used two other instructional activities: computer activities (DLM Express software) and mathematics learning centers, which included materials from the small-group activities and additional mathematics materials from the classroom. Home activity materials parallel to the classroom activities were sent home every one to two weeks for parents to use with their children. Teachers tracked children’s progress using a Math Mastery Form, and treatment fidelity data were collected using the Fidelity of Implementation Record Sheet. |
| Comparison | Children in the business-as-usual comparison group participated in the curriculum used in their schools (Creative Curriculum, High/Scope, Montessori, or specialized literacy curricula and curricula developed by local teachers and school districts). |
| Primary outcomes and measurement | The primary outcome domain assessed was math and it was measured with the researcher-developed Child Math Assessment (see Appendix A2 for a more detailed description of the outcome measure). |
| Teacher training | During the first year of implementation teachers received one four-day workshop at the beginning of the school year to learn how to implement units one to three of the curriculum and another four-day workshop at mid-year to learn how to implement units four to seven of the curriculum. In addition, teachers were provided on-site training and implementation fidelity checks twice each month. In the second year of implementation, teachers received two-day refresher workshops at the beginning of the school year and at mid-year, and received on-site training and fidelity checks twice a month. |
|
1 The study authors submitted an executive summary to the WWC and provided additional details about participants, sample sizes, study design, intervention implementation, and the data necessary to calculate effect sizes upon WWC request.
2 The study authors implemented a pure randomized controlled trial for cohort one (meets WWC evidence standards), whereas in cohort two the study authors replaced classrooms lost to attrition via random selection (meets WWC evidence standards with reservations). Therefore, the WWC bases the rating of effectiveness on the data from cohort one only because it has the strongest research design. The data from cohort two are provided in Appendix A4. 3 The impact of theDLM Express software cannot be separated from the impact of the Pre-K Mathematics curriculum. | |
Appendix A1.2 Study characteristics: Clements & Sarama, 2006 (randomized controlled trial with attrition problems)1
| Characteristic | Description |
|---|---|
| Study citation | Clements, D. H., & Sarama, J. (2006, June). Scaling up the implementation of a pre-Kindergarten mathematics curriculum: The Building Blocks curriculum. Paper presented at the Institute of Education Sciences Research Conference, Washington, D. C. |
| Participants | Teachers were randomly assigned to conditions in two separate steps. Twenty-four teachers from preschool programs serving low-income children were randomly assigned to two intervention groups (Pre-K Mathematics or Building Blocks for Math)2 or a business-as-usual comparison group. Twelve teachers from preschool programs serving mixedincome children were randomly assigned to the Building Blocks for Math group or the business-as-usual comparison group. Consequently there were a total of eight teachers in the Pre-K Mathematics group and 14 teachers in the business-as-usual comparison group. Eight preschool-age children were randomly selected from each classroom for assessment (N=176).3 After attrition, the final sample included 21 teachers and 152 children (seven teachers and 51 children in the Pre-K Mathematic s group; 14 teachers and 101 children in the business-as-usual comparison group). |
| Setting | The study was conducted in Head Start and state-funded preschool programs in New York State. |
| Intervention | Children in the intervention condition used the Pre-K Mathematics curriculum, which was designed to develop informal mathematical knowledge and skills. The curriculum was implemented primarily during 15- to 20-minute small-group activities at least twice a week and 10- to 15-minute whole-class math activities once a week. In addition, these classrooms used DLM Express software for 5 to 10 minutes twice a week. Weekly letters were sent to parents that included math activities similar to those children were learning at school. The intervention lasted for 26 weeks and the teachers maintained their daily activities and schedule while inserting mathematics activities at appropriate times during the day.4 |
| Comparison | Children in the business-as-usual comparison group participated in their regular daily activities and schedule, with emphasis on small groups and computer activities. These included city-wide math activities, Creative Curriculum, Montessori math activities, or “home-grown” math materials based on state standards. |
| Primary outcomes and measurement | The primary outcome domain assessed was math and it was measured with the Early Mathematics Assessment (see Appendix A2 for a more detailed description of the outcome measure). The study authors also assessed implementation fidelity with the Fidelity of Implementation measure and the quality of the mathematics environment using the Classroom Observation of Early Mathematics Environment and Teaching. This WWC review does not include the results from these observations.5 |
| Teacher training | Professional development activities for teachers in the Pre-K Mathematics group consisted of four days of training, a monthly two-hour class, and monthly in-class coaching by project staff. Teacher training covered a number of topics such as supporting mathematical development in the classroom, recognizing and supporting mathematics throughout the day, setting up mathematics learning centers, teaching with computers, small-group activities, and supporting mathematical development in the home. |
|
1 The study was downgraded by the WWC due to within-cluster differential attrition of 11% between the Pre-K Mathematics group and the business-as-usual comparison group. Additionally, the two groups differed at pretest by more than half of a standard deviation.
2 The study also included a Building Blocks for Math intervention group. The study authors labeled the Building Blocks for Math group as the “intervention group” and the Pre-K Mathematics group as the “comparison group”; however, the WWC considers Building Blocks for Math as a separate intervention (see the separate WWC Building Blocks for Math intervention report. For the rating of effectiveness in this WWC intervention report, the WWC includes only the results comparing the Pre-K Mathematics group to the business-as-usual comparison group; however, results for the comparison between the curricula are included in a separate section of this report and Appendix A5. 3 The remaining 14 teachers were assigned to the Building Blocks for Math group, which is not the main focus of this WWC intervention report. 4 The impact of the DLM Express software cannot be separated from the impact of the Pre-K Mathematics curriculum. Children in the Building Blocks for Math intervention group participated in 10- to 15-minute small-group (4–6 children) math activities weekly. These children also participated in 5- to 15-minute whole-group math activities four times a week and 5- to 10-minute computer activities twice a week. Related family activities were sent home weekly. The intervention lasted for 26 weeks, and intervention teachers maintained their daily activities and schedule while inserting mathematics activities at appropriate times during the day. 5 for further details about the outcomes included in the early childhood education topic review, please see the Early Childhood Education Protocol. | |
Appendix A2 Outcome measures in the math domain
| Outcome measure | Description |
|---|---|
| Child Math Assessment | A researcher-developed measure designed to assess young children’s early mathematical knowledge in the areas of number, arithmetic, space and geometry, measurement and pattern knowledge (as cited in Starkey & Klein, 2005).1 |
| Early Mathematics Assessment | A researcher-developed measure that uses two individual child interviews to assess young children’s mathematical knowledge and skills in the areas of number, geometry, measurement, and patterning (as cited in Clements & Sarama, 2006). |
| 1 The Child Math Assessment was developed by the researchers, who are also the program developers, and this measure was developed for the purposes of this research project. The measure was confirmed to have sufficient face validity by the WWC ECE Principal Investigators and a psychometric study to establish its measurement properties by the study authors as a part of the Institute of Education Sciences funded Interagency Education Research Initiative (IERI) Scale-Up project. | |
Appendix A3 Summary of study findings included in the rating for the math domain 1
| Authors'findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome (standard deviation2) | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (Classrooms/children)3 | Pre-K Math group4 | Comparison group4 | Mean difference5 (Pre-K Math – comparison) | Effect size6 | Statistical significance7 (at α= 0.05) | Improvement index8 |
| Starkey & Klein, 2005 (randomized controlled trial)9 | ||||||||
| Child Math Assessment | 3–4 year olds | 40/278 | 0.55 (0.13) | 0.47 (0.14) | 0.08 | 0.58 | Statistically significant | +22 |
| Average10 for math (Starkey & Klein, 2005) | 0.58 | Statistically significant | +22 | |||||
| Clements & Sarama, 2006 (randomized controlled trial with attrition problems)11 | ||||||||
| Early Mathematics Assessment | Preschool children | 21/152 | 58.01 (6.53) | 53.22 (8.38) | 4.79 | 0.61 | Statistically significant | +23 |
| Average10 for math (Clements & Sarama, 2006) | 0.61 | Statistically significant | +23 | |||||
| Domain average10 for math across all studies | 0.60 | na | +22 | |||||
|
na = not applicable 1 This appendix reports findings considered for the effectiveness rating and the average improvement indices. For Starkey and Klein (2005), the findings for cohort one are combined across state; findings for cohort two are reported in Appendix A4.For Clements & Sarama (2006), findings from the head-to-head comparison of Pre-K Mathematics to Building Blocks for Math are not included in these ratings, but are reported in Appendix A5.2 The standard deviation across all students in each group shows how dispersed the participants'outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. For Starkey and Klein (2005), the standard deviations were provided by the study authors upon WWC request. 3 For Clements and Sarama (2006), the cluster-unit is the teacher and the child-level sample size was provided by the study authors upon WWC request. For Starkey and Klein (2005), the classroom and child-level sample sizes were provided by the study authors upon WWC request. 4 For Clements and Sarama (2006), the intervention group mean (58.01) equals the comparison group mean (53.22) plus the program coefficient from the author-conducted HLM analysis (4.79). For Starkey and Klein (2005), the posttest means are covariate-adjusted means provided by the study authors upon WWC request. 5 Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. For Clements and Sarama (2006), the mean difference was the program coefficient from the author-conducted HLM analysis. 6 For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. 7 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 8 The improvement index represents the difference between the percentile rank of the average student in the intervention condition versus the percentile rank of the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting results favorable to the intervention group. 9 The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Starkey and Klein (2005), no corrections for clustering or multiple comparisons were needed because the study authors used HLM to analyze their data and accounted for clustering of children in classrooms. 10 The WWC-computed domain average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average improvement indices are calculated from the average effect sizes. 11 In the case of Clements and Sarama (2006), no corrections for clustering or multiple comparisons were needed because the study authors used HLM to analyze their data and accounted for clustering of children in classrooms. | ||||||||
Appendix A4 Summary of findings from cohort two for the math domain11
| Authors'findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome (standard deviation2) | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (Classrooms/children)3 | Pre-K Math group4 | Comparison group4 | Mean difference5 (Pre-K Math – comparison) | Effect size6 | Statistical significance7 (at α= 0.05) | Improvement index8 |
| Starkey & Klein, 2005 (randomized controlled trial; Cohort 2)9 | ||||||||
| Child Math Assessment | 3–4 year olds | 40/286 | 0.58 (0.16) | 0.47 (0.15) | 0.11 | 0.70 | Statistically significant | +26 |
|
1
This appendix presents findings for cohort two combined across state for measures that fall in the math domain. Combined scores across state for cohort one were used for rating purposes and are presented in Appendix A3.
2 The standard deviation across all students in each group shows how dispersed the participants'outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. The posttest standard deviations were provided by the study authors upon WWC request. 3 The sample sizes were provided by the study authors upon WWC request. 4 The posttest means are covariate-adjusted means provided by the study authors upon WWC request. 5 Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 6 For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. 7 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 8 The improvement index represents the difference between the percentile rank of the average student in the intervention condition versus the percentile rank of the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting results favorable to the intervention group. 9 The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple comparisons were not done for findings not included in the overall intervention rating). For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Starkey and Klein (2005), no correction for clustering was needed because the study authors used HLM to analyze their data and accounted for clustering of children in classrooms. | ||||||||
Appendix A5 Summary of findings for comparisons between Pre-K Mathematics and Building Blocks for Math for the math domain1
| Authors'findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome (standard deviation2) | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (teachers/children)3 | Pre-K Math group4 | Building Blocks for Math group4 | Mean difference5 (Pre-K Math – Building Blocksfor Math) | Effect size6 | Statistical significance7 (at α= 0.05) | Improvement index8 |
| Clements & Sarama, 2006 (randomized controlled trial with attrition problems)9 | ||||||||
| Early Mathematics Assessment | Preschool children | 21/152 | 55.84 (6.53) | 59.39 (7.46) | -3.55 | -0.49 | Statistically significant | -19 |
| Domain average10 for math | -0.49 | Statistically significant | -19 | |||||
|
1
This appendix presents findings for the head-to-head comparison of Pre-K Mathematics and Building Blocks for Math for a measure that falls in the math domain. Comparisons of Pre-K Mathematics and the business-as-usual comparison group were used for rating purposes and are presented in Appendix A3.
| ||||||||
Appendix A6 Pre-K Mathematics rating for the math domain
The WWC rates an intervention'seffects in a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.1
For the outcome domain of math, the WWC rated Pre-K Mathematics as having positive effects. The remaining ratings (potentially positive effects, mixed effects, no discernible effects, potentially negative effects, and negative effects) were not considered, as Pre-K Mathematics was assigned the highest applicable rating.
| Rating received | |
|---|---|
|
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.
| |
Appendix A7 Extent of evidence by domain
| Sample size | ||||
|---|---|---|---|---|
| Outcome domain | Number of studies | Schools | Students | Extent of evidence1 |
| Oral language | 0 | 0 | 0 | na |
| Print knowledge | 0 | 0 | 0 | na |
| Phonological processing | 0 | 0 | 0 | na |
| Early reading/writing | 0 | 0 | 0 | na |
| Cognition | 0 | 0 | 0 | na |
| Math | 2 | 37+ | 61/430 | Medium to large |
|
na = not applicable/not studied 1 A rating of "medium to large" requires at least two studies and two schools across studies in one domain and a total sample size across studies of at least 350 students or 14 classrooms. Otherwise, the rating is "small." | ||||
