Appendix A1.1 STUdy Characteristics: Barnett, Jung, Yarosz, Thomas, Hornbeck, Stechuk, & Burns, 2008 (randomized controlled trial)
| Characteristic | Description |
|---|---|
| Study citation | Barnett, W., Jung, K., Yarosz, D., Thomas, J., Hornbeck, A., Stechuk, R., & Burns, S. (2008). Educational effects of the Tools of the Mind curriculum: a randomized trial. Early Childhood Research Quarterly, 23(3), 299–313. |
| Participants | In one school selected for the study, 7 classrooms on one floor were available for Tools of the Mind implementation and 11 classrooms on another floor were available for the control condition. Teachers and assistants were randomly assigned to classrooms using a stratified assignment procedure, and then three- and four-year-old children were randomly assigned to either Tools of the Mind curriculum classrooms or district curriculum classrooms. Poverty level, achievement, and minority status were similar across intervention and comparison groups. Among the children sampled, 93 percent were Hispanic, and about 70 percent considered Spanish their primary home language. Although the overall student attrition rate was more than 25 percent, and student consent after random assignment led to differential attrition, the post-attrition intervention and comparison samples were equivalent on achievement pretests. After one year, 85 Tools of the Mind students and 117 comparison students remained in the sample. |
| Setting | This study was conducted in 18 classrooms in a low-income urban school with state-financed Abbott full-day preschool education. |
| Intervention | Tools of the Mind aims to aid learning and development while emphasizing emergent literacy and self-regulation. The two main goals of the curriculum are to develop underlying cognitive skills (such as self-regulation, deliberate memory, and focused attention) and to develop specific academic skills (such as symbolic thought, literacy, and an understanding of math). Play is the leading activity for developing such skills and the curriculum emphasizes the teacher's role in supporting the development of mature intentional dramatic play. The study was conducted during the first year of program implementation of Tools of the Mind. |
| Comparison | Control classrooms implemented the standard district-created curriculum, which was described as a full-day PreK balanced literacy curriculum with themes. In structured observations of the control group, frequently observed activities were art projects that correlated with the "letter of the week," free play, large group movement and/or music, and such large group activities as story time. According to the study authors, although the control curriculum covered much of the same academic content and topics as Tools of the Mind, there was greater emphasis on teacher-imposed control and less on children’s self-regulation. |
| Primary outcomes and measurement | For both pre- and post-tests, the authors administered Peabody Picture Vocabulary Test-III, Expressive One-Word Picture Vocabulary Test-Revised, Animal Pegs Subtest of the Wechsler Preschool Primary Scale of Intelligence, and two subtests of the Woodcock-Johnson-Revised test (Applied Problems and Letter-Word Identification). Get Ready to Read! screening tool was used only at post-test assessment. IDEA Oral Language Proficiency Test was administered for the subsample of Spanish-speaking children. Problem Behaviors Scale of the Social Skills Rating System was also used in the study, but not included in this report because it was outside the scope of the Early Childhood Education review. For a more detailed description of these outcome measures, see Appendix A2.1–2.4. |
| Teacher and staff training | Teachers assigned to the Tools of the Mind group received four full days of curriculum training before the start of the school year. During the school year, they received 30-minute classroom visits approximately once a week from a Tools of the Mind trainer to address any difficulties they were having with the curriculum. In addition, Tools of the Mind teachers received 1 half-day workshop and 5 one-hour lunchtime meetings to discuss aspects of the curriculum. Control group teachers received similar amounts of training. They attended workshops on the already established district curriculum given by the district for the same amount of time. |
Appendix A2.1 Outcome measures for the oral language domain
| Outcome measure | Description |
|---|---|
| Peabody Picture Vocabulary Test-III | A standardized measure of children’s receptive vocabulary that requires children to identify pictures that correspond to spoken words (as cited in Barnett et al., 2008). |
| Expressive One-Word Picture Vocabulary Test-Revised | A standardized measure of children’s expressive vocabulary that requires them to name pictures of common objects, actions, and concepts (as cited in Barnett et al., 2008). |
| IDEA Oral Language Proficiency Test | This test assesses the receptive and expressive Spanish language skills of Spanish-speaking children (as cited in Barnett et al., 2008). |
Appendix A2.2 Outcome measures for THE print knowledge domain
| Outcome measure | Description |
|---|---|
| Woodcock-Johnson-Revised: Letter-Word Identification subtest | A standardized measure of children’s ability to name printed letters and words (as cited in Barnett et al., 2008). |
| Get Ready to Read! | A nonstandardized measure of readiness for reading instruction focusing on three core domains (print knowledge, emergent writing skills, and linguistic awareness) across 20 items to which children indicate their response by pointing (as cited in Barnett et al., 2008). |
Appendix A2.3 outcome measureS for the cognition domain
| Outcome measure | Description |
|---|---|
| Wechsler Preschool Primary Scale of Intelligence: Animal Pegs subtest | A subset from a standardized measure that assesses a child's nonverbal problem-solving and visual-motor proficiency as they place pegs of correct colors in a series of holes under pictures of animals (as cited in Barnett et al., 2008). |
Appendix A2.4 Outcome measures for the math domain
| Outcome measure | Description |
|---|---|
| Woodcock-Johnson-Revised: Applied Problems subtest | A subtest from a standardized measure that assesses children’s math skills by asking children to count small sets and to solve simple addition and subtraction questions using pictures (as cited in Barnett et al., 2008). |
Appendix A3.1 Summary of study findings included in the rating for the oral language domain1
| Authors' findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome (standard deviation2) | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (classrooms/students) | Tools of the Mind group | Comparison group | Mean difference3 (Tools of the Mind - comparison) | Effect size4 | Statistical significance5 (at α= 0.05) |
Improvement index6 |
| Barnett et al. (2008) (randomized controlled trial)7 | ||||||||
| Peabody Picture Vocabulary Test-III (PPVT-III) | Three- to four-year-olds | 18/198 | nr (19.19) | nr (15.90) | 3.59 | 0.21 | ns | +8 |
| Expressive One-Word Picture Vocabulary Test-Revised (EOWPVT-R) | Three- to four-year-olds | 18/193 | nr (14.06) | nr (12.22) |
1.19 | 0.09 | ns | +4 |
| Average for oral language8 | 0.15 | ns | +6 | |||||
|
ns = not statistically significant 1 This appendix reports findings considered for the effectiveness rating and the average improvement indices for the oral language domain. Subgroup findings for children who consider Spanish their primary language are not included in these ratings but are reported in Appendix A4.1. |
||||||||
Appendix A3.2 Summary of study findings included in the rating for the print knowledge domain1
| Authors' findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome (standard deviation2) | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (classrooms/students) | Tools of the Mind group | Comparison group | Mean difference3 (Tools of the Mind - comparison) | Effect size4 | Statistical significance5 (at α= 0.05) |
Improvement index6 |
| Barnett et al. (2008) (randomized controlled trial)7 | ||||||||
| Woodcock-Johnson-Revised Letter-Word Identification subtest | Three- to four-year-olds | 18/202 | nr (12.87) | nr (11.92) | –0.45 | –0.04 | ns | –1 |
| Get Ready to Read! | Three- to four-year-olds | 18/220 | nr (3.90) | nr (3.91) | 0.13 | 0.03 | ns | +1 |
| Average for print knowledge8 | 0.00 | ns | 0 | |||||
|
ns = not statistically significant 1 This appendix reports findings considered for the effectiveness rating and the improvement indices for the print knowledge domain. |
||||||||
Appendix A3.3 Summary of study findings included in the rating for the cognition domain1
| Authors' findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome (standard deviation2) | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (classrooms/students) | Tools of the Mind group | Comparison group | Mean difference3 (Tools of the Mind - comparison) | Effect size4 | Statistical significance5 (at α= 0.05) |
Improvement index6 |
| Barnett et al. (2008) (randomized controlled trial)7 | ||||||||
| Wechsler Preschool Primary Scale of Intelligence (WPPSI) Animal Pegs Subtest | Three- to four-year-olds | 18/200 | nr (15.22) | nr (16.35) | 0.84 | 0.05 | ns | +2 |
| Average for cognition8 | 0.05 | ns | +2 | |||||
|
ns = not statistically significant 1 This appendix reports findings considered for the effectiveness rating and the average improvement indices for the cognition domain. |
||||||||
Appendix A3.4 Summary of study findings included in the rating for the math domain1
| Authors' findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome (standard deviation2) | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (classrooms/students) | Tools of the Mind group | Comparison group | Mean difference3 (Tools of the Mind - comparison) |
Effect size4 | Statistical significance5 (at α= 0.05) |
Improvement index6 |
| Barnett et al. (2008) (randomized controlled trial)7 | ||||||||
| Woodcock-Johnson Revised Applied Problems | Three- to four-year-olds | 18/202 | nr (16.19) | nr (18.86) | 3.07 | 0.17 | ns | +7 |
| Average for math8 | 0.17 | ns | +7 | |||||
ns = not statistically significant 1 This appendix reports findings considered for the effectiveness rating and the average improvement indices for the math domain. |
||||||||
Appendix A4.1 Summary of subgroup findings for the oral language domain1
| Authors' findings from the study | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| Mean outcome (standard deviation2) | WWC calculations | ||||||||
| Outcome measure | Study sample | Sample size (classrooms/students) | Tools of the Mind group | Comparison group | Mean difference3 (Tools of the Mind - comparison) | Effect size4 | Statistical significance5 (at α= 0.05) |
Improvement index6 | |
| Barnett et al. (2008) (randomized controlled trial)7 | |||||||||
| IDEA Oral Language Proficiency Test in Spanish | Three- to four-year-olds | 18/132 | nr (8.49) | nr (6.82) | 2.36 | 0.31 | ns | +12 | |
|
ns = not statistically significant 1 This appendix presents subgroup findings for measures that fall in the oral language domain. The Oral Language Proficiency Test in Spanish was administered to children who considered Spanish their primary language (approximately 70 percent of the sample) to assess their Spanish language development. Total group scores were used for rating purposes and are presented in Appendix A3.1. |
|||||||||
Appendix A5.1 Tools of the mind rating for the oral language domain
The WWC rates an intervention’s effects for a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.1
For the outcome domain of oral language, the WWC rated Tools of the Mind as having no discernible effects. It did not meet the criteria for positive effects, potentially positive effects, mixed effects, potentially negative effects, or negative effects because no studies showed statistically significant or substantively important effects, either positive or negative.
| Rating received |
|---|
No discernible effects: No affirmative evidence of effects.
|
Other ratings considered
|
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.
AND
|
Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence.
AND
|
Mixed effects: Evidence of inconsistent effects as demonstrated through EITHER of the following.
AND
|
Potentially negative effects: Evidence of a negative effect with no overriding evidence.
AND
|
Negative effects: Strong evidence of a negative effect with no overriding evidence.
AND
|
| 1 For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of potentially positive or potentially negative effects. For a complete description, see the WWC Intervention Rating Scheme. |
Appendix A5.2 Tools of the Mind rating for the print knowledge domain
The WWC rates an intervention's effects for a given outcome domain as positive, potentially postive, mixed, no discernible effects, potentially negative, or negative.1
For the outcome domain of print knowledge, the WWC rated Tools of the Mind as having no discernible effects. It did not meet the criteria for positive effects, potentially positive effects, mixed effects, potentially negative effects, or negative effects because no studies showed statistically significant or substantively important effects, either positive or negative.
| Rating received |
|---|
|
No discernible effects: No affirmative evidence of effects.
|
| Other ratings considered |
|
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.
AND
|
Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence.
AND
|
Mixed effects: Evidence of inconsistent effects as demonstrated through EITHER of the following.
OR
|
Potentially negative effects: Evidence of a negative effect with no overriding contrary evidence.
AND
|
Negative effects: Strong evidence of a negative effect with no overriding contrary evidence.
AND
|
|
1 For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of potentially positive or potentially negative effects. For a complete description, see the WWC Intervention Rating Scheme. |
Appendix A5.3 Tools of the Mind rating for the cognition domain
The WWC rates an intervention's effects for a given outcome domain as positive, potentially postive, mixed, no discernible effects, potentially negative, or negative.1
For the outcome domain of cognition, the WWC rated Tools of the Mind as having no discernible effects. It did not meet the criteria for positive effects, potentially positive effects, mixed effects, potentially negative effects, or negative effects because no studies showed statistically significant or substantively important effects, either positive or negative.
| Rating received |
|---|
No discernible effects: No affirmative evidence of effects
|
| Other ratings considered |
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.
AND
|
Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence.
AND
|
Mixed effects: Evidence of inconsistent effects as demonstrated through EITHER of the following.
OR
|
Potentially negative effects: Evidence of a negative effect with no overriding contrary evidence.
AND
|
Negative effects: Strong evidence of a negative effect with no overriding contrary evidence.
AND
|
1 For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of potentially positive or potentially negative effects. For a complete description, see the WWC Intervention Rating Scheme. |
Appendix A5.4 Tools of the Mind rating for the math domain
The WWC rates an intervention's effects for a given outcome domain as positive, potentially postive, mixed, no discernible effects, potentially negative, or negative.1
For the outcome domain of math, the WWC rated Tools of the Mind as having no discernible effects. It did not meet the criteria for positive effects, potentially positive effects, mixed effects, potentially negative effects, or negative effects because no studies showed statistically significant or substantively important effects, either positive or negative.
| Rating received |
|---|
No discernible effects: No affirmative evidence of effects
|
| Other ratings considered |
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.
AND
|
Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence.
AND
|
Mixed effects: Evidence of inconsistent effects as demonstrated through EITHER of the following.
OR
|
Potentially negative effects: Evidence of a negative effect with no overriding contrary evidence.
AND
|
Negative effects: Strong evidence of a negative effect with no overriding contrary evidence.
AND
|
1 For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of potentially positive or potentially negative effects. For a complete description, see the WWC Intervention Rating Scheme. |
Appendix A5 Extent of evidence by domain
| Sample size | ||||
|---|---|---|---|---|
| Outcome domain | Number of studies | Schools | Students | Extent of evidence1 |
| Oral language | 1 | 1 | 198 | Small |
| Print knowledge | 1 | 1 | 220 | Small |
| Cognition | 1 | 1 | 200 | Small |
| Math skills | 1 | 1 | 202 | Small |
| Phonological processing | 0 | na | na | na |
| Early reading/writing | 0 | na | na | na |
na = not applicable/not studied 1A rating of "medium to large" requires at least two studies and two schools across studies in one domain and a total sample size across studies of at least 350 students or 14 classrooms. Otherwise, the rating is "small." |
||||
|Institute of Education Sciences
