Appendix A1 Study characteristics: RMC Research Corporation, 2003 (randomized controlled trial)
| Characteristic | Description |
|---|---|
| Study citation | RMC Research Corporation. (2003). Ready, Set, Leap! program: Newark prekindergarten study 2002–2003 final report. Retrieved from LeapFrog SchoolHouse Web site: http://www.leapfrogschoolhouse.com/content/research/RMC_RSLreport.pdf |
| Participants | Seventeen schools were randomly assigned to either an intervention (N=8) or a comparison (N=9) group. The study began with 308 inner-city, low-income preschool children enrolled in 34 classrooms in these 17 schools. The researchers excluded seven of the 34 classrooms because they included only children with moderate to severe disabilities. An additional 20 children were lost to attrition, resulting in a final sample of 254 children.1 The final sample included 129 children in the intervention group and 125 children in the comparison group. At posttest, the mean age of the children in the intervention group was 4.5 years; 57% were female; and 51% were African-American, 42% Hispanic, 5% Caucasian, and 2% Asian or other race/ethnicity. At posttest, the mean age of the children in the comparison group was 4.5 years; 53% were female; and 37% were African-American, 32% Hispanic, 24% Caucasian, and 7% Asian or other race/ethnicity. The intervention group had significantly more minority students than the comparison group. |
| Setting | The study took place in 17 inner-city preschools in Newark, New Jersey. |
| Intervention | Ready, Set, Leap!® is a comprehensive prekindergarten curriculum that focuses on early reading skills such as phonemic awareness, letter knowledge, and letter-sound correspondence. The curriculum was integrated into the existing High/Scope framework, which is a set of guiding principles and practices intended as an "open framework" that teams of adults can adapt to the special needs and conditions of the children in their group, as well as to their setting and their community."Active learning" is a central tenet of the High/Scope approach for all age levels. The intervention was administered from September 2002 through June 2003. No information about intervention implementation was provided. |
| Comparison | Children in the comparison group participated in the High/Scope curriculum. No information about the implementation of the High/Scope curriculum was provided. |
| Primary outcomes and measurement | The primary outcome domains assessed were children's oral language, print knowledge, phonological processing, and early reading/writing. Oral language was assessed with one standardized measure, the Peabody Picture Vocabulary Test-III (PPVT-III). Print knowledge was assessed with two standardized measures: the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) Letter Naming Fluency subtest and the Woodcock-Johnson III (W-J III) Letter-Word Identification subtest. Phonological processing was assessed with three standardized measures: the DIBELS Initial Sound Fluency subtest, the W-J III Sound Awareness-Rhyming subtest, and the Comprehensive Test of Phonological Processing (CTOPP) Blending Words subtest. Early reading/writing was assessed with one standardized measure, the W-J III Passage Comprehension subtest (see Appendices A2.1–A2.4 for more detailed descriptions of outcome measures).2 |
| Teacher training | The intervention group teachers received three days of training on the Ready, Set, Leap!® curriculum over the course of the year. |
| 1 Information about the numbers of children included in the classrooms who were excluded from the analysis was provided by the study authors upon the WWC request. 2 The authors also developed a phonological awareness composite (based on average raw scores from the tests of initial sound fluency, blending, and rhyming) and a letter identification composite (based on average raw scores from tests of letter-word identification, passage comprehension, and letter naming fluency). However, the WWC does not include these composites in this intervention report because the WWC includes each of the individual measures used to develop the composites. The authors also administered a Teacher Knowledge and Attitude Scale to teachers to assess teachers' knowledge about various domains of language and literacy; however, the WWC excluded this measure from the intervention report because it is not a child outcome. For further details about the outcomes included in the Early Childhood Education topic review, please see the Early Childhood Education Protocol. |
|
Appendix A2.1 Outcome measures in the oral language domain
| Characteristic | Description |
|---|---|
| Peabody Picture Vocabulary Test-III (PPVT-III) | A standardized measure of children's receptive vocabulary that requires children to identify pictures that correspond to spoken words (as cited in RMC Research Corporation, 2003). |
Appendix A2.2 Outcome measures in the print knowledge domain
| Characteristic | Description |
|---|---|
| Woodcock-Johnson III (W-J III) Letter-Word Identification subtest | A standardized measure of children's ability to name printed letters and words (as cited in RMC Research Corporation, 2003). |
| Dynamic Indicators of Basic Early Literacy Skills (DIBELS) Letter Naming Fluency subtest | A timed standardized measure to assess children's ability to name printed upper- and lowercase letters in about one minute (as cited in RMC Research Corporation, 2003). |
Appendix A2.3 Outcome measures in the phonological processing domain
| Characteristic | Description |
|---|---|
| Comprehensive Test of Phonological Processing (CTOPP) Blending Words subtest | A standardized measure of children's ability to blend orally presented sounds to form words (as cited in RMC Research Corporation, 2003). |
| DIBELS Initial Sound Fluency subtest | A timed standardized measure to assess children's ability to recognize and produce the initial sound in an orally presented word in about one minute (as cited in RMC Research Corporation, 2003). |
| W-J III Sound Awareness-Rhyming subtest | A standardized measure of children's ability to identify word sounds and rhymes when presented orally (as cited in RMC Research Corporation, 2003). |
Appendix A2.4 Outcome measures in the early reading/writing domain
| Characteristic | Description |
|---|---|
| W-J III Passage Comprehension subtest | A standardized measure of children's listening and reading comprehension skills that use a cloze procedure (as cited in RMC Research Corporation, 2003). |
Appendix A3.1 Summary of study findings included in the rating for the oral language domain1
| Authors' findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome (standard deviation2) | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (schools/children) | Ready, Set,Leap!® group | Comparison group | Mean difference3 (Ready, Set, Leap!® – comparison) | Effect size4 | Statistical significance5 (at α= 0.05) | Improvement index6 |
| RMC Research Corporation, 2003 (randomized controlled trial)7 | ||||||||
| PPVT III | 4 year olds | 17/254 | 56.73 (16.13) | 56.59 (13.82) | 0.14 | 0.01 | ns | 0 |
| Domain average8 for oral language | 0.01 | ns | 0 | |||||
|
ns = not statistically significant 2 The standard deviation across all students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. The standard deviations were provided by the study authors upon the WWC request. 3 Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. The means are regression adjusted. 4 For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. 5 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 6 The improvement index represents the difference between the percentile rank of the average student in the intervention condition versus the percentile rank of the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting results favorable to the intervention group. 7 The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of RMC Research Corporation (2003), a correction for clustering was needed, so the significance levels may differ from those reported in the original study. Because the study authors corrected only for clustering within classrooms, the WWC applied a correction for clustering within the schools because the school was the unit of assignment in the study reviewed. 8 This row provides the study average, which in this instance is also the domain average. The WWC-computed domain average effect size is a simple average rounded to two decimal places. The domain improvement index is calculated from the average effect size. |
||||||||
Appendix A3.2 Summary of study findings included in the rating for the print knowledge domain1
| Authors' findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome (standard deviation2) | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (schools/children) | Ready, Set,Leap!® group | Comparison group | Mean difference3 (Ready, Set,Leap!® – comparison) | Effect size4 | Statistical significance5 (at α= 0.05) | Improvement index6 |
| RMC Research Corporation, 2003 (randomized controlled trial)7 | ||||||||
| W-J III Letter-Word Identification subtest | 4 year olds | 17/254 | 13.59 (5.70) | 12.94 (5.06) | 0.65 | 0.12 | ns | +5 |
| DIBELS Letter Naming Fluency subtest | 4 year olds | 17/254 | 23.63 (14.94) | 24.76 (14.72) | -1.13 | -0.08 | ns | -3 |
| Domain average8 for print knowledge | 0.02 | ns | +1 | |||||
|
ns = not statistically significant 2 The standard deviation across all students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. The standard deviations were provided by the study authors upon the WWC request. 3 Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. The means are regression adjusted. 4 For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. 5 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 6 The improvement index represents the difference between the percentile rank of the average student in the intervention condition versus the percentile rank of the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting results favorable to the intervention group. 7 The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of RMC Research Corporation (2003), a correction for clustering was needed, so the significance levels may differ from those reported in the original study. Because the study authors corrected only for clustering within classrooms, the WWC applied a correction for clustering within the schools because the school was the unit of assignment in the study reviewed. 8 This row provides the study average, which in this instance is also the domain average. The WWC-computed domain average effect size is a simple average rounded to two decimal places. The domain improvement index is calculated from the average effect size. |
||||||||
Appendix A3.3 Summary of study findings included in the rating for the phonological processing domain1
| Authors' findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome (standard deviation2) | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (schools/children) | Ready, Set,Leap!® group | Comparison group | Mean difference3 (Ready, Set,Leap!® – comparison) | Effect size4 | Statistical significance5 (at α= 0.05) | Improvement index6 |
| RMC Research Corporation, 2003 (randomized controlled trial)7 | ||||||||
| CTOPP Blending Words subtest | 4 year olds | 17/254 | 4.24 (4.17) | 3.18 (3.38) | 1.06 | 0.28 | ns | +11 |
| DIBELS Initial Sound Fluency subtest | 4 year olds | 17/254 | 11.03 (8.24) | 9.58 (6.48) | 1.45 | 0.20 | ns | +8 |
| W-J III Sound Awareness-Rhyming subtest | 4 year olds | 17/254 | 5.49 (4.10) | 4.92 (3.72) | 0.57 | 0.15 | ns | +6 |
| Domain average8 for phonological processing | 0.21 | ns | +8 | |||||
|
ns = not statistically significant 2 The standard deviation across all students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. The standard deviations were provided by the study authors upon the WWC request. 3 Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. The means are regression adjusted. 4 For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. 5 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 6 The improvement index represents the difference between the percentile rank of the average student in the intervention condition versus the percentile rank of the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting results favorable to the intervention group. 7 The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of RMC Research Corporation (2003), a correction for clustering was needed, so the significance levels may differ from those reported in the original study. Because the study authors corrected only for clustering within classrooms, the WWC applied a correction for clustering within the schools because the school was the unit of assignment in the study reviewed. 8 This row provides the study average, which in this instance is also the domain average. The WWC-computed domain average effect size is a simple average rounded to two decimal places. The domain improvement index is calculated from the average effect size. |
||||||||
Appendix A3.4 Summary of study findings included in the rating for the early reading/writing domain1
| Authors' findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome (standard deviation2) | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (schools/children) | Ready, Set, Leap!® group | Comparison group | Mean difference3 (Ready, Set,Leap!® – comparison) | Effect size4 | Statistical significance5 (at α= 0.05) | Improvement index6 |
| RMC Research Corporation, 2003 (randomized controlled trial)7 | ||||||||
| W-J III Passage Comprehension subtest | 4 year olds | 17/254 | 5.89 (2.43) | 5.69 (2.24) | 0.20 | 0.09 | ns | +3 |
| Domain average8 for early reading/writing | 0.09 | ns | +3 | |||||
|
ns = not statistically significant 2 The standard deviation across all students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. The standard deviations were provided by the study authors upon the WWC request. 3 Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. The means are regression adjusted. 4 For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. 5 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 6 The improvement index represents the difference between the percentile rank of the average student in the intervention condition versus the percentile rank of the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting results favorable to the intervention group. 7 The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of RMC Research Corporation (2003), a correction for clustering was needed, so the significance levels may differ from those reported in the original study. Because the study authors corrected only for clustering within classrooms, the WWC applied a correction for clustering within the schools because the school was the unit of assignment in the study reviewed. 8 This row provides the study average, which in this instance is also the domain average. The WWC-computed domain average effect size is a simple average rounded to two decimal places. The domain improvement index is calculated from the average effect size. |
||||||||
Appendix A4.1 Ready, Set, Leap!® rating for the oral language domain
The WWC rates an intervention's effects in a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.1
For the outcome domain of oral language, the WWC rated Ready, Set, Leap!® as having no discernible effects. It did not meet the criteria for positive effects, potentially positive effects, mixed effects, potentially negative effects, or negative effects because no studies showed statistically significant or substantively important effects, either positive or negative.
| Rating received |
|---|
|
No discernible effects: No affirmative evidence of effects.
|
| Other ratings considered |
|
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.
|
|
Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence.
|
|
Mixed effects: Evidence of inconsistent effects as demonstrated through either of the following criteria.
|
|
Potentially negative effects: Evidence of a negative effect with no overriding contrary evidence.
|
|
Negative effects: Strong evidence of a negative effect with no overriding contrary evidence.
|
|
1 For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. |
Appendix A4.2 Ready, Set, Leap!® rating for the print knowledge domain
The WWC rates an intervention's effects in a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.1
For the outcome domain of print knowledge, the WWC rated Ready, Set, Leap!® as having no discernible effects. It did not meet the criteria for positive effects, potentially positive effects, mixed effects, potentially negative effects, or negative effects because no studies showed statistically significant or substantively important effects, either positive or negative.
| Rating received |
|---|
|
No discernible effects: No affirmative evidence of effects.
|
| Other ratings considered |
|
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.
|
|
Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence.
|
|
Mixed effects: Evidence of inconsistent effects as demonstrated through either of the following criteria.
|
|
Potentially negative effects: Evidence of a negative effect with no overriding contrary evidence.
|
|
Negative effects: Strong evidence of a negative effect with no overriding contrary evidence.
|
| 1 For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. |
Appendix A4.3 Ready, Set, Leap!® rating for the phonological processing domain
The WWC rates an intervention's effects in a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.1
For the outcome domain of phonological processing, the WWC rated Ready, Set, Leap!® as having no discernible effects. It did not meet the criteria for positive effects, potentially positive effects, mixed effects, potentially negative effects, or negative effects because no studies showed statistically significant or substantively important effects, either positive or negative.
| Rating received |
|---|
|
No discernible effects: No affirmative evidence of effects.
|
| Other ratings considered |
|
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.
|
|
Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence.
|
|
Mixed effects: Evidence of inconsistent effects as demonstrated through either of the following criteria.
|
|
Potentially negative effects: Evidence of a negative effect with no overriding contrary evidence.
|
|
Negative effects: Strong evidence of a negative effect with no overriding contrary evidence.
|
| 1 For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. |
Appendix A4.4 Ready, Set, Leap!® rating for the early reading/writing domain
The WWC rates an intervention's effects in a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.1
For the outcome domain of early reading/writing, the WWC rated Ready, Set, Leap!® as having no discernible effects. It did not meet the criteria for positive effects, potentially positive effects, mixed effects, potentially negative effects, or negative effects because no studies showed statistically significant or substantively important effects, either positive or negative.
| Rating received |
|---|
|
No discernible effects: No affirmative evidence of effects.
|
| Other ratings considered |
|
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.
|
|
Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence.
|
|
Mixed effects: Evidence of inconsistent effects as demonstrated through either of the following criteria.
|
|
Potentially negative effects: Evidence of a negative effect with no overriding contrary evidence.
|
|
Negative effects: Strong evidence of a negative effect with no overriding contrary evidence.
|
|
1 For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. |
Appendix A5 Extent of evidence by domain
| Sample size | ||||
|---|---|---|---|---|
| Outcome domain | Number of studies | Schools | Students | Extent of evidence1 |
| Oral language | 1 | 17 | 254 | Small |
| Print knowledge | 1 | 17 | 254 | Small |
| Phonological processing | 1 | 17 | 254 | Small |
| Early reading/writing | 1 | 17 | 254 | Small |
| Cognition | 0 | 0 | 0 | na |
| Math | 0 | 0 | 0 | na |
|
na = not applicable/not studied 1 A rating of "medium to large" requires at least two studies and two schools across studies in one domain and a total sample size across studies of at least 350 students or 14 classrooms. Otherwise, the rating is "small." |
||||
