Skip Navigation

What Works Clearinghouse


Appendix A1.1 Study characteristics: Scientific Learning Corporation, 2004 (randomized controlled trial)

Characteristic Description
Study citation Scientific Learning Corporation. (2004). Improved language skills by children with low reading performance who used Fast ForWord Language: MAPS for learning. MAPS for Learning, 3 (1), 1–13.
Participants Four-hundred fifty-two academically at-risk students in kindergarten through fifth grade were randomly assigned, within grade and gender, to the treatment group (n=288) or the control group (n=164) at a 1.74: 1 ratio. The original study stated that 85 of the students were categorized as English language learners (53 in the treatment group and 32 in the control group).1 Follow-up correspondence with the developer of the intervention revealed that 98 students were identified as English language learners. Nine English language learning students were also identified as receiving services for special education and were not included in the analysis. Of the remaining 89 students, five from the comparison group and three from the intervention group had incomplete data. There were complete data for a total of 81 English language learning students (52 in the treatment group and 29 in the comparison group).
Setting The study was conducted in nine school districts in the United States. Elementary school teachers identified students as at-risk based on student difficulties with reading and language arts.
Intervention Participants used Fast ForWord Language, an adaptive computer-based training program based on acoustically modified speech and language training. Students were presented with seven exercises as computer games. Exercises began with acoustic reception and moved to more complex skills in syntactic and semantic aspects of language. The difficulty of each task was continuously adapted so that participants would get about 80% of the items correct. Participants used the program about 100 minutes a day, five days a week over an average of 39 days. Each participant worked on multiple 20-minute Fast ForWord Language training exercises during each session.
Comparison The control group used their regular curriculum. No information about the regular curriculum was provided.
Primary outcomes and measurement The study measure in the English language development domain was the Test of Auditory Comprehension of Language Revised Edition (TACL-R). (See Appendix A2.2 for a more detailed description of this outcome measure.) The study measure in the phonological awareness domain was the Phonological Awareness Test (PAT). The WWC review of English language learning interventions does not investigate phonological awareness, so results for this domain are not included in this report.
Teacher training No information about teacher training was provided. However, the teachers were speech-language or educational professionals.
1 Means and standard deviations for this subsample were not provided in the original study, so the WWC contacted the developer to obtain this information.

Top

Appendix A1.2 Study characteristics: Troia, 2004 (quasi-experimental design)

Characteristic Description
Study citation Troia, G. A. (2004). Migrant students with limited English proficiency: Can Fast ForWord Language make a difference in their language skills and academic achievement? Remedial and Special Education, 25 (6), 353–366.
Participants This study included 191 English language learning first- through sixth-grade students. Students from four schools were randomly assigned to the treatment or control group, while three schools created a matched comparison group for the treatment groups. The WWC could not separate effects for groups formed on a random and non-random (that is, those that used matching without random assignment) basis, so the WWC rated this study as a quasi-experimental design. All participants were migrant students (lived in the Unted States for six years on average) whose native language was Spanish, and their average age was 9.49 years old. There were slightly more boys (53%) than girls (47%) in the study and slightly more treatment group students (n=99) than comparison group students (n=92). A total of 168 students (90 treatment, 78 control) completed the Letter-Word posttest, and 167 students (89 treatment, 78 control) completed the Word Attack posttest.
Setting The study was conducted in seven schools from five rural school districts in central Washington state. This region has many agricultural communities that employ seasonal migrant workers, so the schools enroll a large number of migrant students.
Intervention Participants used Fast ForWord Language, an adaptive computer-based training program based on acoustically modified speech and language training. Students were presented with seven exercises as computer games. Exercises began with acoustic reception and moved on to more complex skills in syntactic and semantic aspects of language. The difficulty of each task was continuously adapted so that participants would get about 80% of the items correct. Participants used the Fast ForWord Language program about 100 minutes a day, five days a week for a minimum of four weeks. Each participant worked on multiple 20-minute Fast ForWord Language training exercises during each session.
Comparison The comparison group used their regular curriculum. No information about the regular curriculum was provided.
Primary outcomes and measurement The study measures in the reading achievement domain included the Woodcock-Johnson Psycho-Educational Battery-Revised Word Identification and Word Attack subtests. The study measures in the English language development domain were the Woodcock-Muñoz Language Survey, the Language Assessment Scales, and the Oral and Written Language Survey.1 (See Appendix A2.1 for a more detailed description of outcome measures.) The study measures in the phonological awareness domain included the Lindamood Auditory Conceptualization (LAC) Test, the Sound Blending subtest of the Woodcock-Johnson Psycho-Educational Battery-Revised, and the Experimental Rhyming and Segmentation Tests. The study measure in the social skill development domain was the Social Skills Rating System (SSRS). The WWC review of English language learning interventions does not investigate phonological awareness or social skill development, so results for these domains are not included in this report.
Teacher training No information about teacher training was provided, except that teachers were instructed not to provide the children with the correct responses. The primary role of the teacher present during the intervention was to troubleshoot any technical difficulties.
1 Two findings (the Language Assessment Scales and the Oral and Written Language Survey) are not included in Appendix A3 because of differential attrition rates between the intervention group and the comparison group. The Woodcock-Muñoz Language Survey was administered in those schools where the Language Assessment Scale was not used (20 students). The WWC could not determine whether this group was representative of the larger sample, so did not present these outcomes in these appendices.

Top

Appendix A2.1 Outcome measures in the reading achievement domain

Outcome measure Description
Woodcock-Johnson Psycho-Educational Battery Revised (WJ-R): Letter-Word Identification The Woodcock-Johnson Psycho-Educational Battery-Revised (Woodcock & Johnson, 1990) is made up of many subtests. The Letter-Word Identification subtest assesses basic reading skills by having participants read aloud as many letters and real words as possible in five seconds (as cited in Troia, 2004).
WJ-R: Word Attack The Word Attack subtest assesses basic reading skills by having participants read aloud a list of nonsense words (as cited in Troia, 2004).

Top

Appendix A2.2 Outcome measures in the English language development domain1

Outcome measure Description
Test of Auditory Comprehension of Language Revised Edition (TACL-R)2 The TACL-R (Carrow-Woolfolk, 1985) assesses the comprehension of spoken language and consists of three subtests, including word classes and relations, grammatical morphemes, and elaborated sentences (as cited in Scientific Learning Corporation, 2004).
1 The Scientific Learning Corporation included outcome measures on phonological awareness, but these were not included in this review because they were judged to fall outside the reading achievement and English language development domains.
2 Although limited information was provided on the TACL-R in the article by the Scientific Learning Corporation, the TACL-3 was normed on a standardization sample of more than 1,000 children. This information was found at www.proedinc.com/Scripts/prodView.asp?idProduct=1571.

Top

Appendix A3.1 Summary of study findings included in the rating for the reading achievement domain1

  Author's findings from the study  
  Mean outcome (standard deviation2) WWC calculations
Outcome measure Study sample Sample size (students) Fast ForWord Language group Comparison group Mean difference3 (Fast ForWord Language – comparison) Effect size4 Statistical significance5 (at α= 0.05) Improvement index6
Troia, 2004 (quasi-experimental design)
Woodcock-Johnson Psycho-Educational Battery Revised (WJ-R): Letter-Word Identification Grades 1–6 168 92.47 (14.89) 90.59 (18.02) 1.88 0.11 ns +5
WJ-R: Word Attack Grades 1–6 167 94.64 (17.25) 94.46 (15.85) 0.18 0.01 ns 0
Domain average7 for reading achievement 0.06 ns +3

ns = not statistically significant

1 This appendix reports findings considered for the effectiveness rating and the improvement index. Two findings (the Language Assessment Scale and the Oral and Written Language Survey) were not reported because of problematic attrition rates.
2 The standard deviation across all students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes.
3 Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group.
4 For an explanation of the effect size calculation, please see the Technical Details of WWC-Conducted Computations.
5 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. The level of statistical significance was calculated by the WWC and, where necessary, corrects for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See the Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Fast ForWord Language, no corrections for clustering or multiple comparisons were needed.
6 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results.
7 This row provides the study average, which is also the domain average in this case. The WWC-computed domain average effect size is a simple average rounded to two decimal places. The domain improvement index is calculated from the average effect size.

Top

Appendix A3.2 Summary of study findings included in the rating for the English language development domain1

  Author's findings from the study  
  Mean outcome (standard deviation2) WWC calculations
Outcome measure Study sample Sample size (students) Fast ForWord Language group Comparison group Mean difference3 (Fast ForWord Language – comparison) Effect size4 Statistical significance5 (at α= 0.05) Improvement index6
Scientific Learning Corporation, 2004 (randomized controlled trial)7
Test of Auditory Comprehension of Language-Revised (TACL-R) Grades K–5 81 45.21 (8.33) 37.93 (8.07) 7.28 0.88 Statistically significant +31
Domain average8 for English language development 0.88 Statistically significant +31

ns = not statistically significant

1 This appendix reports findings considered for the effectiveness rating and the improvement index.
2 The standard deviation across all students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes.
3 Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group.
4 For an explanation of the effect size calculation, please see the Technical Details of WWC-Conducted Computations.
5 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. The level of statistical significance was calculated by the WWC and, where necessary, corrects for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See the Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Fast ForWord Language, no corrections for clustering or multiple comparisons were needed.
6 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results.
7 The article did not include subsample data specifically for English language learning students, so the WWC obtained these data from the study author.
8 This row provides the study average, which is also the domain average in this case. The WWC-computed domain average effect size is a simple average rounded to two decimal places. The domain improvement index is calculated from the average effect size.

Top

Appendix A4.1 Fast ForWord Language rating for the reading achievement domain

The WWC rates interventions as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.1

For the outcome domain of reading achievement, the WWC rated Fast ForWord Language as having no discernible effects. It did not meet the criteria for positive effects, because it had only one study. In addition, it did not meet the criteria for the other ratings (potentially positive effects, mixed effects, potentially negative effects, and negative effects) because the single study that met WWC standards did not show statistically significant or substantively important effects.

Rating received

No discernible effects: No affirmative evidence of effects.

  • Criterion 1: None of the studies shows a statistically significant or substantively important effect, either positive or negative.

    Met. The WWC analysis found no statistically significant or substantively important effects in this domain.

Other ratings considered

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.

  • Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design.

    Not met. Fast ForWord Language had only one study in this domain meeting WWC evidence standards. Further, this study did not meet WWC evidence standards for a strong design and did not show statistically significant positive effects.

  • Criterion 2: No studies showing statistically significant or substantively important negative effects.

    Met. The WWC analysis found no statistically significant or substantively important negative effects in this domain.

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence.

  • Criterion 1: At least one study showing a statistically significant or substantively important positive effect.

    Not met. The WWC analysis found no statistically significant or substantively important positive effects in this domain.

  • Criterion 2: No studies showing a statistically significant or substantively important negative effect. Fewer or the same number of studies showing indeterminate effects than showing statistically significant or substantively important positive effects.

    Not met. The WWC analysis found no statistically significant or substantively important negative effects in this domain. Because one study showed indeterminate effects and no studies showed statistically significant or substantively important positive effects, Fast ForWord Language did not meet this criterion.

Mixed effects: Evidence of inconsistent effects as demonstrated through EITHER of the following:

  • Criterion 1: At least one study showing a statistically significant or substantively important positive effect. At least one study showing a statistically significant or substantively important negative effect, but no more such studies than the number showing a statistically significant or substantively important positive effect

    Not met. The WWC analysis found no statistically significant or substantively important effects in this domain.

OR
  • Criterion 2: At least one study showing a statistically significant or substantively important effect, and more studies showing an indeterminate effect than showing a statistically significant or substantively important effect.

    Not met. The WWC analysis found no statistically significant or substantively important effects in this domain.

Potentially negative effects: Evidence of a negative effect with no overriding contrary evidence

  • Criterion 1: At least one study showing a statistically significant or substantively important negative effect.

    Not met. The WWC analysis found no statistically significant or substantively important negative effects in this domain.

    • Criterion 2: No studies showing a statistically significant or substantively important positive effect, OR more studies showing statistically significant or substantively important negative effects than showing statistically significant or substantively important positive effects.

      Not met. The WWC analysis found no statistically significant or substantively important positive effects in this domain.

Negative effects: Strong evidence of a negative effect with no overriding contrary evidence.

  • Criterion 1: Two or more studies showing statistically significant negative effects, at least one of which met WWC evidence standards for a strong design.

    Not met. Fast ForWord Language had only one study reviewed by the WWC in this domain, and it showed no statistically significant effects.

  • Criterion 2: No studies showing statistically significant or substantively important positive effects.

    Met. The WWC analysis found no statistically significant or substantively important positive effects in this domain.

1 For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain level effect. The WWC also considers the size of the domain level effect for ratings of potentially positive effects. See the WWC Intervention Rating Scheme for a complete description.

Top

Appendix A5.1 Fast ForWord Language rating for the English language development domain

The WWC rates interventions as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.1

For the outcome domain of English language development, the WWC rated Fast ForWord Language as having potentially positive effects. It did not meet the criteria for positive effects, because it had only one study. The remaining ratings (mixed effects, no discernible effects, potentially negative effects, and negative effects) were not considered because Fast ForWord Language was assigned the highest applicable rating.

Rating received

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence.

  • Criterion 1: At least one study showing a statistically significant or substantively important positive effect.

    Met. One study reported a positive average effect size that was both statistically significant and substantively important.

  • Criterion 2: No studies showing a statistically significant or substantively important negative effect. Fewer or the same number of studies showing indeterminate effects than showing statistically significant or substantively important positive effects.

    Met. The WWC analysis found no indeterminate, statistically significant negative, or substantively important negative effects in this domain.

Other ratings considered

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.

  • Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design.

    Not met. Fast ForWord Language had only one study meeting WWC evidence standards and so did not meet this criterion. This one study did, however, use a strong design and showed statistically significant positive effects.

  • Criterion 2: No studies showing statistically significant or substantively important negative effects.

    Met. The WWC analysis found no statistically significant or substantively important negative effects in this domain.

1 For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain level effect. The WWC also considers the size of the domain level effect for ratings of potentially positive effects. See the WWC Intervention Rating Scheme for a complete description.

Top