Putting Computerized Instruction to the Test: A Randomized Evaluation of a ''Scientifically Based'' Reading Program
Rouse, Cecilia Elena; Krueger, Alan B. (2004). Economics of Education Review, v23 n4 p323-338. Retrieved from: https://eric.ed.gov/?id=EJ730375
-
examining454Students, grades3-6
Grant Competition
Review Details
Reviewed: September 2016
- Grant Competition (findings for Fast ForWord®)
- Randomized Controlled Trial
- Meets WWC standards without reservations
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
State Standardized Reading Test |
Fast ForWord® vs. Business as usual |
8 Weeks |
Full sample;
|
44.57 |
43.03 |
No |
-- |
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
CELF-3 |
Fast ForWord® vs. Business as usual |
8 Weeks |
Grade: 4;
|
32.09 |
31.01 |
No |
-- |
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Success for All assessment |
Fast ForWord® vs. Business as usual |
8 Weeks |
Full sample;
|
4.11 |
4.03 |
No |
-- |
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Female: 52%
Male: 48% -
Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
Northeast
-
Race Black 27% -
Ethnicity Hispanic 65% Not Hispanic or Latino 35%
Study Details
Setting
The study took place in four elementary schools in one urban school district in the northeast. The district has roughly 20,000 students with the majority of students being ethnic minorities: 40% African American and 50% Hispanic. Nearly 70% of the students qualify for free/reduced lunch program and 56% speak a language other than English at home. Since Fast ForWord is a type of pull-out instructional program, participants in intervention and comparison groups were not from intact classrooms.
Study sample
There were 49% female participants in the intervention group, and 55% of the comparison group were female. 27% of the intervention and comparison groups were African-American. 64% of the intervention group was Latino, and 67% of the comparison group was Latino. In the district, almost 70% of the students qualify for free or reduced price lunch, but this information was not provided for the intervention and comparison groups. In the intervention group, 15% of students were identified as being in special education, and 16% of the students in the comparison group were identified as special education.
Intervention Group
The FFW is a computer software program distributed by the Scientific Learning Corporation (SLC). The program uses computer games to try and retrain the brain. The games work by slowing and magnifying the acoustic changes within normal speech. The program is comprised of three programs, including FFW language, FFW language-to-reading, and FFW reading. FFW language helps to develop oral language skills that are foundational skills for reading. The FFW language-to-reading helps students make the connection between spoken and written language. FFW Reading helps build readings skills, such as word recognition and fluency. The program goes 6 - 8 weeks and students work between 90-100 minutes per day, five days a week. Students start at a basic level and progress through proficiency as they acquire skills. A student is said to have completed the program after at least 20 days and has completed at least 80% of the 5 - 7 program games. Intervention group participants had regular reading instruction offered by the school, and received an additional 90-100 minutes per day with the Fast ForWord reading computer games (p. 324). Students in the intervention group missed a variety of activities during the Fast ForWord intervention time, specified in Table 1 (p. 327). These missed activities included homeroom, math, science, language arts, specials (art, music, gym). All students were in Success for All, a whole school reform adopted by most schools in the district at the time of the study (p. 327). Study participation was broken into two ""flights"" as illustrated in Table 3 (p. 332). Students had a potential for 37 training days in Flight 1, and about 30 training days in Flight 2. In Table A2, the authors reported that 268 students were in Flight 1, and 244 students were in Flight 2 among the randomized sample size.
Comparison Group
Comparison group participants had regular reading instruction offered by the school, and participated in a variety of activities that the intervention group students missed while doing Fast ForWord. Table 1 (p. 327) detailed the types of activities in which the comparison group participated, and the proportion of the students in the comparison group who were affected. These missed activities included homeroom, math, science, language arts, specials (art, music, gym).
Support for implementation
The school district supported implementation of Fast ForWard in the four schools by licensing 30 computers per school for 1 year at a cost of $30,000. In two schools, the intervention took place in computers in the library, and in two other schools, the intervention took place in a computer lab (p. 326, note 3). The training package cost $100 per site, and any adult who completed the training could supervise the students (did not have to be a certified teacher). The schools were equipped with sufficient power to run the software, color printers, head phones, Y-connectors, a quiet space for students to work, and an adult supervisor trained in the program (p. 326). Study researchers conducted site visits to ensure that computer labs were set up properly and the instructors were adequately trained (p. 327). The Scientific Learning Corporation provided support at the beginning of the evaluation, conducted site visits, and provided telephone support (p. 327).
Fast ForWord® Intervention Report - Beginning Reading
Review Details
Reviewed: March 2013
- The study is ineligible for review because it does not disaggregate findings for the age or grade range specified in the protocol.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Please see the WWC summary of evidence for Fast ForWord®.
Findings
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Study sample characteristics were not reported.Fast ForWord® Intervention Report - Adolescent Literacy
Review Details
Reviewed: August 2010
- Randomized Controlled Trial
- Meets WWC standards without reservations
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Please see the WWC summary of evidence for Fast ForWord®.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Clinical Evaluation of Language Fundamentals- Third Edition (CELF-3): Receptive Language |
Fast ForWord® vs. Business as usual |
Posttest |
grades 3-6;
|
31.70 |
31.01 |
No |
-- | |
|
State Standardized Reading Test |
Fast ForWord® vs. Business as usual |
Posttest |
grades 3-6;
|
44.18 |
43.03 |
No |
-- |
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
Northeast
-
Race Black 42% -
Ethnicity Hispanic 51% Not Hispanic or Latino 49%
Study Details
Setting
The study took place in four schools in an urban district in the northeastern United States. Forty percent of the district’s students were African-American and more than 50% were Hispanic. Almost 70% of students in the district qualified for the free or reduced-price lunch program, and 56% of the district’s students spoke a language other than English at home. The authors describe test scores in these schools as well below average and note that schools in the district adopted a whole-school reform, Success for All.
Study sample
Groups were formed through a multistep process. Authors first identified an eligible population of students from four schools within one urban school district, focusing on third- to sixth-grade students who scored in the bottom 20% on the state’s standardized reading test administered in the 2001–02 school year. Consent letters were sent to these students’ parents. Principals in the schools were asked to identify students who could not sit through the daily 90- to 100-minute use of Fast ForWord®, those who had transferred to another school, and those students who might otherwise be unavailable (family away on long trip, for example). The remaining students were randomly assigned to either the treatment or control group, within each grade and school. In all, 237 students in the Fast ForWord® group and 217 students in the comparison group were included in the analysis sample.
Intervention Group
Fast ForWord® was primarily an add-on to regular reading instruction. In three schools, students in the treatment condition were pulled out of their regular classroom instruction for 90–100 minutes of computerized Fast ForWord® instruction per day and, in one school, they used Fast ForWord® for that same amount of time before or after school. Each school had to find a way to fit the use of Fast ForWord® into its unique schedule. In no case were students taken out of Success for All. The study reported students’ outcomes after six to eight weeks of program implementation
Comparison Group
The control group continued to receive the standard curriculum being used in district schools. Because the Fast ForWord® students used Fast ForWord® either during subjects such as math, science, language arts, special subjects (such as art, music, or gym), or homeroom, or—in the case of one school—before or after school, the counterfactual condition for the control group students was mixed.
Outcome descriptions
For both the pretest and posttest, the authors administered the Success for All assessment, the Clinical Evaluation of Language Fundamentals–Third Edition (the receptive portion and the Listening to Paragraph supplemental test), and a state standardized reading test (the authors did not indicate which state). For a more detailed description of test outcome measures, see Appendix A2.4.
Support for implementation
Fast ForWord® staff provided training for Fast ForWord® instructors (those interacting with students) at the beginning of the study. Phone support was also provided for the duration of the study. Detailed information on the training of instructors was not provided.
Additional Sources
In the case of multiple manuscripts that report on one study, the WWC selects one manuscript as the primary citation and lists other manuscripts that describe the study as additional sources.
-
Rouse, C. E., Krueger, A. B., & Markman, L. (2004). Putting computerized instruction to the test: A randomized evaluation of a “scientifically based” reading program. Cambridge, MA: National Bureau of Economic Research.
An indicator of the effect of the intervention, the improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention.
For more, please see the WWC Glossary entry for improvement index.
An outcome is the knowledge, skills, and attitudes that are attained as a result of an activity. An outcome measures is an instrument, device, or method that provides data on the outcome.
A finding that is included in the effectiveness rating. Excluded findings may include subgroups and subscales.
The sample on which the analysis was conducted.
The group to which the intervention group is compared, which may include a different intervention, business as usual, or no services.
The timing of the post-intervention outcome measure.
The number of students included in the analysis.
The mean score of students in the intervention group.
The mean score of students in the comparison group.
The WWC considers a finding to be statistically significant if the likelihood that the finding is due to chance alone, rather than a real difference, is less than five percent.
The WWC reviews studies for WWC products, Department of Education grant competitions, and IES performance measures.
The name and version of the document used to guide the review of the study.
The version of the WWC design standards used to guide the review of the study.
The result of the WWC assessment of the study. The rating is based on the strength of evidence of the effectiveness of the intervention. Studies are given a rating of Meets WWC Design Standards without Reservations, Meets WWC Design Standards with Reservations, or >Does Not Meet WWC Design Standards.
A related publication that was reviewed alongside the main study of interest.
Study findings for this report.
Based on the direction, magnitude, and statistical significance of the findings within a domain, the WWC characterizes the findings from a study as one of the following: statistically significant positive effects, substantively important positive effects, indeterminate effects, substantively important negative effects, and statistically significant negative effects. For more, please see the WWC Handbook.
The WWC may review studies for multiple purposes, including different reports and re-reviews using updated standards. Each WWC review of this study is listed in the dropdown. Details on any review may be accessed by making a selection from the drop down list.
Tier 1 Strong indicates strong evidence of effectiveness,
Tier 2 Moderate indicates moderate evidence of effectiveness, and
Tier 3 Promising indicates promising evidence of effectiveness,
as defined in the
non-regulatory guidance for ESSA
and the regulations for ED discretionary grants (EDGAR Part 77).