|
|
Thirty-six studies reviewed by the WWC investigated the effects of SuccessMaker®. Three studies (Beattie, 2000; Campbell, 2000; Gallagher, 1996), one randomized controlled trial and two quasi-experimental designs, meet WWC evidence standards with reservations. Of the remaining studies, 33 studies do not meet WWC evidence standards or eligibility screens.
Beattie (2000) conducted a randomized controlled trial of middle and middle-high school students in suburban northern Virginia. Students with language deficits, ranging in age from 11 to 16 years, were randomly assigned by computer-generated procedures to one of five groups (Appendix A1.1 provides more details about these groups). The WWC based its effectiveness ratings on findings from comparisons of 14 students that received SuccessMaker® and 12 control group students that received regular reading instruction. Although these analytic samples were shown to be equivalent at baseline, differential attrition between groups led to the study’s rating of meets standards with reservations. The study reported student outcomes after two months of program implementation.
Campbell (2000) conducted a quasi-experiment that examined the effects of SuccessMaker® on students in upper elementary grades in Alabama. The schools that used SuccessMaker® and traditional instruction (Accelerated Reader in conjunction with a basal reader) were matched to schools that used only traditional instruction based on the intellectual ability, poverty level, and demographic characteristics of students in each school. The WWC based its effectiveness ratings on findings for grade 4 students: 143 students in four intervention schools and 186 students in four comparison schools. The study reported student outcomes after one year of program implementation.
Gallagher (1996) conducted a quasi-experiment that examined the effects of SuccessMaker® on at-risk students in grades 4–7 at an inner city elementary school in Chicago, IL. Students in each classroom were sorted by either reading achievement test score or student identification number (ID), and then alternately assigned to treatment and control groups.7 The WWC based its effectiveness ratings on findings from comparisons of the 48 students that received two reading components of SuccessMaker® (Readers Workshop and Reading Adventures) and the 47 control group students that received math components of SuccessMaker®. Both groups received their regular reading curriculum outside of the SuccessMaker® instruction. The study reported student outcomes after six weeks of program implementation.
Extent of evidence
The WWC categorizes the extent of evidence in each domain as small or medium to large (see the WWC Procedures and Standards Handbook, Appendix G). The extent of evidence takes into account the number of studies and the total sample size across the studies that meet WWC evidence standards with or without reservations.8
The WWC considers the extent of evidence for SuccessMaker® to be small for alphabetics, reading fluency, and general literacy achievement, and medium to large for comprehension.
7 The authors either sorted the students by student identification numbers (ID) or Iowa Test of Basic Skills (ITBS) reading comprehension scores, and then assigned students to groups in an alternating fashion, but it is not clear which method was used from the text. If they sorted by student ID and
then assigned students to groups, the assignment might be functionally random, but if they sorted by ITBS score, and always assigned students in
an alternating fashion (starting with the treatment group, for example), the groups would be imbalanced, because they were always assigning the lower
(or higher) scores to the treatment group. The WWC could not confirm that the assignment was truly random, as the authors had not responded to the
WWC query at the time of publication of this review.
8 The extent of evidence categorization was developed to tell readers how much evidence was used to determine the intervention rating, focusing on
the number and size of studies. Additional factors associated with a related concept—external validity, such as the students’ demographics and the
types of settings in which studies took place—are not taken into account for the categorization. Information about how the extent of evidence rating
was determined for SuccessMaker® is in Appendix A6.
|Institute of Education Sciences
Adolescent Literacy