|
|
The WWC review of interventions for SuccessMaker® addresses student outcomes in four domains: alphabetics, reading fluency, comprehension, and general literacy achievement. The studies included in this report cover all four domains. The findings below present the authors’ estimates and WWC-calculated estimates of the size and the statistical significance of the effects of SuccessMaker® on students.9
Alphabetics. Beattie (2000) did not find statistically significant effects of SuccessMaker® on alphabetics measures, including the Woodcock-Johnson subtests of Letter-Word Identification, Word Attack, and Auditory Processing, and the Wide Range Achievement Spelling subtest. The WWC-calculated average effect size across the four outcomes was not large enough to be considered substantively important according to WWC criteria (that is, an effect size at least 0.25).10
Reading fluency. Beattie (2000) did not find a statistically significant effect of SuccessMaker® on the Gray Oral Reading Test, and the effect was not large enough to be considered substantively important according to WWC criteria.
Comprehension. Beattie (2000) did not find statistically significant effects of SuccessMaker® on the Woodcock-Johnson Passage Comprehension subtest, but the effect size was large enough to be considered substantively important according to WWC criteria (that is, an effect size at least 0.25). Campbell (2000) did not find statistically significant effects of SuccessMaker® on either measure of comprehension examined (the Stanford Achievement Reading Vocabulary and Reading Comprehension subtests). The WWC-calculated average effect size across the two outcomes was not large enough to be considered substantively important according to WWC criteria. Gallagher (1996) found a statistically significant effect of SuccessMaker® on the reading comprehension subtest of the Iowa Test of Basic Skills. The WWC found that the effect was not statistically significant but large enough to be considered substantively important according to WWC criteria.11
General literacy achievement. Beattie (2000) did not find statistically significant effects of SuccessMaker® on the Clinical Evaluation of Language Fundamentals Receptive Language Score, but the effect size was large enough to be considered substantively important according to WWC criteria (that is, an effect size at least 0.25).
The WWC rates the effects of an intervention in a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative. The rating of effectiveness takes into account four factors: the quality of the research design, the statistical significance of the findings, the size of the difference between participants in the intervention and the comparison conditions, and the consistency in findings across studies (see the WWC Procedures and Standards Handbook, Appendix E).
9 The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms
or schools and for multiple comparisons. For an explanation, see the WWC Tutorial on Mismatch. For the formulas the WWC used to calculate
the statistical significance, see WWC Procedures and Standards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D for multiple comparisons. In the case of Beattie (2000), a correction for multiple comparisons
was needed, so the significance levels may differ from those reported in the original study. In the case of Campbell (2000), corrections for clustering and
multiple comparisons were needed, so the significance levels may differ from those reported in the original study. In the case of Gallagher (1996), no
corrections for clustering or multiple comparisons were needed.
10 The WWC computes an average effect size (ES) as a simple average of the ESs across all individual findings within the study domain. For information on how the WWC characterizes study effects, consult the WWC Procedures and Standards Handbook, Appendix E.
11 The study is not consistent in reporting the numbers of students allocated to treatment and control groups. The WWC calculated the groups’ sample sizes, means, and standard deviations from the raw data presented in the study appendices.
|Institute of Education Sciences
Adolescent Literacy