Gatti, G. (2011). Pittsburgh, PA: Gatti Evaluation, Inc.
Reviewed: November 2015
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Please see the WWC summary of evidence for Successmaker.
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
Group Reading Assessment and Diagnostic Evaluation (GRADE): Overall Score |
Successmaker vs. Business as usual |
post intervention |
Grade 7;
|
54.56 |
52.70 |
No |
-- | ||
|
Group Reading Assessment and Diagnostic Evaluation (GRADE): Overall Score |
Successmaker vs. Business as usual |
post intervention |
Grade 5;
|
60.59 |
60.03 |
No |
-- | ||
| Show Supplemental Findings | |||||||||
|
Group Reading Assessment and Diagnostic Evaluation (GRADE): Overall Score |
Successmaker vs. Business as usual |
post intervention |
Grade 5: Hispanic subgroup;
|
57.43 |
55.48 |
No |
-- | ||
|
Group Reading Assessment and Diagnostic Evaluation (GRADE): Overall Score |
Successmaker vs. Business as usual |
post intervention |
Grade 5: Reduced-price lunch subgroup;
|
57.26 |
55.32 |
No |
-- | ||
|
Group Reading Assessment and Diagnostic Evaluation (GRADE): Overall Score |
Successmaker vs. Business as usual |
post intervention |
Grade 7: Male subgroup;
|
60.22 |
57.70 |
No |
-- | ||
|
Group Reading Assessment and Diagnostic Evaluation (GRADE): Overall Score |
Successmaker vs. Business as usual |
post intervention |
Grade 7: African American subgroup;
|
46.61 |
45.13 |
No |
-- | ||
|
Group Reading Assessment and Diagnostic Evaluation (GRADE): Overall Score |
Successmaker vs. Business as usual |
post intervention |
Grade 7: Female subgroup;
|
54.96 |
53.66 |
No |
-- | ||
|
Group Reading Assessment and Diagnostic Evaluation (GRADE): Overall Score |
Successmaker vs. Business as usual |
post intervention |
Grade 7: Reduced-price lunch subgroup;
|
47.50 |
46.08 |
No |
-- | ||
|
Group Reading Assessment and Diagnostic Evaluation (GRADE): Overall Score |
Successmaker vs. Business as usual |
post intervention |
Grade 5: Male subgroup;
|
61.28 |
60.31 |
No |
-- | ||
|
Group Reading Assessment and Diagnostic Evaluation (GRADE): Overall Score |
Successmaker vs. Business as usual |
post intervention |
Grade 5: African American subgroup;
|
51.15 |
50.65 |
No |
-- | ||
|
Group Reading Assessment and Diagnostic Evaluation (GRADE): Overall Score |
Successmaker vs. Business as usual |
post intervention |
Grade 5: Female subgroup;
|
60.01 |
59.80 |
No |
-- | ||
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
AIMSweb Reading Curriculum-Based Measurement |
Successmaker vs. Business as usual |
post intervention |
Grade 7;
|
164.47 |
165.37 |
No |
-- | ||
|
AIMSweb Reading Curriculum-Based Measurement |
Successmaker vs. Business as usual |
post intervention |
Grade 5;
|
152.20 |
156.01 |
No |
-- | ||
| Show Supplemental Findings | |||||||||
|
AIMSweb Reading Curriculum-Based Measurement |
Successmaker vs. Business as usual |
post intervention |
Grade 5: African American subgroup;
|
143.74 |
133.52 |
No |
-- | ||
|
AIMSweb Reading Curriculum-Based Measurement |
Successmaker vs. Business as usual |
post intervention |
Grade 7: Female subgroup;
|
170.84 |
170.97 |
No |
-- | ||
|
AIMSweb Reading Curriculum-Based Measurement |
Successmaker vs. Business as usual |
post intervention |
Grade 7: Male subgroup;
|
157.82 |
159.42 |
No |
-- | ||
|
AIMSweb Reading Curriculum-Based Measurement |
Successmaker vs. Business as usual |
post intervention |
Grade 7: Reduced-price lunch subgroup;
|
151.06 |
152.51 |
No |
-- | ||
|
AIMSweb Reading Curriculum-Based Measurement |
Successmaker vs. Business as usual |
post intervention |
Grade 7: African American subgroup;
|
150.93 |
153.28 |
No |
-- | ||
|
AIMSweb Reading Curriculum-Based Measurement |
Successmaker vs. Business as usual |
post intervention |
Grade 5: Male subgroup;
|
153.23 |
156.95 |
No |
-- | ||
|
AIMSweb Reading Curriculum-Based Measurement |
Successmaker vs. Business as usual |
post intervention |
Grade 5: Female subgroup;
|
151.71 |
155.23 |
No |
-- | ||
|
AIMSweb Reading Curriculum-Based Measurement |
Successmaker vs. Business as usual |
post intervention |
Grade 5: Reduced-price lunch subgroup;
|
138.40 |
141.69 |
No |
-- | ||
|
AIMSweb Reading Curriculum-Based Measurement |
Successmaker vs. Business as usual |
post intervention |
Grade 7: Hispanic subgroup;
|
146.74 |
149.71 |
No |
-- | ||
|
AIMSweb Reading Curriculum-Based Measurement |
Successmaker vs. Business as usual |
post intervention |
Grade 5: Hispanic subgroup;
|
139.32 |
145.04 |
No |
-- | ||
Characteristics of study sample as reported by study author.
| Black | 16% | |
| Other or unknown | 10% | |
| White | 50% |
| Hispanic | 24% | |
| Not Hispanic or Latino | 76% |
The study was conducted in eight urban and suburban school districts located in seven states: Arizona, California, Indiana, Kansas, Michigan, Missouri, and Texas.
Nine schools participated in the study. The schools had to meet the following conditions: they had to (1) have no prior exposure to SuccessMaker®; (2) have at least two teachers per study grade level; (3) be geographically diverse; (4) agree that teachers would uphold random assignment; and (5) agree that intervention group classrooms would have their students use SuccessMaker® at least 1 hour per week. English language arts classes (or sections) within the selected schools and grade levels were randomly assigned to either the intervention or the business-as-usual comparison group.7 The fifth-grade sample included 16 classrooms implementing SuccessMaker® and 14 implementing the school’s regular English language arts program. The seventh-grade sample included 11 classrooms implementing SuccessMaker® and nine implementing the school’s regular English language arts program. Of the 641 fifth-grade students that participated in the study, 342 received SuccessMaker® and 299 received the school’s regular English language arts program. Of the 453 seventhgrade students that participated in the study, 254 received SuccessMaker® and 199 received the school’s regular English language arts program. About 48% of the total sample were male, 39% were minority (about 23.6% Hispanic and 15.8% African American), and 100% received free or reduced-price lunch.
SuccessMaker® is an adaptive, computer-based learning program which includes an instructional management system, formative assessments, a progress reporting system, and individualized reading curriculum resources for elementary and middle school instruction. For this study, the program was typically implemented with the entire class in a computer laboratory during the regular reading instruction time. Intervention group students were expected to use the software for at least 1 hour each week. Over the course of the school year, most teachers went to the lab two or three times a week, with a median time of 22 hours for fifth-grade classes and 18 hours for seventh-grade classes.
Comparison classes received the business-as-usual English language arts instruction, which generally did not involve computer-based instruction. The majority of students in fifth grade (62%) received instruction from four widely-used published reading programs, while the remainder received instruction from non-published teacher-developed curricula. In contrast, the majority of students in seventh grade (63%) received instruction from non-published, largely teacher-created curricula, while the remainder received instruction from three different widely-used published literacy programs.
Assessments were administered at the onset of the intervention and in the last month of the school year. Outcomes in the comprehension domain included Group Reading Assessment and Diagnostic Evaluation (GRADE) Overall Score and three subtest scores of the GRADE; Passage Comprehension, Sentence Comprehension, and Vocabulary. One measure in the reading fluency domain was administered, the AIMSweb Reading Curriculum-Based Measurement. Supplemental findings for the three GRADE subtest outcomes and for subgroups of students on the GRADE overall score and the AIMSweb outcome are presented in Appendix D.8 The supplemental findings do not factor into the intervention’s rating of effectiveness. For a more detailed description of these outcome measures, see Appendix B. The study also included a researcher-designed student reading attitude survey. However, this outcome is not eligible for review based on the Adolescent Literacy review protocol (version 3.0).
Teachers received 1 day of initial training, some before school started and some in the second month of the school year. The teachers also received a 3-hour follow-up training and additional assistance from Pearson when needed.
An indicator of the effect of the intervention, the improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention.
For more, please see the WWC Glossary entry for improvement index.
An outcome is the knowledge, skills, and attitudes that are attained as a result of an activity. An outcome measures is an instrument, device, or method that provides data on the outcome.
A finding that is included in the effectiveness rating. Excluded findings may include subgroups and subscales.
The sample on which the analysis was conducted.
The group to which the intervention group is compared, which may include a different intervention, business as usual, or no services.
The timing of the post-intervention outcome measure.
The number of students included in the analysis.
The mean score of students in the intervention group.
The mean score of students in the comparison group.
The WWC considers a finding to be statistically significant if the likelihood that the finding is due to chance alone, rather than a real difference, is less than five percent.
The WWC reviews studies for WWC products, Department of Education grant competitions, and IES performance measures.
The name and version of the document used to guide the review of the study.
The version of the WWC design standards used to guide the review of the study.
The result of the WWC assessment of the study. The rating is based on the strength of evidence of the effectiveness of the intervention. Studies are given a rating of Meets WWC Design Standards without Reservations, Meets WWC Design Standards with Reservations, or >Does Not Meet WWC Design Standards.
A related publication that was reviewed alongside the main study of interest.
Study findings for this report.
Based on the direction, magnitude, and statistical significance of the findings within a domain, the WWC characterizes the findings from a study as one of the following: statistically significant positive effects, substantively important positive effects, indeterminate effects, substantively important negative effects, and statistically significant negative effects. For more, please see the WWC Handbook.
The WWC may review studies for multiple purposes, including different reports and re-reviews using updated standards. Each WWC review of this study is listed in the dropdown. Details on any review may be accessed by making a selection from the drop down list.
Tier 1 Strong indicates strong evidence of effectiveness,
Tier 2 Moderate indicates moderate evidence of effectiveness, and
Tier 3 Promising indicates promising evidence of effectiveness,
as defined in the
non-regulatory guidance for ESSA
and the regulations for ED discretionary grants (EDGAR Part 77).
Resources
Newsflash
Contact Us
Facebook
Twitter
YouTube