Effectiveness of Reading and Mathematics Software Products: Findings From Two Student Cohorts. NCEE 2009-4041
Campuzano, Larissa; Dynarski, Mark; Agodini, Roberto; Rall, Kristina (2009). National Center for Education Evaluation and Regional Assistance. Retrieved from: https://eric.ed.gov/?id=ED504657
-
examining2,588Students, grades1-12
Cognitive Tutor Algebra I Intervention Report - Secondary Mathematics
Review Details
Reviewed: June 2016
- Randomized Controlled Trial
- Meets WWC standards with reservations
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Please see the WWC summary of evidence for Cognitive Tutor Algebra I.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Educational Testing Service (ETS) End-of-Course Algebra Test |
Cognitive Tutor Algebra I vs. Business as usual |
9 Months |
Grades 8 and 9;
|
32.39 |
35.31 |
No |
-- |
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Female: 50%
Male: 50% -
Urban
Study Details
Setting
This 2-year congressionally-mandated study of education technology included several algebra interventions, including Cognitive Tutor® Algebra I. Findings from the first year of the study pooled all of the algebra interventions together. Therefore, this WWC intervention report focuses on findings from the second year, in which results are reported separately by intervention. In the second year of the study, Cognitive Tutor® Algebra I was implemented in nine schools in four districts. Districts were located in urban and urban fringe areas. Within each of the nine study schools, teachers were randomly assigned to the intervention and comparison groups prior to the first year of the study. Teachers maintained their assignment in the second year of the study.
Study sample
Students in the year one and two combined sample were 51 percent female for the intervention group and 46 percent female for the comparison group. The average age of students in the imputed analytic sample was 14.93 for both study groups. No other information about sample characteristics for either the full sample of the second-year sample is provided in the report.
Intervention Group
Students in the intervention group were taught using Cognitive Tutor® Algebra I as their core math curriculum. The curriculum covered proportional reasoning, solving linear equations and inequalities, solving systems of linear equations, analyzing data, and using polynomial functions, powers, and exponents. Teachers were in their second year of implementing Cognitive Tutor® Algebra I, while students were experiencing the curriculum for the first time. According to the study authors, students in the study used the computer portion of the curriculum for an average of 1,840 minutes during 18 weeks.
Comparison Group
Students in the comparison classes received traditional algebra instruction using their district’s standard algebra curriculum. The comparison curricula were not specified and may have varied across districts and schools.
Support for implementation
Within each district, teachers in the intervention group received 4 days of initial training from the publisher in the summer of 2004. They were trained on classroom management and the curriculum, and provided with opportunities to practice using the product. An unspecified amount of phone and email support was provided throughout the study.
Additional Sources
In the case of multiple manuscripts that report on one study, the WWC selects one manuscript as the primary citation and lists other manuscripts that describe the study as additional sources.
-
Dynarski, M., Agodini, R., Heaviside, S., Novak, T., Carey, N., Campuzano, L., ... Sussex, W. (2007). Effectiveness of reading and mathematics software products: Findings from the first student cohort: Report to Congress. Washington, DC: U.S. Department of Education, Institute of Education Sciences.
Effectiveness of Reading and Mathematics Software Products: Findings From Two Student Cohorts. NCEE 2009-4041
Review Details
Reviewed: March 2010
- Quick Review (85 KB) (findings for Larson Algebra 1)
- Randomized Controlled Trial
- Meets WWC standards without reservations
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Absence of conflict of interest: This study was conducted by staff from Mathematica Policy Research. Therefore, Mathematica reviewers were not involved in the WWC review of this study.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Larson Algebra 1 |
Larson Algebra 1 vs. Business as usual |
Spring test |
Grade 9;
|
30.55 |
32.76 |
No |
-- |
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Rural, Suburban, Urban
-
Race Asian 4% Black 39% White 30% -
Ethnicity Hispanic 28% Not Hispanic or Latino 62%
Quick Review
Review Details
Reviewed: March 2010
- Quick Review (findings for Larson Pre-Algebra)
- Randomized Controlled Trial
- Meets WWC standards without reservations
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Absence of conflict of interest: This study was conducted by staff from Mathematica Policy Research. Therefore, Mathematica reviewers were not involved in the WWC review of this study.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Larson Pre-Algebra |
Larson Pre-Algebra vs. Business as usual |
Spring test NCE |
Grade 6;
|
53.42 |
52.59 |
Yes |
|
|
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Rural, Suburban, Urban
-
Race Asian 4% Black 39% White 30% -
Ethnicity Hispanic 28% Not Hispanic or Latino 62%
Effectiveness of Reading and Mathematics Software Products: Findings From Two Student Cohorts. NCEE 2009-4041
Review Details
Reviewed: March 2010
- Quick Review (85 KB) (findings for Cognitive Tutor Algebra I)
- Randomized Controlled Trial
- Meets WWC standards without reservations
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Absence of conflict of interest: This study was conducted by staff from Mathematica Policy Research. Therefore, Mathematica reviewers were not involved in the WWC review of this study.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Cognitive Tutor |
Cognitive Tutor Algebra I vs. Business as usual |
Spring test |
Grade 9;
|
38.64 |
38.37 |
No |
-- |
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Rural, Suburban, Urban
-
Race Asian 4% Black 39% White 30% -
Ethnicity Hispanic 28% Not Hispanic or Latino 62%
Quick Review
Review Details
Reviewed: March 2010
- Quick Review (findings for PLATO Achieve Now)
- Randomized Controlled Trial
- Meets WWC standards without reservations
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Absence of conflict of interest: This study was conducted by staff from Mathematica Policy Research. Therefore, Mathematica reviewers were not involved in the WWC review of this study.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Plato Achieve Now |
PLATO Achieve Now vs. Business as usual |
Spring test NCE |
Grade 6;
|
46.06 |
50.67 |
No |
-- |
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Rural, Suburban, Urban
-
Race Asian 4% Black 39% White 30% -
Ethnicity Hispanic 28% Not Hispanic or Latino 62%
Quick Review
Review Details
Reviewed: March 2010
- Quick Review (findings for LeapTrack Assessment & Instruction System)
- Randomized Controlled Trial
- Meets WWC standards without reservations
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Absence of conflict of interest: This study was conducted by staff from Mathematica Policy Research. Therefore, Mathematica reviewers were not involved in the WWC review of this study.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
LeapTrack |
LeapTrack Assessment & Instruction System vs. Business as usual |
Spring test NCE |
Grade 4;
|
45.31 |
45.95 |
No |
-- |
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Rural, Suburban, Urban
-
Race Asian 4% Black 39% White 30% -
Ethnicity Hispanic 28% Not Hispanic or Latino 62%
Quick Review
Review Details
Reviewed: March 2010
- Quick Review (findings for Academy of READING®)
- Randomized Controlled Trial
- Meets WWC standards without reservations
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Absence of conflict of interest: This study was conducted by staff from Mathematica Policy Research. Therefore, Mathematica reviewers were not involved in the WWC review of this study.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Academy of Reading |
Academy of READING® vs. Business as usual |
Spring test NCE |
Grade 4;
|
38.63 |
41.45 |
No |
-- |
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Rural, Suburban, Urban
-
Race Asian 4% Black 39% White 30% -
Ethnicity Hispanic 28% Not Hispanic or Latino 62%
Quick Review
Review Details
Reviewed: March 2010
- Quick Review (findings for Waterford Early Reading Program)
- Randomized Controlled Trial
- Meets WWC standards without reservations
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Absence of conflict of interest: This study was conducted by staff from Mathematica Policy Research. Therefore, Mathematica reviewers were not involved in the WWC review of this study.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Waterford Early Reading Program |
Waterford Early Reading Program vs. Business as usual |
Spring test NCE |
Grade 1;
|
49.83 |
49.21 |
No |
-- |
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Rural, Suburban, Urban
-
Race Asian 4% Black 39% White 30% -
Ethnicity Hispanic 28% Not Hispanic or Latino 62%
Quick Review
Review Details
Reviewed: March 2010
- Quick Review (findings for PLATO Focus)
- Randomized Controlled Trial
- Meets WWC standards without reservations
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Absence of conflict of interest: This study was conducted by staff from Mathematica Policy Research. Therefore, Mathematica reviewers were not involved in the WWC review of this study.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Plato Focus |
PLATO Focus vs. Business as usual |
Spring test NCE |
Grade 1;
|
51.15 |
50.40 |
No |
-- |
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Rural, Suburban, Urban
-
Race Asian 4% Black 39% White 30% -
Ethnicity Hispanic 28% Not Hispanic or Latino 62%
Quick Review
Review Details
Reviewed: March 2010
- Quick Review (findings for Headsprout Early Reading)
- Randomized Controlled Trial
- Meets WWC standards without reservations
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Absence of conflict of interest: This study was conducted by staff from Mathematica Policy Research. Therefore, Mathematica reviewers were not involved in the WWC review of this study.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Headsprout Early Reading |
Headsprout Early Reading vs. Business as usual |
Spring test NCE |
Grade 1;
|
55.24 |
57.10 |
No |
-- |
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Rural, Suburban, Urban
-
Race Asian 4% Black 39% White 30% -
Ethnicity Hispanic 28% Not Hispanic or Latino 62%
Quick Review
Review Details
Reviewed: March 2010
- Quick Review (findings for Destination Reading)
- Randomized Controlled Trial
- Meets WWC standards without reservations
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Absence of conflict of interest: This study was conducted by staff from Mathematica Policy Research. Therefore, Mathematica reviewers were not involved in the WWC review of this study.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Destination Reading |
Destination Reading vs. Business as usual |
Spring test NCE |
Grade 1;
|
50.82 |
49.13 |
No |
-- |
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Rural, Suburban, Urban
-
Race Asian 4% Black 39% White 30% -
Ethnicity Hispanic 28% Not Hispanic or Latino 62%
PLATO Intervention Report - Middle School Math
Review Details
Reviewed: March 2010
- Randomized Controlled Trial
- Meets WWC standards without reservations
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Please see the WWC summary of evidence for PLATO.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Math NCE |
PLATO vs. Business as usual |
2004-2006 |
Grade 6;
|
50.09 |
50.67 |
No |
-- |
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Female: 53%
Male: 47% -
Rural, Suburban, Urban
-
Race Black 40% -
Ethnicity Hispanic 42% Not Hispanic or Latino 58%
Study Details
Setting
The study took place in 13 schools in three districts in multiple states across the country.
Study sample
The study sample included 1,037 sixth-grade students (547 PLATO® Achieve Now; 490 control) taught by 39 teachers (21 PLATO® Achieve Now; 18 control) in 13 schools across three districts in multiple states across the country during the 2004–05 and 2005–06 school years. Of the study sample, approximately 53% were female (52% PLATO® Achieve Now and 55% control), 74% received free or reduced-price lunch (not reported by intervention status), 42% were Hispanic (not reported by intervention status), and 40% were African-American (not reported by intervention status). Approximately 80% of the teachers in the study were female (81% PLATO® Achieve Now; 78% control) with an average of 11 years of teaching experience (9 years PLATO® Achieve Now; 13 years control) and 33% of whom obtained a master’s degree (24% PLATO® Achieve Now; 44% control).
Intervention Group
Students were taught using PLATO® Achieve Now during the 2004–05 and/or 2005–06 school years. PLATO® Achieve Now supplemented standard mathematics instruction for the treatment group. According to the study authors, PLATO® Achieve Now students used the product for independent practice and reinforcement of math skills. Students worked at their own pace on activities identified by the teacher. According to the authors, the recommended usage is 30 minutes per day, four days per week, for at least 10 weeks.
Comparison Group
Comparison students were taught in traditional classes, with the teachers utilizing any technology products already available to them.
Outcome descriptions
The primary outcome measures in Year 2 of the study were the Stanford Achievement Test–Tenth Edition (SAT–10), the Iowa Test of Basic Skills (ITBS), and the New Mexico Standards Based Assessment (NMSBA). Districts in this study used at least one of these three exams as the outcome measure to obtain pretest and/or posttest scores. The study authors converted the scale scores from these tests to normal curve equivalent (NCE) scores with a range of 1 to 99 and an average of 50 to standardize the measures across tests and cohorts. For a more detailed description of these outcome measures, see Appendix A2.
Support for implementation
Vendor training sessions generally took place in host districts, and sometimes host schools, during the summer or early fall of 2004. The initial training lasted about 6 hours and varied by product from 4 hours to about 8 hours. Vendors delivered ongoing support in several modes. Product representatives visited teachers; vendors also provided support through email, telephone help desks, and additional training at schools.
Additional Sources
In the case of multiple manuscripts that report on one study, the WWC selects one manuscript as the primary citation and lists other manuscripts that describe the study as additional sources.
-
Dynarski, M., Agodini, R., Heaviside, S., Novak, T., Carey, N., Campuzano, L., ... Sussex, W. (2007). Effectiveness of reading and mathematics software products: Findings from the first student cohort: Report to Congress. Washington, DC: U.S. Department of Education, Institute of Education Sciences.
Headsprout Early Reading Intervention Report - Early Childhood Education
Review Details
Reviewed: October 2009
- The study is ineligible for review because it does not use a sample within the age or grade range specified in the protocol.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Please see the WWC summary of evidence for Headsprout Early Reading.
Findings
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Study sample characteristics were not reported.Additional Sources
In the case of multiple manuscripts that report on one study, the WWC selects one manuscript as the primary citation and lists other manuscripts that describe the study as additional sources.
-
Dynarski, M., Agodini, R., Heaviside, S., Novak, T., Carey, N., Campuzano, L., ... Sussex, W. (2007). Effectiveness of reading and mathematics software products: Findings from the first student cohort: Report to Congress. Washington, DC: U.S. Department of Education, Institute of Education Sciences.
An indicator of the effect of the intervention, the improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention.
For more, please see the WWC Glossary entry for improvement index.
An outcome is the knowledge, skills, and attitudes that are attained as a result of an activity. An outcome measures is an instrument, device, or method that provides data on the outcome.
A finding that is included in the effectiveness rating. Excluded findings may include subgroups and subscales.
The sample on which the analysis was conducted.
The group to which the intervention group is compared, which may include a different intervention, business as usual, or no services.
The timing of the post-intervention outcome measure.
The number of students included in the analysis.
The mean score of students in the intervention group.
The mean score of students in the comparison group.
The WWC considers a finding to be statistically significant if the likelihood that the finding is due to chance alone, rather than a real difference, is less than five percent.
The WWC reviews studies for WWC products, Department of Education grant competitions, and IES performance measures.
The name and version of the document used to guide the review of the study.
The version of the WWC design standards used to guide the review of the study.
The result of the WWC assessment of the study. The rating is based on the strength of evidence of the effectiveness of the intervention. Studies are given a rating of Meets WWC Design Standards without Reservations, Meets WWC Design Standards with Reservations, or >Does Not Meet WWC Design Standards.
A related publication that was reviewed alongside the main study of interest.
Study findings for this report.
Based on the direction, magnitude, and statistical significance of the findings within a domain, the WWC characterizes the findings from a study as one of the following: statistically significant positive effects, substantively important positive effects, indeterminate effects, substantively important negative effects, and statistically significant negative effects. For more, please see the WWC Handbook.
The WWC may review studies for multiple purposes, including different reports and re-reviews using updated standards. Each WWC review of this study is listed in the dropdown. Details on any review may be accessed by making a selection from the drop down list.
Tier 1 Strong indicates strong evidence of effectiveness,
Tier 2 Moderate indicates moderate evidence of effectiveness, and
Tier 3 Promising indicates promising evidence of effectiveness,
as defined in the
non-regulatory guidance for ESSA
and the regulations for ED discretionary grants (EDGAR Part 77).