WWC review of this study

Effectiveness of reading and mathematics software products: Findings from two student cohorts.

Campuzano, L., Dynarski, M., Agodini, R., & Rall, K. (2009). Washington, DC: U.S. Department of Education, Institute of Education Sciences. Retrieved from: https://eric.ed.gov/?id=ED504657

  • Randomized Controlled Trial
     examining 
    2,588
     Students
    , grades
    1-12
No statistically significant positive
findings
Meets WWC standards with reservations

Reviewed: June 2016

Algebra outcomes—Indeterminate effects found
Outcome
measure
Comparison Period Sample Intervention
mean
Comparison
mean
Significant? Improvement
index

Educational Testing Service (ETS) End-of-Course Algebra Test

Cognitive Tutor® Algebra I vs. Business as usual

9 Months

Grades 8 and 9;
270 students

32.39

35.31

No

--

Characteristics of study sample as reported by study author.


  • Female: 50%
    Male: 50%

  • Urban

Setting

This 2-year congressionally-mandated study of education technology included several algebra interventions, including Cognitive Tutor® Algebra I. Findings from the first year of the study pooled all of the algebra interventions together. Therefore, this WWC intervention report focuses on findings from the second year, in which results are reported separately by intervention. In the second year of the study, Cognitive Tutor® Algebra I was implemented in nine schools in four districts. Districts were located in urban and urban fringe areas. Within each of the nine study schools, teachers were randomly assigned to the intervention and comparison groups prior to the first year of the study. Teachers maintained their assignment in the second year of the study.

Study sample

Students in the year one and two combined sample were 51 percent female for the intervention group and 46 percent female for the comparison group. The average age of students in the imputed analytic sample was 14.93 for both study groups. No other information about sample characteristics for either the full sample of the second-year sample is provided in the report.

Intervention Group

Students in the intervention group were taught using Cognitive Tutor® Algebra I as their core math curriculum. The curriculum covered proportional reasoning, solving linear equations and inequalities, solving systems of linear equations, analyzing data, and using polynomial functions, powers, and exponents. Teachers were in their second year of implementing Cognitive Tutor® Algebra I, while students were experiencing the curriculum for the first time. According to the study authors, students in the study used the computer portion of the curriculum for an average of 1,840 minutes during 18 weeks.

Comparison Group

Students in the comparison classes received traditional algebra instruction using their district’s standard algebra curriculum. The comparison curricula were not specified and may have varied across districts and schools.

Support for implementation

Within each district, teachers in the intervention group received 4 days of initial training from the publisher in the summer of 2004. They were trained on classroom management and the curriculum, and provided with opportunities to practice using the product. An unspecified amount of phone and email support was provided throughout the study.

In the case of multiple manuscripts that report on one study, the WWC selects one manuscript as the primary citation and lists other manuscripts that describe the study as additional sources.

  • Dynarski, M., Agodini, R., Heaviside, S., Novak, T., Carey, N., Campuzano, L., ... Sussex, W. (2007). Effectiveness of reading and mathematics software products: Findings from the first student cohort: Report to Congress. Washington, DC: U.S. Department of Education, Institute of Education Sciences.

No statistically significant positive
findings
Meets WWC standards without reservations

Reviewed: March 2010

Algebra outcomes—Indeterminate effects found
Outcome
measure
Comparison Period Sample Intervention
mean
Comparison
mean
Significant? Improvement
index

Larson Algebra 1

Larson Algebra 1 vs. Business as usual

Spring test

Grade 9;
755 students

30.55

32.76

No

--

Characteristics of study sample as reported by study author.


  • 61% Free or reduced price lunch
  • Race
    Asian
    4%
    Black
    39%
    White
    30%
  • Ethnicity
    Hispanic
    28%
    Not Hispanic
    62%

  • Rural, Suburban, Urban
At least one statistically significant positive finding
Meets WWC standards without reservations

Reviewed: March 2010

Mathematics achievement outcomes—Statistically significant positive effects found
Outcome
measure
Comparison Period Sample Intervention
mean
Comparison
mean
Significant? Improvement
index

Larson Pre-Algebra

Larson Pre-Algebra vs. Business as usual

Spring test NCE

Grade 6;
2,588 students

53.42

52.59

Yes

 
 
4

Characteristics of study sample as reported by study author.


  • 61% Free or reduced price lunch
  • Race
    Asian
    4%
    Black
    39%
    White
    30%
  • Ethnicity
    Hispanic
    28%
    Not Hispanic
    62%

  • Rural, Suburban, Urban
No statistically significant positive
findings
Meets WWC standards without reservations

Reviewed: March 2010

Algebra outcomes—Indeterminate effects found
Outcome
measure
Comparison Period Sample Intervention
mean
Comparison
mean
Significant? Improvement
index

Cognitive Tutor

Cognitive Tutor® Algebra I vs. Business as usual

Spring test

Grade 9;
1,204 students

38.64

38.37

No

--

Characteristics of study sample as reported by study author.


  • 61% Free or reduced price lunch
  • Race
    Asian
    4%
    Black
    39%
    White
    30%
  • Ethnicity
    Hispanic
    28%
    Not Hispanic
    62%

  • Rural, Suburban, Urban
No statistically significant positive
findings
Meets WWC standards without reservations

Reviewed: March 2010

Mathematics achievement outcomes—Indeterminate effects found
Outcome
measure
Comparison Period Sample Intervention
mean
Comparison
mean
Significant? Improvement
index

Plato Achieve Now

PLATO Achieve Now vs. Business as usual

Spring test NCE

Grade 6;
1,037 students

46.06

50.67

No

--

Characteristics of study sample as reported by study author.


  • 61% Free or reduced price lunch
  • Race
    Asian
    4%
    Black
    39%
    White
    30%
  • Ethnicity
    Hispanic
    28%
    Not Hispanic
    62%

  • Rural, Suburban, Urban
No statistically significant positive
findings
Meets WWC standards without reservations

Reviewed: March 2010

Reading achievement outcomes—Indeterminate effects found
Outcome
measure
Comparison Period Sample Intervention
mean
Comparison
mean
Significant? Improvement
index

LeapTrack

LeapTrack vs. Business as usual

Spring test NCE

Grade 4;
1,274 students

45.31

45.95

No

--

Characteristics of study sample as reported by study author.


  • 61% Free or reduced price lunch
  • Race
    Asian
    4%
    Black
    39%
    White
    30%
  • Ethnicity
    Hispanic
    28%
    Not Hispanic
    62%

  • Rural, Suburban, Urban
No statistically significant positive
findings
Meets WWC standards without reservations

Reviewed: March 2010

Reading achievement outcomes—Indeterminate effects found
Outcome
measure
Comparison Period Sample Intervention
mean
Comparison
mean
Significant? Improvement
index

Academy of Reading

Academy of READING® vs. Business as usual

Spring test NCE

Grade 4;
899 students

38.63

41.45

No

--

Characteristics of study sample as reported by study author.


  • 61% Free or reduced price lunch
  • Race
    Asian
    4%
    Black
    39%
    White
    30%
  • Ethnicity
    Hispanic
    28%
    Not Hispanic
    62%

  • Rural, Suburban, Urban
No statistically significant positive
findings
Meets WWC standards without reservations

Reviewed: March 2010

Reading achievement outcomes—Indeterminate effects found
Outcome
measure
Comparison Period Sample Intervention
mean
Comparison
mean
Significant? Improvement
index

Waterford Early Reading Program

Waterford Early Reading Program vs. Business as usual

Spring test NCE

Grade 1;
1,155 students

49.83

49.21

No

--

Characteristics of study sample as reported by study author.


  • 61% Free or reduced price lunch
  • Race
    Asian
    4%
    Black
    39%
    White
    30%
  • Ethnicity
    Hispanic
    28%
    Not Hispanic
    62%

  • Rural, Suburban, Urban
No statistically significant positive
findings
Meets WWC standards without reservations

Reviewed: March 2010

Reading achievement outcomes—Indeterminate effects found
Outcome
measure
Comparison Period Sample Intervention
mean
Comparison
mean
Significant? Improvement
index

Plato Focus

PLATO Focus vs. Business as usual

Spring test NCE

Grade 1;
618 students

51.15

50.4

No

--

Characteristics of study sample as reported by study author.


  • 61% Free or reduced price lunch
  • Race
    Asian
    4%
    Black
    39%
    White
    30%
  • Ethnicity
    Hispanic
    28%
    Not Hispanic
    62%

  • Rural, Suburban, Urban
No statistically significant positive
findings
Meets WWC standards without reservations

Reviewed: March 2010

Reading achievement outcomes—Indeterminate effects found
Outcome
measure
Comparison Period Sample Intervention
mean
Comparison
mean
Significant? Improvement
index

Headsprout Early Reading

Headsprout Early Reading vs. Business as usual

Spring test NCE

Grade 1;
1,079 students

55.24

57.1

No

--

Characteristics of study sample as reported by study author.


  • 61% Free or reduced price lunch
  • Race
    Asian
    4%
    Black
    39%
    White
    30%
  • Ethnicity
    Hispanic
    28%
    Not Hispanic
    62%

  • Rural, Suburban, Urban
No statistically significant positive
findings
Meets WWC standards without reservations

Reviewed: March 2010

Reading achievement outcomes—Indeterminate effects found
Outcome
measure
Comparison Period Sample Intervention
mean
Comparison
mean
Significant? Improvement
index

Destination Reading

Destination Reading vs. Business as usual

Spring test NCE

Grade 1;
742 students

50.82

49.13

No

--

Characteristics of study sample as reported by study author.


  • 61% Free or reduced price lunch
  • Race
    Asian
    4%
    Black
    39%
    White
    30%
  • Ethnicity
    Hispanic
    28%
    Not Hispanic
    62%

  • Rural, Suburban, Urban
No statistically significant positive
findings
Meets WWC standards without reservations

Reviewed: March 2010

Mathematics achievement outcomes—Indeterminate effects found
Outcome
measure
Comparison Period Sample Intervention
mean
Comparison
mean
Significant? Improvement
index

Math NCE

PLATO vs. Business as usual

2004-2006

Grade 6;
1,037 students

50.09

50.67

No

--

Characteristics of study sample as reported by study author.


  • 74% Free or reduced price lunch

  • Female: 53%
    Male: 47%
  • Race
    Black
    40%
  • Ethnicity
    Hispanic
    42%
    Not Hispanic
    58%

  • Rural, Suburban, Urban

Setting

The study took place in 13 schools in three districts in multiple states across the country.

Study sample

The study sample included 1,037 sixth-grade students (547 PLATO® Achieve Now; 490 control) taught by 39 teachers (21 PLATO® Achieve Now; 18 control) in 13 schools across three districts in multiple states across the country during the 2004–05 and 2005–06 school years. Of the study sample, approximately 53% were female (52% PLATO® Achieve Now and 55% control), 74% received free or reduced-price lunch (not reported by intervention status), 42% were Hispanic (not reported by intervention status), and 40% were African-American (not reported by intervention status). Approximately 80% of the teachers in the study were female (81% PLATO® Achieve Now; 78% control) with an average of 11 years of teaching experience (9 years PLATO® Achieve Now; 13 years control) and 33% of whom obtained a master’s degree (24% PLATO® Achieve Now; 44% control).

Intervention Group

Students were taught using PLATO® Achieve Now during the 2004–05 and/or 2005–06 school years. PLATO® Achieve Now supplemented standard mathematics instruction for the treatment group. According to the study authors, PLATO® Achieve Now students used the product for independent practice and reinforcement of math skills. Students worked at their own pace on activities identified by the teacher. According to the authors, the recommended usage is 30 minutes per day, four days per week, for at least 10 weeks.

Comparison Group

Comparison students were taught in traditional classes, with the teachers utilizing any technology products already available to them.

Outcome descriptions

The primary outcome measures in Year 2 of the study were the Stanford Achievement Test–Tenth Edition (SAT–10), the Iowa Test of Basic Skills (ITBS), and the New Mexico Standards Based Assessment (NMSBA). Districts in this study used at least one of these three exams as the outcome measure to obtain pretest and/or posttest scores. The study authors converted the scale scores from these tests to normal curve equivalent (NCE) scores with a range of 1 to 99 and an average of 50 to standardize the measures across tests and cohorts. For a more detailed description of these outcome measures, see Appendix A2.

Support for implementation

Vendor training sessions generally took place in host districts, and sometimes host schools, during the summer or early fall of 2004. The initial training lasted about 6 hours and varied by product from 4 hours to about 8 hours. Vendors delivered ongoing support in several modes. Product representatives visited teachers; vendors also provided support through email, telephone help desks, and additional training at schools.

In the case of multiple manuscripts that report on one study, the WWC selects one manuscript as the primary citation and lists other manuscripts that describe the study as additional sources.

  • Dynarski, M., Agodini, R., Heaviside, S., Novak, T., Carey, N., Campuzano, L., ... Sussex, W. (2007). Effectiveness of reading and mathematics software products: Findings from the first student cohort: Report to Congress. Washington, DC: U.S. Department of Education, Institute of Education Sciences.

Reviewed: October 2009

Study sample characteristics were not reported.

In the case of multiple manuscripts that report on one study, the WWC selects one manuscript as the primary citation and lists other manuscripts that describe the study as additional sources.

  • Dynarski, M., Agodini, R., Heaviside, S., Novak, T., Carey, N., Campuzano, L., ... Sussex, W. (2007). Effectiveness of reading and mathematics software products: Findings from the first student cohort: Report to Congress. Washington, DC: U.S. Department of Education, Institute of Education Sciences.

 

Your export should download shortly as a zip archive.

This download will include data files for study and findings review data and a data dictionary.

Connect With the WWC

loading
back to top