WWC review of this study

Comparative effectiveness of Carnegie Learning’s Cognitive Tutor Algebra I curriculum: A report of a randomized experiment in the Maui School District.

Cabalo, J. V., Jaciw, A., & Vu, M.-T. (2007). Palo Alto, CA: Empirical Education, Inc. Retrieved from: https://eric.ed.gov/?id=ED538958

  • Randomized Controlled Trial
     examining 
    569
     Students
    , grades
    6-PS
No statistically significant positive
findings
Meets WWC standards without reservations

Reviewed: September 2016

Mathematics achievement outcomes—Indeterminate effects found
Outcome
measure
Comparison Period Sample Intervention
mean
Comparison
mean
Significant? Improvement
index

Northwest Evaluation Association General Math Test

Cognitive Tutor® vs. Business as usual

1 Semester

Full sample;
569 students

222.37

223.04

No

-2
 
 

Characteristics of study sample as reported by study author.


  • Urban
    • B
    • A
    • C
    • D
    • E
    • F
    • G
    • I
    • H
    • J
    • K
    • L
    • P
    • M
    • N
    • O
    • Q
    • R
    • S
    • V
    • U
    • T
    • W
    • X
    • Z
    • Y
    • a
    • h
    • i
    • b
    • d
    • e
    • f
    • c
    • g
    • j
    • k
    • l
    • m
    • n
    • o
    • p
    • q
    • r
    • s
    • t
    • u
    • x
    • w
    • y

    Hawaii

Setting

This study assesses the impact of Cognitive Tutor on mathematics achievement in a sample of middle school students in Hawaii's Maui School District. Cognitive Tutor is a computer-based math program developed by Carnegie Learning that includes six unique aspects: simple and straightforward design, research-based pedagogy, immediate feedback, word problems represented in multiple ways, skill bars to show students what they have mastered, and a curriculum that is partly classroom-based and partly computer-lab based. Approximately 40% of the time is geared to students' individualized lessons. Twelve teachers participated in the study, and their sections of pre-algebra were randomly assigned to intervention or comparison groups (that is, each teacher taught both types of groups).

Study sample

Based on the fall 2006 roster, 809 students were available in the 32 classes that participated in the study. Seven hundred six students were in the study from start to finish, but only 476 students had both pre-test and post-test scores. Twelve teachers participated in the study, although one teacher dropped out, along with his/her four classes (two in each condition)

Intervention Group

Cognitive Tutor is a computer-based math program developed by Carnegie Learning that includes six unique aspects: simple and straightforward design, research-based pedagogy, immediate feedback, word problems represented in multiple ways, skill bars to show students what they have mastered, and a curriculum that is partly classroom-based and partly computer-lab based. Approximately 40% of the time is geared to students' individualized lessons. Teachers also present problem-solving that requires collaborative work by the students.

Comparison Group

Classes in the comparison condition received the existing math curriculum that was already in place. The study reports that multiple pre-Algebra textbooks were used across the classes.

Support for implementation

Teachers were offered the opportunity to attend a 3 day professional development course where there received CT materials. No further information is offered around support for implementation, although information on teacher perceptions of implementation are included (e.g., if they felt they had the resources to properly implement; percent time using the CT software vs the textbook).

No statistically significant positive
findings
Meets WWC standards without reservations

Reviewed: July 2016

Algebra outcomes—Indeterminate effects found
Outcome
measure
Comparison Period Sample Intervention
mean
Comparison
mean
Significant? Improvement
index

Northwest Evaluation Association (NWEA) Algebra End-of-Course Achievement Level Test

Cognitive Tutor® Algebra I vs. Business as usual

8 Months

Grades 8–13;
344 students

243.37

244.71

No

--

Characteristics of study sample as reported by study author.


  • 6% English language learners

  • 27% Free or reduced price lunch
  • Race
    Asian
    32%
    Not specified
    14%
    Pacific Islander
    33%
    White
    11%
  • Ethnicity
    Hispanic
    3%

  • Rural, Suburban
    • B
    • A
    • C
    • D
    • E
    • F
    • G
    • I
    • H
    • J
    • K
    • L
    • P
    • M
    • N
    • O
    • Q
    • R
    • S
    • V
    • U
    • T
    • W
    • X
    • Z
    • Y
    • a
    • h
    • i
    • b
    • d
    • e
    • f
    • c
    • g
    • j
    • k
    • l
    • m
    • n
    • o
    • p
    • q
    • r
    • s
    • t
    • u
    • x
    • w
    • y

    Hawaii

Setting

The study took place at Maui Community College and in five schools (grades 8–12) within the Maui School District, both located in Maui County, Hawaii. A total of nine teachers and 22 Algebra I classrooms participated in the study. At the beginning of the study, students in grades 9–12 comprised 73% of the sample, with 19% in grade 8 and 7% enrolled at Maui Community College.

Study sample

Among the participating Maui School District schools, overall student demographics in the prior school year were as follows: 32% Filipino, 28% Part-Hawaiian, 11% White, 8% Japanese, 5% Hawaiian, 3% Hispanic, and 14% Other. Approximately 27% of students participated in the National School Lunch Program, and approximately 6% were designated as Limited English Proficient. The authors report a similar distribution of ethnicities at the Maui Community College.

Intervention Group

Each teacher taught an intervention class and a comparison class. For the classes selected for the intervention classes, teachers implemented the Cognitive Tutor Algebra I curriculum for final six months during the 2005-2006 school year. The authors report that the curriculum is designed for 40% computer lab time and 60% classroom activities, combining software-based individualized lessons with collaborative problem-solving activities. Because the study began in October/November (lasting through the end of the school year) students in the intervention group received some exposure to the instruction provided in the comparison classroom.

Comparison Group

In the comparison classrooms, teachers continued to use the textbook program in use at the time of study implementation. The authors describe a variety of branded Algebra I textbooks in use in the comparison classrooms.

Support for implementation

Teachers implementing Cognitive Tutor Algebra I received three days of professional development led by a consultant from the curriculum developer. Teachers received a brief classroom observation and opportunity to ask questions of a developer representative early in the implementation period, but were given flexibility in actual classroom implementation. No ongoing technical assistance was provided.

 

Your export should download shortly as a zip archive.

This download will include data files for study and findings review data and a data dictionary.

Connect With the WWC

newsflash icon contact icon facebook icon twitter icon
loading
back to top