Comparative Effectiveness of Carnegie Learning's "Cognitive Tutor Bridge to Algebra" Curriculum: A Report of a Randomized Experiment in the Maui School District. Research Report
Cabalo, Jessica Villaruz; Ma, Boya; Jaciw, Andrew (2007). Empirical Education Inc. Retrieved from: https://eric.ed.gov/?id=ED538958
-
examining569Students, grades6-PS
Grant Competition
Review Details
Reviewed: September 2016
- Grant Competition (findings for Cognitive Tutor®)
- Randomized Controlled Trial
- Meets WWC standards without reservations
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Northwest Evaluation Association General Math Test |
Cognitive Tutor® vs. Business as usual |
1 Semester |
Full sample;
|
222.37 |
223.04 |
No |
-- |
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
Hawaii
Study Details
Setting
This study assesses the impact of Cognitive Tutor on mathematics achievement in a sample of middle school students in Hawaii's Maui School District. Cognitive Tutor is a computer-based math program developed by Carnegie Learning that includes six unique aspects: simple and straightforward design, research-based pedagogy, immediate feedback, word problems represented in multiple ways, skill bars to show students what they have mastered, and a curriculum that is partly classroom-based and partly computer-lab based. Approximately 40% of the time is geared to students' individualized lessons. Twelve teachers participated in the study, and their sections of pre-algebra were randomly assigned to intervention or comparison groups (that is, each teacher taught both types of groups).
Study sample
Based on the fall 2006 roster, 809 students were available in the 32 classes that participated in the study. Seven hundred six students were in the study from start to finish, but only 476 students had both pre-test and post-test scores. Twelve teachers participated in the study, although one teacher dropped out, along with his/her four classes (two in each condition)
Intervention Group
Cognitive Tutor is a computer-based math program developed by Carnegie Learning that includes six unique aspects: simple and straightforward design, research-based pedagogy, immediate feedback, word problems represented in multiple ways, skill bars to show students what they have mastered, and a curriculum that is partly classroom-based and partly computer-lab based. Approximately 40% of the time is geared to students' individualized lessons. Teachers also present problem-solving that requires collaborative work by the students.
Comparison Group
Classes in the comparison condition received the existing math curriculum that was already in place. The study reports that multiple pre-Algebra textbooks were used across the classes.
Support for implementation
Teachers were offered the opportunity to attend a 3 day professional development course where there received CT materials. No further information is offered around support for implementation, although information on teacher perceptions of implementation are included (e.g., if they felt they had the resources to properly implement; percent time using the CT software vs the textbook).
Cognitive Tutor Algebra I Intervention Report - Secondary Mathematics
Review Details
Reviewed: July 2016
- Randomized Controlled Trial
- Meets WWC standards without reservations
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Please see the WWC summary of evidence for Cognitive Tutor Algebra I.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Northwest Evaluation Association (NWEA) Algebra End-of-Course Achievement Level Test |
Cognitive Tutor Algebra I vs. Business as usual |
8 Months |
Grades 8–13;
|
243.37 |
244.71 |
No |
-- |
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
6% English language learners -
Rural, Suburban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
Hawaii
-
Race Asian 32% Other or unknown 14% Pacific Islander 33% White 11% -
Ethnicity Hispanic 3%
Study Details
Setting
The study took place at Maui Community College and in five schools (grades 8–12) within the Maui School District, both located in Maui County, Hawaii. A total of nine teachers and 22 Algebra I classrooms participated in the study. At the beginning of the study, students in grades 9–12 comprised 73% of the sample, with 19% in grade 8 and 7% enrolled at Maui Community College.
Study sample
Among the participating Maui School District schools, overall student demographics in the prior school year were as follows: 32% Filipino, 28% Part-Hawaiian, 11% White, 8% Japanese, 5% Hawaiian, 3% Hispanic, and 14% Other. Approximately 27% of students participated in the National School Lunch Program, and approximately 6% were designated as Limited English Proficient. The authors report a similar distribution of ethnicities at the Maui Community College.
Intervention Group
Each teacher taught an intervention class and a comparison class. For the classes selected for the intervention classes, teachers implemented the Cognitive Tutor Algebra I curriculum for final six months during the 2005-2006 school year. The authors report that the curriculum is designed for 40% computer lab time and 60% classroom activities, combining software-based individualized lessons with collaborative problem-solving activities. Because the study began in October/November (lasting through the end of the school year) students in the intervention group received some exposure to the instruction provided in the comparison classroom.
Comparison Group
In the comparison classrooms, teachers continued to use the textbook program in use at the time of study implementation. The authors describe a variety of branded Algebra I textbooks in use in the comparison classrooms.
Support for implementation
Teachers implementing Cognitive Tutor Algebra I received three days of professional development led by a consultant from the curriculum developer. Teachers received a brief classroom observation and opportunity to ask questions of a developer representative early in the implementation period, but were given flexibility in actual classroom implementation. No ongoing technical assistance was provided.
An indicator of the effect of the intervention, the improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention.
For more, please see the WWC Glossary entry for improvement index.
An outcome is the knowledge, skills, and attitudes that are attained as a result of an activity. An outcome measures is an instrument, device, or method that provides data on the outcome.
A finding that is included in the effectiveness rating. Excluded findings may include subgroups and subscales.
The sample on which the analysis was conducted.
The group to which the intervention group is compared, which may include a different intervention, business as usual, or no services.
The timing of the post-intervention outcome measure.
The number of students included in the analysis.
The mean score of students in the intervention group.
The mean score of students in the comparison group.
The WWC considers a finding to be statistically significant if the likelihood that the finding is due to chance alone, rather than a real difference, is less than five percent.
The WWC reviews studies for WWC products, Department of Education grant competitions, and IES performance measures.
The name and version of the document used to guide the review of the study.
The version of the WWC design standards used to guide the review of the study.
The result of the WWC assessment of the study. The rating is based on the strength of evidence of the effectiveness of the intervention. Studies are given a rating of Meets WWC Design Standards without Reservations, Meets WWC Design Standards with Reservations, or >Does Not Meet WWC Design Standards.
A related publication that was reviewed alongside the main study of interest.
Study findings for this report.
Based on the direction, magnitude, and statistical significance of the findings within a domain, the WWC characterizes the findings from a study as one of the following: statistically significant positive effects, substantively important positive effects, indeterminate effects, substantively important negative effects, and statistically significant negative effects. For more, please see the WWC Handbook.
The WWC may review studies for multiple purposes, including different reports and re-reviews using updated standards. Each WWC review of this study is listed in the dropdown. Details on any review may be accessed by making a selection from the drop down list.
Tier 1 Strong indicates strong evidence of effectiveness,
Tier 2 Moderate indicates moderate evidence of effectiveness, and
Tier 3 Promising indicates promising evidence of effectiveness,
as defined in the
non-regulatory guidance for ESSA
and the regulations for ED discretionary grants (EDGAR Part 77).