Purpose: The primary purpose of this study was to evaluate the impact of the Cognitive Tutor® Algebra I curriculum on mathematics achievement when the curriculum was implemented at scale. Although many educators recognize the need for improved mathematics instruction, school staff currently have few evidence-based options from which to choose. Therefore, this study addresses the need for scale-up evaluations of mathematics curriculums.

Project Activities: The research team conducted a cluster randomized trial in which schools were randomly assigned either to continue using their current algebra curriculum (control condition) or to use the Cognitive Tutor Algebra I (CTAI) curriculum. The Cognitive Tutor was designed to promote students' understanding of algebraic concepts and principles, problem solving skills, and mastery of higher-order mathematical concepts. A central component of the Cognitive Tutor was an automated computer-based tutor that provided individualized instruction to address students' specific needs. The individualization was built into the software and was facilitated by detailed computational models of student thinking in algebra. In addition to studying the impact of Cognitive Tutor, the researchers examined factors that may affect the implementation of Cognitive Tutor and other technology-based mathematics curricula.

Key Outcomes: The main outcomes of this study are reported in Pane et at., 2014; Pane et al., 2013 and include the following:

Each study school participated in the study for 2 years, with different cohorts of students taking algebra.

• In the first year, posttest scores did not differ significantly between schools using CTAI and control group schools.
• In the second year, high schools using CTAI showed significantly higher posttest scores than high schools in the control group.
• The standardized effect of approximately 0.2 is equivalent to moving a student from the 50th percentile to the 58th percentile on the posttest.
• The study estimated a similar effect for middle schools the second year, although it was not statistically significant, possibly due to smaller sample sizes.

Structured Abstract

Purpose: The primary purpose of this study was to evaluate the impact of the Cognitive Tutor® Algebra I curriculum on mathematics achievement when the curriculum was implemented at scale—that is, when it was implemented across diverse school populations and conditions and with no more support than schools would have access to if they had selected Cognitive Tutor as their algebra curriculum apart from participation in a research project.

Setting: Fifty-four high schools and 68 middle schools from school districts in Texas, Connecticut, New Jersey, Michigan, and Kentucky participated.

Sample: Participating school districts were demographically diverse, with African-American populations ranging up to 67 percent, Hispanic populations ranging up to 58 percent, economically disadvantaged populations ranging up to 96 percent, and English learner populations ranging up to 29 percent. For the middle school study, all sixth to eighth graders in the designated classes in a given school participated. For the high school study, all 9th to 12th graders in the designated classes in a given school participated.

Intervention: The Cognitive Tutor was designed to promote students' understanding of algebraic concepts and principles, problem solving skills, and mastery of higher order mathematical concepts. A central component of the Cognitive Tutor was an automated computer-based tutor that provided individualized instruction to address students' specific needs. The individualization was built into the software and was facilitated by detailed computational models of student thinking in algebra. Teachers received four days of training on the new curriculum as part of a standard curriculum package.

Research Design and Methods: The research team conducted a cluster randomized trial in which schools were randomly assigned either to continue using their current algebra curriculum (control condition) or to use the Cognitive Tutor Algebra I curriculum.

Control Condition: The control classrooms used the Algebra I curricula currently in place in their schools.

Key Measures: The researchers used exams from the CTB/McGraw-Hill TerraNova Algebra Assessment System for the primary student achievement outcome measure, as well as for the pre-test. A student survey provided additional information about student confidence and attitudes about mathematics and technology, plans for subsequent course taking and college admission testing, and schooling and career plans.

Data Analytic Strategy: Analyses included hierarchical linear models and generalized linear mixed models. These models allowed for school- and teacher-level clustering and captured potential school-by-treatment interactions. These analyses were complemented with qualitative analyses to gain a description of curricular implementation processes and factors influencing implementation.

Products and Publications

ERIC Citations: Find available citations in ERIC for this award here.

WWC Review: Pane, J. F., Griffin, B. A., McCaffrey, D. F., & Karam, R. (2014). Effectiveness of Cognitive Tutor Algebra I at Scale. Educational Evaluation and Policy Analysis, 36(2), 127–144. [WWC Review]

Select Publications:

Journal articles

Pane, J.F., Griffin, B.A., McCaffrey, D.F., and Karam, R. (2014). Effectiveness of Cognitive Tutor Algebra I at Scale. Educational Evaluation and Policy Analysis, 36(2): 127–144.

Kaufman, J.H., Karam, R., Pane, J., and Junker, B. (2012). How Curriculum and Classroom Achievement Predict Teacher Time on Lecture- and Inquiry-Based Mathematics Activities. Journal of Mathematics Education at Teachers College, 3(2): 56–62.

Peer-reviewed nongovernment reports

Daugherty, L., Phillips, A., Pane, J.F., and Karam, R. (2012). Analysis of Costs in an Algebra I Curriculum Effectiveness Study.Santa Monica, CA: RAND Corporation.

Pane, J.F., Griffin, B.A., McCaffrey, D.F., Karam, R., Daugherty, L., and Phillips, A. (2013). Does an Algebra Course With Tutoring Software Improve Student Learning?.Santa Monica, CA: RAND Corporation.

Pane, J.F., Griffin, B.A., McCaffrey, D.F., and Karam, R. (2014). Addendum to Effectiveness of Cognitive Tutor Algebra I at Scale(WR-1050–DEIES). Santa Monica, CA: RAND Corporation Working Paper.