WWC review of this study

After two years, three elementary math curricula outperform a fourth (NCEE 2013-4019).

Agodini, R., Harris, B., Seftor, N., Remillard, J., & Thomas, M. (2013). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Retrieved from: https://eric.ed.gov/?id=ED544185

  • Randomized Controlled Trial
     examining 
    2,045
     Students
    , grades
    1-2
At least one statistically significant positive finding
Meets WWC standards with reservations

Reviewed: May 2017

Mathematics achievement outcomes—Indeterminate effects found
Outcome
measure
Comparison Period Sample Intervention
mean
Comparison
mean
Significant? Improvement
index

Early Childhood Longitudinal Study- Kindergarten (ECLS-K): 2nd grade adaptation

Saxon Math vs. Investigations in Number, Data, and Space®

2 Years

Grade: 2;
882 students

71.72

67.31

Yes

 
 
10
More Outcomes

Early Childhood Longitudinal Study- Kindergarten (ECLS-K): 2nd grade adaptation

Saxon Math vs. SFAW/enVisionMATH

2 Years

Grade: 2;
1,136 students

68.9

68.87

No

--

Early Childhood Longitudinal Study- Kindergarten (ECLS-K): 2nd grade adaptation

Saxon Math vs. Math Expressions

2 Years

Grade: 2;
931 students

67.56

67.99

No

--
Show Supplemental Findings

Early Childhood Longitudinal Study- Kindergarten (ECLS-K): 2nd grade adaptation

Saxon Math vs. Investigations in Number, Data, and Space®

1 Year

Grade: 1 first cohort;
636 students

47.56

44.87

Yes

 
 
13

Early Childhood Longitudinal Study- Kindergarten (ECLS-K): 2nd grade adaptation

Saxon Math vs. Scott Foresman-Addison Wesley Elementary Mathematics

1 Year

Grade: 1, first cohort;
663 students

46.17

44.28

Yes

 
 
9

Early Childhood Longitudinal Study- Kindergarten (ECLS-K): 2nd grade adaptation

Saxon Math vs. Scott Foresman-Addison Wesley Elementary Mathematics

1 Year

Grade: 2, both cohorts;
1,706 students

72.98

70.31

Yes

 
 
7

Early Childhood Longitudinal Study- Kindergarten (ECLS-K): 2nd grade adaptation

Saxon Math vs. Investigations in Number, Data, and Space®

1 Year

Grade: 2, both cohorts;
1,711 students

71.88

69.85

Yes

 
 
5

Early Childhood Longitudinal Study- Kindergarten (ECLS-K): 2nd grade adaptation

Saxon Math vs. Math Expressions

1 Year

Grade: 2, both cohorts;
1,721 students

73.09

71.38

No

--

Early Childhood Longitudinal Study- Kindergarten (ECLS-K): 2nd grade adaptation

Saxon Math vs. Scott Foresman-Addison Wesley Elementary Mathematics

1 Year

Grade: 1 both cohorts;
2,377 students

45.08

44.43

No

--

Early Childhood Longitudinal Study- Kindergarten (ECLS-K): 2nd grade adaptation

Saxon Math vs. Investigations in Number, Data, and Space®

1 Year

Grade: 1, both cohorts;
2,235 students

45.13

44.51

No

--

Early Childhood Longitudinal Study- Kindergarten (ECLS-K): 2nd grade adaptation

Saxon Math vs. Math Expressions

1 Year

Grade: 1, first cohort;
618 students

45.4

45.45

No

--

Early Childhood Longitudinal Study- Kindergarten (ECLS-K): 2nd grade adaptation

Saxon Math vs. Math Expressions

1 Year

Grade: 1, both cohorts;
2,320 students

44.52

44.74

No

--

Characteristics of study sample as reported by study author.


  • 17% English language learners

  • Female: 49%
    Male: 51%
  • Race
    Black
    40%
    Not specified
    60%
  • Ethnicity
    Hispanic
    28%
    Not Hispanic
    72%

  • Rural, Suburban, Urban
    • B
    • A
    • C
    • D
    • E
    • F
    • G
    • I
    • H
    • J
    • K
    • L
    • P
    • M
    • N
    • O
    • Q
    • R
    • S
    • V
    • U
    • T
    • W
    • X
    • Z
    • Y
    • a
    • h
    • i
    • b
    • d
    • e
    • f
    • c
    • g
    • j
    • k
    • l
    • m
    • n
    • o
    • p
    • q
    • r
    • s
    • t
    • u
    • x
    • w
    • y

    Connecticut, Florida, Kentucky, Minnesota, Missouri, Mississippi, Nevada, New York, South Carolina, Texas

Setting

The study was conducted in 111 schools in 12 districts in 10 states (Connecticut, Florida, Kentucky, Minnesota, Mississippi, Missouri, Nevada, New York, South Carolina, and Texas). Of the 12 districts, three were in urban areas, five were in suburban areas, and four were in rural areas. The study data were collected during the 2006–07, 2007–08, and 2008–09 school years. Of the 111 schools, 58 participated in the study for 2 years. These 58 schools were located in seven districts in up to seven states (the authors did not specify which states).

Study sample

The study authors randomly assigned 111 schools within 12 school districts to one of four math curricula (Saxon Math, Investigations, Math Expressions, or SFAW). Random assignment was conducted within district and stratified on characteristics such as school size, free or reduced-price meal eligibility, math proficiency, and race/ethnicity. Random assignment was conducted before the school year began in the first year of the study, and the student sample was defined immediately prior to the pretest. Thus, the study may have included students in the analytic sample that enrolled in schools after random assignment. Within study schools, all students at the target grade levels (first and/or second grades) used their school’s assigned curriculum. Approximately 30 students per grade level were randomly sampled for testing by the study team. The primary findings that contribute to the rating of effectiveness are based on students from 58 of the 111 schools (within seven of the 12 districts) who participated in the study and used their assigned curriculum for 2 consecutive years. This analytic sample was comprised of 2,045 students in 222 classrooms who experienced their assigned curriculum in first and second grades. Students were pretested at the beginning of first grade and posttested at the end of second grade. Of the 58 study schools, 12 used Saxon Math, 14 used Investigations, 14 used Math Expressions, and 18 used SFAW/enVisionMATH. In the analytic sample of students, 49% were female, 40% were non-Hispanic Black, 32% were other non-Hispanic, and 28% were Hispanic. Students with limited English proficiency or classified as English language learners were 17% of the sample, and 9% of students had individualized education plans (IEPs) or were receiving special education services. This review also includes supplemental findings from two study reports. The first set of supplemental findings is based on the 2009 report that presented findings for 1,309 first-grade students in 39 schools that participated in the study during the 2006–07 school year. The second set of findings is from the 2010 report that presented finding for 8,060 first- or second-grade students in 110 schools that used the study curricula for 1 year and participated in the study in either the 2006–07 or 2007–08 school years. In the 2009 and 2010 reports, the findings are 1-year effects where students were pre- and posttested in first or second grade, and findings are reported separately by grade. The supplemental findings are presented in Appendix D and do not factor into the intervention’s rating of effectiveness.

Intervention Group

Students in the intervention group used Saxon Math as their core curriculum in the first and second grades in the 2006–07, 2007–08, or 2008–09 school years. The study did not specify which edition of Saxon Math was used but indicated that the 2005 and 2008 copyright years were used. All teachers in the intervention group reported using Saxon Math as their core math curriculum in first and second grades and provided, on average, 6 hours of math instruction per week in each grade. Additionally, most teachers (87% in first grade and 71% in second grade) reported completing at least 80% of the lessons from Saxon Math.

Comparison Group

Students in the comparison group used Investigations, Math Expressions, or SFAW/enVisionMATH as their core math curriculum. Investigations in Number, Data, and Space is a K–5, student-centered curriculum that emphasizes reasoning and communicating about math concepts, using multiple approaches to problem solving, through in-depth investigations of problems. Study schools used the first or second editions of the curriculum. All teachers in the first grade and 96% in the second grade reported using Investigations as their core math curriculum and provided, on average, 5 hours of math instruction per week in each grade. Additionally, most teachers (86% in first grade and 56% in second grade) reported completing at least 80% of the lessons from Investigations. Math Expressions is a K–5 curriculum that uses a combination of teacher-directed and student-centered approaches within consistent daily routines. The study schools used the 2005 or 2008 copyright years of the curriculum. Approximately 98% of teachers in the first grade and all teachers in the second grade reported using Math Expressions as their core math curriculum and provided, on average, 5 hours of math instruction per week in each grade. Additionally, most teachers (82% in first grade and 79% in second grade) reported completing at least 80% of the lessons from Math Expressions. SFAW is a K–6, teacher-directed curriculum that uses a consistent daily structure, involving a brief review of previous material, exploration of a new concept, explicit instruction on the new concept, and a closure activity to check student understanding. During the course of the study, the publisher revised the SFAW curriculum and renamed it enVisionMATH. This change affected four of the seven study districts. Results for this comparison group should be interpreted as a mix of students just receiving SFAW curriculum in grades 1–2, and some that received SFAW in grade 1 and enVisionMATH in grade 2. Outcomes that include grade 2 in the 2008–09 school year are referred to as SFAW/enVisionMATH. All teachers in the first grade and 99% in the second grade reported using SFAW or enVisionMATH as their core math curriculum and provided, on average, 5 hours of math instruction per week in each grade. Additionally, most teachers (87% in first grade and 71% in second grade) reported completing at least 80% of the lessons from SFAW or enVisionMATH.

Support for implementation

Training was provided by Houghton Mifflin Harcourt trainers to Saxon Math teachers. Saxon teachers were provided 1 day of initial training in the summer before the school year began and one follow-up training session in the fall, tailored to meet each district’s needs. In the follow-up training, some teachers watched demonstration lessons or participated in a math workshop. In other cases, trainers observed teachers conduct lessons and provided feedback afterwards. Approximately 90% and 81% of first- and second-grade Saxon teachers, respectively, attended an initial or refresher curriculum training; 76% and 17% of first- and second-grade teachers, respectively, attended a follow-up training. The publishers of the comparison curricula provided between 1–2 days of initial training and varying levels of follow-up support. Investigations and SFAW/enVisionMATH trainers offered afterschool sessions, every 4–6 weeks for about 3–4 hours each. Math Expressions trainers conducted classroom observations and provided feedback once or twice a year. For Investigations, all first-grade teachers and 83% of second-grade teachers attended an initial or refresher training; 88% and 69% of first- and second-grade teachers, respectively, attended a follow-up training. For Math Expressions, 92% and 80% of first- and second-grade teachers, respectively, attended an initial or refresher training; 88% and 76% attended a follow-up training. For SFAW and enVisionMATH, 85% and 91% of first- and second-grade teachers, respectively, attended an initial or refresher training; 94% and 79% attended a follow-up training. In addition to formal trainings, teachers received ongoing support from the publishers of each curriculum in person, by phone, and through published materials, as well as instructional support provided by coaches and math specialists in their school or district. Take-up rates for these types of supports are not provided for the analytic sample.

In the case of multiple manuscripts that report on one study, the WWC selects one manuscript as the primary citation and lists other manuscripts that describe the study as additional sources.

  • Agodini, R., & Harris, B. (2010). An experimental evaluation of four elementary school math curricula. Journal of Research on Educational Effectiveness, 3(3), 199–253.

  • Agodini, R., Harris, B., Atkins-Burnett, S., Heaviside, S., Novak, T., & Murphy, R. (2009). Achievement effects of four early elementary school math curricula: Findings from first graders in 39 schools (NCEE 2009–4052). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

  • Agodini, R., Harris, B., Seftor, N., Remillard, J., & Thomas, M. (2013). After two years, three elementary math curricula outperform a fourth. NCEE Technical Appendix (NCEE 2013-4019-APPENDIX). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

  • Agodini, R., Harris, B., Thomas, M., Murphy, R., & Gallagher, L. (2010). Achievement effects of four early elementary school math curricula: Findings for first and second graders (NCEE 2011-4001). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

  • Clements, D. H., Agodini, R., & Harris, B. (2013). Instructional practices and student math achievement: Correlations from a study of math curricula (NCEE 2013-4020). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

  • Remillard, J. T., Harris, B., & Agodini, R. (2014). The influence of curriculum material design on opportunities for student learning. ZDM: The International Journal on Mathematics Education, 46(5), 735-749.

  • Clements, D. H., Agodini, R., & Harris, B. (2013). Technical appendix: Data and methodological approach (NCEE 2013-4020). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Retrieved from https://eric. ed.gov/?&id=ED544192

Reviewed: May 2016

Study sample characteristics were not reported.

Reviewed: May 2016

Study sample characteristics were not reported.

Reviewed: May 2016

Study sample characteristics were not reported.

Reviewed: May 2016

Study sample characteristics were not reported.

Reviewed: May 2016

Study sample characteristics were not reported.

In the case of multiple manuscripts that report on one study, the WWC selects one manuscript as the primary citation and lists other manuscripts that describe the study as additional sources.

  • Remillard, J. T., Harris, B., & Agodini, R. (2014). The influence of curriculum material design on opportunities for student learning. ZDM: The International Journal on Mathematics Education, 46(5), 735-749.

  • Agodini, R., Harris, B., Atkins-Burnett, S., Heaviside, S., Novak, T., & Murphy, R. (2009). Achievement effects of four early elementary school math curricula: Findings from first graders in 39 schools (NCEE 2009–4052). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

  • Agodini, R., Harris, B., Atkins-Burnett, S., Heaviside, S., Novak, T., Murphy, R., & Institute of Education Sciences (ED), National Center for Education Evaluation and Regional Assistance. (2009). Achievement effects of four early elementary school math curricula: Findings from first graders in 39 schools. Executive summary (NCEE 2009–4053). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

  • Agodini, R., Harris, B., Thomas, M., Murphy, R., & Gallagher, L. (2010). Achievement effects of four early elementary school math curricula: Findings for first and second graders (NCEE 2011-4001). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

  • Clements, D. H., Agodini, R., & Harris, B. (2013). Instructional practices and student math achievement: Correlations from a study of math curricula (NCEE 2013-4020). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

  • Agodini, R., & Harris, B. (2010). An experimental evaluation of four elementary school math curricula. Journal of Research on Educational Effectiveness, 3(3), 199–253.

 

Your export should download shortly as a zip archive.

This download will include data files for study and findings review data and a data dictionary.

Connect With the WWC

newsflash icon contact icon facebook icon twitter icon
loading
back to top