After Two Years, Three Elementary Math Curricula Outperform a Fourth. NCEE Evaluation Brief. NCEE 2013-4019

Findings

General Mathematics Achievement outcomes—Indeterminate effect found for the domain

Outcome measure	Comparison	Period	Sample	Intervention mean	Comparison mean	Significant?	Improvement     index	Evidence tier
Early Childhood Longitudinal Study- Kindergarten (ECLS-K): 2nd grade adaptation	Saxon Math vs. Investigations in Number, Data, and Space®	2 Years	Grade: 2; 882 students	71.72	67.31	Yes	10
Early Childhood Longitudinal Study- Kindergarten (ECLS-K): 2nd grade adaptation	Saxon Math vs. SFAW/enVisionMATH	2 Years	Grade: 2; 1,136 students	68.90	68.87	No	--
Early Childhood Longitudinal Study- Kindergarten (ECLS-K): 2nd grade adaptation	Saxon Math vs. Math Expressions	2 Years	Grade: 2; 931 students	67.56	67.99	No	--
Show Supplemental Findings
Early Childhood Longitudinal Study- Kindergarten (ECLS-K): 2nd grade adaptation	Saxon Math vs. Investigations in Number, Data, and Space®	1 Year	Grade: 1 first cohort; 636 students	47.56	44.87	Yes	13
Early Childhood Longitudinal Study- Kindergarten (ECLS-K): 2nd grade adaptation	Saxon Math vs. Scott Foresman-Addison Wesley Elementary Mathematics	1 Year	Grade: 1, first cohort; 663 students	46.17	44.28	Yes	9
Early Childhood Longitudinal Study- Kindergarten (ECLS-K): 2nd grade adaptation	Saxon Math vs. Scott Foresman-Addison Wesley Elementary Mathematics	1 Year	Grade: 2, both cohorts; 1,706 students	72.98	70.31	Yes	7
Early Childhood Longitudinal Study- Kindergarten (ECLS-K): 2nd grade adaptation	Saxon Math vs. Investigations in Number, Data, and Space®	1 Year	Grade: 2, both cohorts; 1,711 students	71.88	69.85	Yes	5
Early Childhood Longitudinal Study- Kindergarten (ECLS-K): 2nd grade adaptation	Saxon Math vs. Math Expressions	1 Year	Grade: 2, both cohorts; 1,721 students	73.09	71.38	No	--
Early Childhood Longitudinal Study- Kindergarten (ECLS-K): 2nd grade adaptation	Saxon Math vs. Scott Foresman-Addison Wesley Elementary Mathematics	1 Year	Grade: 1 both cohorts; 2,377 students	45.08	44.43	No	--
Early Childhood Longitudinal Study- Kindergarten (ECLS-K): 2nd grade adaptation	Saxon Math vs. Investigations in Number, Data, and Space®	1 Year	Grade: 1, both cohorts; 2,235 students	45.13	44.51	No	--
Early Childhood Longitudinal Study- Kindergarten (ECLS-K): 2nd grade adaptation	Saxon Math vs. Math Expressions	1 Year	Grade: 1, first cohort; 618 students	45.40	45.45	No	--
Early Childhood Longitudinal Study- Kindergarten (ECLS-K): 2nd grade adaptation	Saxon Math vs. Math Expressions	1 Year	Grade: 1, both cohorts; 2,320 students	44.52	44.74	No	--

Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.

Study Details

Setting

The study was conducted in 111 schools in 12 districts in 10 states (Connecticut, Florida, Kentucky, Minnesota, Mississippi, Missouri, Nevada, New York, South Carolina, and Texas). Of the 12 districts, three were in urban areas, five were in suburban areas, and four were in rural areas. The study data were collected during the 2006–07, 2007–08, and 2008–09 school years. Of the 111 schools, 58 participated in the study for 2 years. These 58 schools were located in seven districts in up to seven states (the authors did not specify which states).

Study sample

The study authors randomly assigned 111 schools within 12 school districts to one of four math curricula (Saxon Math, Investigations, Math Expressions, or SFAW). Random assignment was conducted within district and stratified on characteristics such as school size, free or reduced-price meal eligibility, math proficiency, and race/ethnicity. Random assignment was conducted before the school year began in the first year of the study, and the student sample was defined immediately prior to the pretest. Thus, the study may have included students in the analytic sample that enrolled in schools after random assignment. Within study schools, all students at the target grade levels (first and/or second grades) used their school’s assigned curriculum. Approximately 30 students per grade level were randomly sampled for testing by the study team. The primary findings that contribute to the rating of effectiveness are based on students from 58 of the 111 schools (within seven of the 12 districts) who participated in the study and used their assigned curriculum for 2 consecutive years. This analytic sample was comprised of 2,045 students in 222 classrooms who experienced their assigned curriculum in first and second grades. Students were pretested at the beginning of first grade and posttested at the end of second grade. Of the 58 study schools, 12 used Saxon Math, 14 used Investigations, 14 used Math Expressions, and 18 used SFAW/enVisionMATH. In the analytic sample of students, 49% were female, 40% were non-Hispanic Black, 32% were other non-Hispanic, and 28% were Hispanic. Students with limited English proficiency or classified as English language learners were 17% of the sample, and 9% of students had individualized education plans (IEPs) or were receiving special education services. This review also includes supplemental findings from two study reports. The first set of supplemental findings is based on the 2009 report that presented findings for 1,309 first-grade students in 39 schools that participated in the study during the 2006–07 school year. The second set of findings is from the 2010 report that presented finding for 8,060 first- or second-grade students in 110 schools that used the study curricula for 1 year and participated in the study in either the 2006–07 or 2007–08 school years. In the 2009 and 2010 reports, the findings are 1-year effects where students were pre- and posttested in first or second grade, and findings are reported separately by grade. The supplemental findings are presented in Appendix D and do not factor into the intervention’s rating of effectiveness.

Intervention Group

Students in the intervention group used Saxon Math as their core curriculum in the first and second grades in the 2006–07, 2007–08, or 2008–09 school years. The study did not specify which edition of Saxon Math was used but indicated that the 2005 and 2008 copyright years were used. All teachers in the intervention group reported using Saxon Math as their core math curriculum in first and second grades and provided, on average, 6 hours of math instruction per week in each grade. Additionally, most teachers (87% in first grade and 71% in second grade) reported completing at least 80% of the lessons from Saxon Math.

Comparison Group

Students in the comparison group used Investigations, Math Expressions, or SFAW/enVisionMATH as their core math curriculum. Investigations in Number, Data, and Space is a K–5, student-centered curriculum that emphasizes reasoning and communicating about math concepts, using multiple approaches to problem solving, through in-depth investigations of problems. Study schools used the first or second editions of the curriculum. All teachers in the first grade and 96% in the second grade reported using Investigations as their core math curriculum and provided, on average, 5 hours of math instruction per week in each grade. Additionally, most teachers (86% in first grade and 56% in second grade) reported completing at least 80% of the lessons from Investigations. Math Expressions is a K–5 curriculum that uses a combination of teacher-directed and student-centered approaches within consistent daily routines. The study schools used the 2005 or 2008 copyright years of the curriculum. Approximately 98% of teachers in the first grade and all teachers in the second grade reported using Math Expressions as their core math curriculum and provided, on average, 5 hours of math instruction per week in each grade. Additionally, most teachers (82% in first grade and 79% in second grade) reported completing at least 80% of the lessons from Math Expressions. SFAW is a K–6, teacher-directed curriculum that uses a consistent daily structure, involving a brief review of previous material, exploration of a new concept, explicit instruction on the new concept, and a closure activity to check student understanding. During the course of the study, the publisher revised the SFAW curriculum and renamed it enVisionMATH. This change affected four of the seven study districts. Results for this comparison group should be interpreted as a mix of students just receiving SFAW curriculum in grades 1–2, and some that received SFAW in grade 1 and enVisionMATH in grade 2. Outcomes that include grade 2 in the 2008–09 school year are referred to as SFAW/enVisionMATH. All teachers in the first grade and 99% in the second grade reported using SFAW or enVisionMATH as their core math curriculum and provided, on average, 5 hours of math instruction per week in each grade. Additionally, most teachers (87% in first grade and 71% in second grade) reported completing at least 80% of the lessons from SFAW or enVisionMATH.

Support for implementation

Training was provided by Houghton Mifflin Harcourt trainers to Saxon Math teachers. Saxon teachers were provided 1 day of initial training in the summer before the school year began and one follow-up training session in the fall, tailored to meet each district’s needs. In the follow-up training, some teachers watched demonstration lessons or participated in a math workshop. In other cases, trainers observed teachers conduct lessons and provided feedback afterwards. Approximately 90% and 81% of first- and second-grade Saxon teachers, respectively, attended an initial or refresher curriculum training; 76% and 17% of first- and second-grade teachers, respectively, attended a follow-up training. The publishers of the comparison curricula provided between 1–2 days of initial training and varying levels of follow-up support. Investigations and SFAW/enVisionMATH trainers offered afterschool sessions, every 4–6 weeks for about 3–4 hours each. Math Expressions trainers conducted classroom observations and provided feedback once or twice a year. For Investigations, all first-grade teachers and 83% of second-grade teachers attended an initial or refresher training; 88% and 69% of first- and second-grade teachers, respectively, attended a follow-up training. For Math Expressions, 92% and 80% of first- and second-grade teachers, respectively, attended an initial or refresher training; 88% and 76% attended a follow-up training. For SFAW and enVisionMATH, 85% and 91% of first- and second-grade teachers, respectively, attended an initial or refresher training; 94% and 79% attended a follow-up training. In addition to formal trainings, teachers received ongoing support from the publishers of each curriculum in person, by phone, and through published materials, as well as instructional support provided by coaches and math specialists in their school or district. Take-up rates for these types of supports are not provided for the analytic sample.

Additional Sources