|
|
Research
Nineteen studies reviewed by the WWC investigated the effects of Saxon Math. One study (Resendez & Azin, 2006) is a randomized controlled trial that meets WWC evidence standards. Four studies (Peters, 1992; Crawford & Raia, 1986; Resendez, Fahmy, & Manley, 2005, Cohort A; and Resendez, Fahmy, & Manley, 2005, Cohort F) are randomized controlled trials or quasi-experimental designs that meet WWC evidence standards with reservations. Two of these studies (Resendez, Fahmy, & Manley, 2005, Cohort A and Resendez, Fahmy, & Manley, 2005, Cohort F) were included within one research report.6 The remaining 14 studies do not meet either WWC evidence standards or eligibility screens.
Meets evidence standards
Resendez and Azin (2006) conducted a randomized controlled trial to investigate the effect of Saxon Math on math achievement in one northeastern Ohio middle school and one southwestern Ohio junior high school.7 The schools served sixth-, seventh-, and eighth-grade students living in urban and suburban locations; classes were randomly assigned to use Saxon Math or control curricula during the 2005–06 school year. The analysis sample included 14 Saxon Math classrooms (approximately 260 students) and 11 control classrooms (more than 200 students). Classes in the intervention group used one of four Saxon Math curricula (Saxon 7/6, 2004-4th Ed.; Saxon 8/7, 2004-4th Ed.; Saxon Algebra ½, 2004-3rd Ed.; or Saxon Algebra 1, 2003-3rd Ed.); classes in the control group used either a traditional basal program or a mixed curriculum consisting primarily of teacher-created materials.
Meets evidence standards with reservations
Peters (1992) conducted a randomized controlled trial that had non-random allocations after random assignment to investigate the effect of Saxon Math on math achievement of 36 “math-talented” eighth-grade students from one junior high school in Nebraska during the 1991–92 school year.8,9 The district borders two large cities (Lincoln and Omaha) and its students lived in rural and suburban areas. Students in the intervention group used the Saxon Algebra 1 textbook (1981), while students in the comparison group used the University of Chicago School Mathematics Project (UCSMP) Algebra first-edition textbook.
Crawford and Raia (1986) conducted a within-teacher, matched-comparison quasi-experiment to investigate the effect of Saxon Math on math achievement of eighth-grade students in Oklahoma City Public Schools during the 1984–85 school year. The analysis sample included 78 eighth-grade students (39 Saxon and 39 comparison) taught by four teachers from four middle schools. Students in the intervention group used the Saxon Algebra ½ (1983) textbook, and students in the comparison group used the Scott-Foresman Mathematics (1980) textbook.
Resendez, Fahmy, and Manley (2005, Cohort A) conducted a quasi-experiment to investigate the effect of Saxon Math on math achievement in 25 middle schools serving sixth-, seventh-, and eighth-grade students in rural, suburban, and urban districts in Texas. The analysis sample for Cohort A (grade 8 of Sample 1) included 1,472 students from 12 intervention schools who received three years of Saxon Math exposure in grades 6, 7, and 8 and 1,582 students from 13 comparison schools during the 1998–99 through 2000–01 school years. Schools in the intervention group used three Saxon Math curricula (Saxon 7/6, Saxon 8/7, or Saxon Algebra ½); the majority of schools in the comparison group used core basal math curricula, which typically consist of a chapter-based approach to math instruction.10
Resendez, Fahmy, and Manley (2005, Cohort F) conducted a quasi-experiment to investigate the effect of Saxon Math on math achievement in 20 middle schools serving sixth-, seventh-, and eighth-grade students in rural, suburban, and urban districts in Texas. The analysis sample for Cohort F (grade 6 of Sample 3) included 1,526 sixth-grade students from ten intervention schools who received one year of Saxon Math exposure and 1,407 students from ten comparison schools during the 2003–04 school year. Schools in the intervention group used two Saxon Math curricula (Saxon 7/6 or Saxon 8/7); the majority of schools in the comparison group used core basal math curricula, which typically consist of a chapter-based approach to math instruction.10
Extent of evidence
The WWC categorizes the extent of evidence in each domain as small or medium to large (see the WWC Procedures and Standards Handbook, Appendix G). The extent of evidence takes into account the number of studies and the total sample size across the studies that meet WWC evidence standards with or without reservations.11
The WWC considers the extent of evidence for Saxon Math to be medium to large for math achievement.
6 The study authors reported on three independent samples containing multiple cohorts of students. Sample 2 was excluded from WWC review because this sample was used for a pre-post analysis of the students in Saxon Math schools and did not include a comparison group. Cohorts B and C of Sample 1 were included only in an analysis of tenth-grade math performance. Because it is unknown whether the intervention and comparison groups for these cohorts attended similar schools in grades 9 and 10, it is impossible to determine whether the effects can be attributed solely to Saxon Math; therefore, the WWC excluded these cohorts from this review. Cohorts G and H of Sample 3 were excluded because pre-Saxon math achievement data were not available for these students and, consequently, baseline equivalence could not be established. Cohort A (grade 8 of Sample 1) and Cohort F (grade 6 of Sample 3) were treated as separate studies because they examined the effects of Saxon Math on different samples of students at two different
times. Students in Cohort A were in grade 8 in 2000–01; students in Cohort F were in grade 6 in 2003–04.
7 In the case of Resendez and Azin (2006), differences between intervention and control groups on pretest, and several demographic characteristics, were statistically significant at the student level but not at the classroom level. The study authors statistically controlled for these baseline differences in their analysis. Because random assignment was well executed, the WWC categorizes the study as a randomized controlled trial.
8 Peters (1992) compared Saxon Math with UCSMP Algebra. The author indicates that a random selection of numbers was used to divide participants between the intervention and comparison groups. However, the assignment of students was altered to accommodate scheduling difficulties and student requests for other course offerings. Therefore, the study meets standards with reservations, according to WWC criteria.
9 The “math-talented” designation is based on teacher recommendations and prior academic achievement. No information is provided on the specific thresholds that were used in delineating the math-talented criteria; however, all students in the sample scored at or above the 87th percentile on the California Achievement Test total math battery.
10 The authors did not provide information on the edition and publication year of the Saxon Math texts used in the study.
11 The extent of evidence categorization was developed to assess how much evidence was used to determine the intervention rating, focusing on the number and size of studies. Additional factors associated with a related concept—external validity, such as the students’ demographics and the types of settings in which studies took place—are not taken into account for the categorization. Information about how the extent of evidence rating was determined for Saxon Math is in Appendix A6.