The study’s main findings include information about curriculum implementation and
the relative effects of the curricula on student math achievement. Statistical tests
were used to assess the significance of all the results. Hierarchical linear modeling
(HLM) techniques—which account for the extent to which students are clustered in
classrooms and schools according to achievement—were used to conduct the statistical
tests. When comparing results for pairs of curricula, the Tukey-Kramer method (Tukey
1952, 1953; Kramer 1956) was used to adjust the statistical tests for the six unique
pair-wise comparisons that can be made with four curricula, as described above.
Only results that are statistically significant at the 5 percent level of confidence
are discussed.^{5}

Before presenting the main findings, it is worth mentioning the information that
is and is not provided by the study. The relative effects of the curricula presented
below reflect differences between the curricula, including differences in teacher
training, instructional strategies, content coverage, and curriculum materials.
Of course, the relative effects ultimately depend on how teachers implemented the
curricula, and implementation reflects what publishers and teachers achieved, not
some level of implementation specified by the study. Information about curriculum
implementation presented in this report is based only on teacher reports—the study
team is observing classrooms and plans to present that information in a future report.^{6} Also, the relative effects of the curricula
are based only on the ECLS-K math assessment administered by the study team—in the
third grade and perhaps even the second grade, districts administer their own math
assessments to students and the study team is investigating the possibility of obtaining
those scores for our future analyses of second and third graders. Lastly, because
the participating sites are not a representative sample of districts and schools,
the design does not support making statements about effects for districts and schools
outside of the study.

Curriculum Implementation. The main findings from the implementation analysis are:

- All teachers received initial training from the publishers and 96 percent received follow-up training. Taken together, training varied by curriculum, ranging from 1.4 to 3.9 days.
- Nearly all teachers (99 percent in the fall, 98 percent in the spring) reported using their assigned curriculum as their core math curriculum according to the fall and spring surveys, and about a third (34 percent in fall and 36 percent in spring) reported supplementing their curriculum with other materials.
- Eighty-eight percent of teachers reported completing at least 80 percent of their
assigned curriculum.
^{7} - On average, Saxon teachers reported spending one more hour on math instruction per week than did teachers of the other curricula.

Achievement Effects. The figure below illustrates the relative effects of the study’s curricula on student math achievement. The figure includes a symbol for each of the four curricula, where the dot in the middle of each symbol indicates the average spring math score of students in the respective curriculum groups. The average scores are adjusted for baseline measures of several student, teacher, and school characteristics related to student spring achievement (such as student fall math scores) to improve the precision of the results. The bars that extend from each dot represent the 95 percent confidence interval around each average score. HLM techniques were used to calculate the average scores and confidence intervals.

Curricula with non-overlapping confidence intervals have average scores that are significantly different at the 5 percent level of confidence. The results are presented in standard deviations, which means that subtracting the average values (the dots) for any two curricula indicates the effect size of using the first curriculum instead of the second. The effect sizes discussed below were calculated by dividing each pair-wise curriculum comparison by the pooled standard deviation of the spring score for the two curricula being compared, and Hedges’ g formula (with the correction for small-sample bias) was used to calculate the pooled standard deviations. Appendix D presents averages of the unadjusted math scores (see Table D.3). The relative effects of the curricula described below are similar when based on the simple averages, although the confidence intervals are wider than those based on the HLM-adjusted averages, as expected.

The figure shows that:

**Student math achievement was significantly higher in schools assigned to Math Expressions and Saxon, than in schools assigned to Investigations and SFAW.**Average HLM-adjusted spring math achievement of Math Expressions and Saxon students was 0.30 standard deviations higher than Investigations students, and 0.24 standard deviations higher than SFAW students. For a student at the 50th percentile in math achievement, these effects mean that the student’s percentile rank would be 9 to 12 points higher if the school used Math Expressions or Saxon, instead of Investigations or SFAW.**Math achievement in schools assigned to the two more effective curricula (Math Expressions and Saxon) was not significantly different, nor was math achievement in schools assigned to the two less effective curricula (Investigations and SFAW).**The Math Expressions-Saxon and Investigations-SFAW differentials equal 0.02 and -0.07 standard deviations, respectively, and neither is statistically significant.

We also examined whether the relative effects of the curricula differ along six characteristics that differentiate instructional settings: (1) participating districts, (2) school fall achievement, (3) school free/reduced-price meals eligibility, (4) teacher education, (5) teacher experience, and (6) teacher math content/pedagogical knowledge that was measured before curriculum training began using an assessment administered by the study team. These characteristics were used to create 15 subgroups—one for each of the four districts, three based on school fall achievement, and two subgroups for each of the other four characteristics.

Eight of the fifteen subgroup analyses found statistically significant differences in student math achievement between curricula. The significant curriculum differences ranged from 0.28 to 0.71 standard deviations, and all of the significant differences favored Math Expressions or Saxon over Investigations or SFAW. There were no subgroups for which Investigations or SFAW showed a statistically significant advantage.