Appendix A1.1 Study characteristics: Ridgway, Zawojewski, Hoover, & Lambdin, 2002 (quasi-experimental design)
| Characteristic | Description |
|---|---|
| Study citation | Ridgway, J. E., Zawojewski, J. S., Hoover, M. N., & Lambdin, D. V. (2002). Student attainment in the Connected Mathematics curriculum. In S. L. Senk & D. R. Thompson (Eds.), Standards-based school mathematics curricula: What are they? What do students learn? Mahwah, N. J. : Lawrence Erlbaum Associates, Inc. |
| Participants | The 1994/95 sample included over 340 sixth graders and 630 seventh graders from 9 CMP schools (2 classrooms per grade from each school), and 160 sixth graders and 250 seventh graders from 9 comparison schools (1 classroom per grade from each school).1 The 1995/96 sample included over 780 eighth-graders from an unspecified number of CMP schools and 300 eighth graders from an unspecified number of comparison schools. Some students were included in both the 1994/95 sample and the 1995/96 sample. Demographic characteristics of the participants are not reported. The WWC review only included findings from the sixth grade, because baseline data were available only for the sixth-grade students and not for the other grade levels. |
| Setting | Participating classrooms were from schools located in the midwestern, western, and eastern regions of the country. |
| Intervention | Teachers in the intervention group were using the CMP as the core curriculum throughout the school year. The study authors did not report how the program was actually implemented in those classrooms. All the sixth-grade students in the study were new to the program, and about three-fourths of the seventh- and eighth-grade students in the study had used the program in the previous year. |
| Comparison | Teachers in the comparison group did not implement the CMP, nor were they involved in any reform efforts. Data were not available about the mathematics textbook series used by those teachers. |
| Primary outcomes and measurement | Math achievement was assessed using the Iowa Test of Basic Skills (ITBS) Survey Battery and Balanced Assessment (BA) Test (see Appendix A2 for more detailed descriptions of the outcome measure). |
| Teacher training | All CMP teachers attended the summer CMP institutes at Michigan State University. |
|
1 The WWC requested and received from one of the study authors information about the number of schools that participated in the study. |
|
Appendix A1.2 Study characteristics: Riordan & Noyce, 2001 (quasi-experimental design)
| Characteristic | Description |
|---|---|
| Study citation | Riordan, J. E., & Noyce, P. E. (2001). The impact of two standards-based mathematics curricula on student achievement in Massachusetts. Journal for Research in Mathematics Education, 32 (4), 368–398. |
| Participants | This study looked at 20 CMP schools with 1,879 eighth-graders and 30 matched comparison schools with 4,978 eighth-graders. Overall, 10% of the student participants were eligible for free or reduced-price lunches, and 87% were white. All students were regular education students.1 |
| Setting | This study included relatively advantaged middle schools with predominantly white students and a low percentage of students receiving free or reduced-price lunches in Massachusetts. |
| Intervention | Schools in the intervention group had implemented at least 11 student units in grades 6–8 by 1998/99, but none of the schools implemented all eight units that the CMP has available for each grade. Further, it is not clear how the program was actually implemented in those schools. The 20 schools in the intervention group had implemented the program for two to three years. |
| Comparison | The 30 comparison schools did not implement the CMP, but used 15 different textbook programs, which, in the aggregate, represented the instructional norm in Massachusetts. The most commonly used programs were those published by Heath, Addison-Wesley, Prentice Hall, and Houghton-Mifflin. |
| Primary outcomes and measurement | Math achievement was measured using the Massachusetts Comprehensive Assessment System (see Appendix A2 for more detailed descriptions of the outcome measure). |
| Teacher training | No information about teacher training was provided. |
|
1 This study also included an additional separate examination of a single intervention school that has been using the program for four years and its four comparison schools. Because the results of students in the single intervention schools were confounded with the school site, this analysis was not included in the WWC review. |
|
Appendix A1.3 Study characteristics: Schneider, 2000 (quasi-experimental design)
| Characteristic | Description |
|---|---|
| Study citation | Schneider, C. L. (2000). Connected Mathematics and the Texas Assessment of Academic Skills. Dissertation Abstracts International, 61 (12), 4709A. (UMI No. 9997642). |
| Participants | The study included 3 cohorts from 23 CMP schools and 25 matched comparison schools overall. Cohort 1 was from 23 intervention and 19 comparison schools. Cohort 2 was from 22 intervention and 19 comparison schools. Cohort 3 was from 18 intervention and 18 comparison schools. For TAAS TLI scores, data was collected from 16 schools per condition (CMP, comparison) and per cohort (that is, cohort 1, cohort 2, cohort 3), because six schools were omitted from the analysis due to missing data or because the intervention was discontinued. Not all cohorts participated in every analysis because different grade levels were followed through for each cohort. |
| Setting | The participating schools were located in rural, suburban, and urban and both low and high socioeconomic areas of Texas. Those schools varied in the racial composition, socioeconomic status, special education status, and English language learner status of the student populations that they served. Many of the schools had predominantly minority student populations. |
| Intervention | Schools in the intervention group were using the CMP for grades 6–8. There were substantial variations in the extent to which the curriculum was used at each grade and each year across these schools. The three cohorts in the intervention group received the CMP for three years, two years, and one year, respectively, between 1996/97 and 1998/99. |
| Comparison | The 25 comparison schools did not implement the CMP, and it is unclear what mathematics curricula they were using. |
| Primary outcomes and measurement | Students' mathematic achievement was measured using the Texas Assessment of Academic Skills (TAAS) passing rate and Texas Learning Index (TLI) (see Appendix A2 for more detailed descriptions of the outcome measures). |
| Teacher training | Teachers who taught grade 6, 7, or 8 at the 23 CMP schools participated in a six-day summer professional development conducted by the Texas Statewide Systemic Initiative in 1996, 1997, and 1998. |
Appendix A2 Outcome measures in the math achievement domain
| Outcome measure | Description |
|---|---|
| Iowa Test of Basic Skills (ITBS) | The ITBS is a norm-referenced state standardized test. The mathematics sections measure the following math abilities: problem solving, data interpretation, math concepts, estimation, and computation (as cited in Ridgway, Zawojewski, Hoover, & Lambdin, 2002). The Ridgway et al. (2002) study used the ITBS Survey Battery Form K (levels 12,13, and 14). |
| Balanced Assessment (BA) Test | The BA test was designed to assess students' math achievement in a variety of curricular areas through constructed-response items that require a range of responses from short answer to extended response. It was developed through the collaboration between the CMP developer and the Balanced Assessment Project (as cited in Ridgway et al., 2002). |
| Massachusetts Comprehensive Assessment System (MCAS) | The MCAS is a criterion-referenced state standardized test that includes both multiple-choice and open-response questions (as cited in Riordan & Noyce, 2001). |
| Texas Assessment of Academic Skills (TAAS)–passing rates | The TAAS is a criterion-referenced state standardized test that measures problem-solving and critical-thinking skills. The passing rate is the percentage of students that reached a proficient level (as cited in Schneider, 2000). |
| Texas Assessment of Academic Skills (TAAS)–Texas Learning Index (TLI) | The TAAS is a criterion-referenced state standardized test that measures problem-solving and critical-thinking skills. TLI is a TAAS-based statistic designed for comparing student progress between administrations and between grades (as cited in Schneider, 2000). |
Appendix A3 Summary of study findings included in the rating for the math achievement domain1
| Author's findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome (standard deviation2) | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (schools/students) | CMP group3 | Comparison group | Mean difference4 (CMP – comparison) | Effect size5 | Statistical significance6 (at α= 0.05) | Improvement index7 |
| Ridgway, Zawojewski, Hoover, & Lambdin, 2002 (quasi-experimental design)8 | ||||||||
| ITBS | Grade 6 | 18/500 | 8.10 (2.30) | 8.60 (2.70) | -0.50 | -0.20 | ns | -8 |
| Balanced Assessment Test | Grade 6 | 18/500 | 21.50 (12.60) | 18.10 (11.90) | 3.40 | 0.27 | ns | +11 |
| Average9 for math achievement (Ridgway, Zawojewski, Hoover, & Lambdin, 2002) | 0.04 | ns | +1 | |||||
| Riordan & Noyce, 2001 (quasi-experimental design)8 | ||||||||
| MCAS | Grade 8 | 50/6,857 | 238.20 (9.08) | 233.85 (10.57) | 4.35 | na10 | Statistically significant | na10 |
| Average9 for math achievement (Riordan & Noyce, 2001) | na10 | Statistically significant | na10 | |||||
| Schneider, 2000 (quasi-experimental design)8 | ||||||||
| TAAS TLI score | Grade 6–8 (Cohort 1) | 32/5,701 | 73.20 (6.37) | 72.90 (5.46) | 0.30 | 0.05 | ns | +2 |
| TAAS TLI score | Grades 6–7 (Cohort 2) | 32/6,461 | 72.30 (6.59) | 74.20 (5.29) | -1.90 | -0.32 | ns | -12 |
| TAAS TLI score | Grade 6 (Cohort 3) | 32/7,339 | 73.60 (5.41) | 74.50 (6.37) | -0.90 | -0.15 | ns | -6 |
| Average9 for math achievement (Schneider, 2000) | -0.14 | ns | -6 | |||||
| Domain average9 for academic achievement across all studies | -0.05 | na | -2 | |||||
|
ns = not statistically significant 1 This appendix reports findings considered for the effectiveness rating and the average improvement indices. Additional findings are presented in Appendix A4. 2 The standard deviation across all students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. Means and standard deviations for the Riordan & Noyce (2001) and Schneider (2000) studies were received from the study authors. 3 In the Ridgeway et al. (2002) study, the CMP group mean equals the comparison group mean plus the mean difference. The computation of the mean difference took into account the pretest difference between the study groups. 4 Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 5 For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. 6 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 7 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and the percentile rank of the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results. 8 The level of statistical significance was reported by the study authors, or where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Ridgway, Zawojewski, Hoover, & Lambdin (2002) corrections for clustering and multiple comparisons were needed, so the significance levels may differ from those reported in the original study. In the case of Schneider (2000), corrections for clustering and multiple comparisons were not needed. Riordan & Noyce (2001) examined student outcomes for low socioeconomic status (SES) students and for male and female students separately, and reported statistically significant positive effects favoring the CMP group for low-SES and female students. The results of these analyses were not included for rating purposes because of overlap with results for the entire sample. In addition, standard deviations were not available for this subset of findings. 9 The WWC-computed average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average improvement indices are calculated from the average effect size. 10 Student-level standard deviations were not available for this study. School-level standard deviations were 9.08 for the intervention group and 10.57 for the comparison group. Because the student-level effect size and improvement index could not be computed, the magnitude of the effect size was not considered for rating purposes. However, the statistical significance for this study is comparable to other studies and is included in the intervention rating. For further details, please see Technical Details of WWC-Conducted Computations. |
||||||||
Appendix A4 Summary of additional findings for the math achievement domain1
| Author's findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome (standard deviation2) | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (schools/students) | CMP group | Comparison group | Mean difference3 (CMP – comparison) | Effect size4 | Statistical significance5 (at α= 0.05) | Improvement index6 |
| Schneider, 2000 (quasi-experimental design)7 | ||||||||
| TAAS pass rates | Grade 8 (Cohort 1) | 48/1,440 | 80 percent of students (na) | 86 percent of students (na) | 0.65 | -0.26 | ns | -10 |
| TAAS pass rates | Grade 7 (Cohort 2) | 42/1,260 | 76 percent of students (na) | 83 percent of students (na) | 0.65 | -0.26 | ns | -10 |
| TAAS pass rates | Grade 6 (Cohort 3) | 42/1,260 | 78 percent of students (na) | 82 percent of students (na) | 0.78 | -0.15 | ns | -6 |
|
ns = not statistically significant 2 The standard deviation across all students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. 3 Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 4 Effect sizes for the binary outcome measure of TAAS pass rates in Schneider (2000) were calculated using the odds ratio formula and converted to standardized mean difference. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. 5 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 6 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and the percentile rank of the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results. 7 The level of statistical significance was reported by the study authors, or where necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple comparisons were not done for findings not included in the overall intervention rating). For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Schneider (2000), no correction for clustering was needed. |
||||||||
Appendix A5 Connected Mathematics Project rating for the math achievement domain
The WWC rates the effects of an intervention for a given outcome domain as: positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.1
For the outcome domain of math achievement, the WWC rated CMP as having mixed effects. It did not meet the criteria for positive effects or potentially positive effects because only one study showed a statistically significant positive effect and two studies showed indeterminate effects. The remaining ratings (no discernible effects, potentially negative effects, or negative effects) were not considered as CMP was assigned the highest applicable rating.
| Rating received |
|---|
|
Mixed effects: Evidence of inconsistent effects as demonstrated through either of the following criteria.
|
| Other ratings considered |
|
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.
|
|
Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence.
|
|
1 For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain level effects. The WWC also considers the size of the domain level effects for ratings of potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. |
|Institute of Education Sciences
