Appendix A1 Study characteristics: Schneider, 2000 (quasi-experimental design)
| Characteristic | Description |
|---|---|
| Study citation | Schneider, C. L. (2000). Connected Mathematics and the Texas Assessment of Academic Skills (Doctoral dissertation, University of Texas at Austin, 2000). Dissertation Abstracts International, 61(12), 4709A. (UMI No. 9997642) |
| Participants | The study included three cohorts from 23 CMP schools and 25 matched comparison schools overall. However, because baseline equivalence was established only for cohorts 1 and 2 through 1998, the WWC excluded cohort 3 from this review.1 Cohort 1 included more than 3,000 CMP students and 2,600 comparison students. Cohort 2 included more than 3,400 CMP students and 2,900 comparison students. |
| Setting | The participating schools were located in rural, suburban, and urban, as well as both low and high socioeconomic, areas of Texas. The schools varied in the English language learner status of the student populations that they served. Many of the schools had predominantly minority student populations. |
| Intervention | Schools in the treatment group used CMP, starting with grade 6 in 1996–97, adding grade 7 in 1997–98, and adding grade 8 in 1998–99. Cohorts in the school-level analysis represented all students in grades using CMP, regardless of whether the students were enrolled every year of implementation. By 1998, students in cohort 1 had received the intervention for up to two years (grades 6 and 7), and students in cohort 2 had received the intervention for one year (grade 6). There was substantial variation in the extent to which the curriculum was used at each grade level and each year across these schools. |
| Comparison | Schools in the comparison group did not use CMP. The author did not report the mathematics curricula used by comparison schools. The 25 comparison schools were matched to treatment schools using a regression analysis of variables that predicted 1996 Texas Assessment of Academic Skills (TAAS) passing rates. |
| Primary outcomes and measurement | The primary outcome measure included in this review was the school-level passing rate on the mathematics portion of the TAAS. For a more detailed description of this outcome measure, see Appendix A2. |
| Staff/teacher training | In the summer prior to implementation, teachers in the CMP schools participated in a six-day summer professional development provided by the Texas Statewide System Initiative. The training discussed units and emphasized mathematical understanding and pedagogy. Many teachers also participated in a two-day follow-up professional development in the summer after implementation. |
|
1 The author conducted both student-level and school-level analyses. However, baseline equivalence was established only for the school-level analysis. In addition, the sample of students for whom achievement scores were reported changed over time—special education students were included in 1999 achievement data but were excluded in prior years. Because baseline equivalence was established only for regular education students, the WWC excluded the 1999 data from this review. This exclusion of the 1999 data also resulted in the exclusion of cohort 3 because 1999 is the only year for which posttest data were reported for cohort 3. |
|
Appendix A2 Outcome measure FOR the math achievement domain
| Outcome measure | Description |
|---|---|
| Texas Assessment of Academic Skills (TAAS) pass rate | The TAAS is a criterion-referenced test that measures problem-solving and critical-thinking skills. The measure used is the percentage of students that passed the math portion of the TAAS (as cited in Schneider, 2000). |
Appendix A3 Summary of study findings included in the rating for the math achievement domain1
Author's findings from the study |
||||||||
|---|---|---|---|---|---|---|---|---|
Mean outcome (standard deviation)2 |
WWC calculations |
|||||||
Outcome measure |
Study sample |
Sample size (schools/students) |
CMP group |
Comparison group |
Mean difference3 (CMP – comparison) |
Effect size4 |
Improvement index6 |
|
Schneider, 2000 (quasi-experimental design)7 |
||||||||
| TAAS pass rate | Grades 6 & 7
(Cohorts 1 & 2)7 |
96/>12,162 |
82%8 of students (na) |
82%9 of students (na) |
0.00 | 0.00 | ns | 0 |
| Average for math achievement (Schneider, 2000)9 | 0.0010 | ns | 0 | |||||
| Domain average for academic achievement across all studies9 | 0.00 | na | 0 | |||||
|
ns = not statistically significant 2 The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. 3 Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 4For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B. 5 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 6 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between –50 and +50, with positive numbers denoting results favorable to the intervention group. 7 The WWC calculated a sample-weighted average of the cohort 1 and 2 effects to use in the intervention rating. Cohort 1, the 7th grade cohort, had the intervention for two years and cohort 2, the 6th grade cohort, had the intervention for one year. Separate findings for cohorts 1 and 2 are reported in Appendix A4. 8 The intervention group value from Schneider (2000) is the unadjusted comparison group mean plus the difference in mean gains between the intervention (CMP) and comparison groups. 9 The comparison group mean from Schneider (2000) is unadjusted. 10 The WWC used school-level pass rates to calculate the effect size because student-level pass rates were not available. In general, the school-level figures should serve as a reasonable approximation of the individual figures if there are not large differences in pass rates between schools of substantially different enrollments. Based on prior pass rates (1996) provided by the author in the appendix, the WWC concluded that the pass rates computed using school-level data and student-level data were not substantially different. As a result, based on the available school-level pass rates, the WWC estimated that the mean difference of 0 percentage points resulted in an effect size of 0. |
||||||||
Appendix A4 Summary of cohort findings for the math achievement domain1
Author's findings from the study |
||||||||
|---|---|---|---|---|---|---|---|---|
Mean outcome (standard deviation)2 |
WWC calculations |
|||||||
Outcome measure |
Study sample |
Sample size (schools/students) |
CMP group3 |
Comparison group4 |
Mean difference5 (CMP – comparison) |
Effect size6 |
Improvement index8 |
|
Schneider, 2000 (quasi-experimental design)9 |
||||||||
| TAAS pass rates | Grade 7 (Cohort 1) | 48/>5,701 | 0.80 (na) |
0.80 (na) |
0.00 (na) |
0.00 | ns | 0 |
| TAAS pass rates | Grade 6 (Cohort 2) | 48/>6,461 | 0.84 (na) |
0.83 (na) |
0.01 (na) |
0.06 | ns | +2 |
|
ns = not statistically significant 2 The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. 3 The intervention group value from Schneider (2000) is the unadjusted comparison group mean plus the difference in mean gains between the intervention (CMP) and comparison groups. 4 The comparison group mean from Schneider (2000) is unadjusted. 5 Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 6 For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B. 7 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 8 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between –50 and +50, with positive numbers denoting results favorable to the intervention group. 9 The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. For the formulas the WWC used to calculate statistical significance, see WWC Procedures and Standards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D for multiple comparisons. In the case of Schneider (2000), no corrections for clustering or multiple comparisons were needed. |
||||||||
Appendix A5 CMP rating for the math achievement domain
The WWC rates an intervention’s effects for a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.1
For the outcome domain of math achievement, the WWC rated CMP as having no discernible effects.
Rating received |
|---|
|
No discernible effects: No affirmative evidence of effects.
|
Other ratings considered |
|
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.
AND |
|
Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence.
AND |
Mixed effects: Evidence of inconsistent effects as demonstrated through either of the following criteria.
OR |
Potentially negative effects: Evidence of a negative effect with no overriding contrary evidence.
AND |
Negative effects: Strong evidence of a negative effect with no overriding contrary evidence.
AND |
| 1 For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of potentially positive or potentially negative effects. For a complete description, see the WWC Procedures and Standards Handbook, Appendix E. |
Appendix A6 Extent of evidence by domain
| Sample size | ||||
|---|---|---|---|---|
| Outcome domain | Number of studies | Schools | Students | Extent of evidence1 |
| Math achievement | 1 | 96 | >12,1622 | Small |
| 1A rating of “medium to large” requires at least two studies and two schools across studies in one domain and a total sample size across studies of at least 350 students or 14 classrooms. Otherwise, the rating is “small.” For more details on the extent of evidence categorization, see the WWC Procedures and Standards Handbook, Appendix G. 2 Schneider (2000) reported the number of students only for the student-level analysis. The student-level analysis included only students who were at the same school every year of implementation, whereas the school-level analysis reviewed in this report includes all students in the schools, regardless of whether they were enrolled every year. The WWC used the number of students in the student-level analysis as the minimum for the number of students in the school-level analysis: 5,701 students in cohort 1 and 6,461 students in cohort 2. |
||||