¹The study authors excluded from the analysis two schools that did not randomly assign classrooms to a curriculum. One school did not have sufficient computer resources to implement Cognitive Tutor ^® Algebra I. Due to a scheduling error, teachers at the other school taught either Cognitive Tutor ^® Algebra I or the traditional curriculum but not both. Only the three schools that implemented the within-teacher random assignment design were analyzed by the study authors and included in this report.
² Eleven students whose records indicated a conflict between the curriculum and class assignment were excluded by the study authors due to uncertainty about their classroom experience. The school registrar reported that these students were assigned to the control group that received the traditional curriculum but were actually enrolled in a Cognitive Tutor ^® Algebra I classroom.

ns = not statistically significant

¹ This appendix presents additional findings for measures that fall in the math achievement domain. ETS scale scores were used for rating purposes and are presented in Appendix A3. The protocol for the middle school math area states that subjective measures, such as student grades assigned by teachers, do not qualify as relevant outcome measures.
² The standard deviation across all students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes.
³ The Cognitive Tutor ^® values are the unadjusted control scores group means plus the program coefficients from the hierarchical linear modeling (HLM) analysis. The standard deviations were obtained from the study authors.
⁴ The control group means are unadjusted. The means and standard deviations were obtained from the study authors.
⁵ Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group.
⁶ For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B.
⁷ Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups.
⁸ The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between –50 and +50, with positive numbers denoting results favorable to the intervention group.
⁹ The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and Standards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D for multiple comparisons. In the case of Ritter, Kulikowich, Lei, McGuire, & Morgan (2007), a correction for multiple comparisons was needed, so the significance levels may differ from those reported in the original study; no correction for clustering was needed because the authors accounted for clustering in their HLM analysis.

¹For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of potentially positive or potentially negative effects. For a complete description, see the WWC Procedures and Standards Handbook, Appendix E.

¹A rating of “medium to large” requires at least two studies and two schools across studies in one domain and a total sample size across studies of at least 350 students or 14 classrooms. Otherwise, the rating is “small.” For more details on the extent of evidence categorization, see the WWC Procedures and Standards Handbook, Appendix G.