¹ See the product description at http://www.pearsonassessments.com/HAIWEB/Cultures/en-us/Productdetail.htm?Pid=E139A.
² The original CRCT scores shown in the report are by objective. Upon request from the WWC, the author calculated the mean overall score across all objectives, controlling for pretest, for each grade.
³ Georgia Department of Education. (n.d.). Criterion-referenced competency tests. Retrieved November 17, 2009 from http://www.doe.k12.ga.us/ci_testing.aspx?PageReq=CI_TESTING_CRCT.

ns = not statistically significant
na = not applicable
nr = not reported
ECLS-K = Early Childhood Longitudinal Survey–Kindergarten
SAT 9 = Stanford Achievement Test, Ninth Edition
CRCT = Georgia’s Criterion-Referenced Competency Test
Investigations = Investigations in Number, Data, and Space
SFAW = Scott Foresman–Addison Wesley Mathematics

¹ This appendix reports findings considered for the effectiveness rating and the average improvement indices for the mathematics achievement domain. Subgroup and subtest findings from the same studies are not included in these ratings but are reported in Appendices A4.1 and A4.2, respectively.
2 The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes.
3 Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group.
4 For an explanation of the effect size calculation, see the WWC Procedures and Standards Handbook, Appendix B.
5 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups.
6 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between –50 and +50, with positive numbers denoting favorable results for the intervention group.
7 The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. For the formulas the WWC used to calculate the statistical significance, see the WWC Procedures and Standards Handbook, Appendix C for clustering and the WWC Procedures and Standards Handbook, Appendix D for multiple comparisons. In the cases of Agodini et al. (2009) and Resendez and Manley (2005), no corrections for clustering or multiple comparisons were needed. In the case of Good, Bickel, and Howley (2006), a correction for clustering was needed, so the significance levels may differ from those reported in the original study.
8 The treatment group coefficient represents the sum of the unadjusted control group mean and the hierarchical linear modeling (HLM) coefficient for the difference between the two groups in the study.
9 The WWC-computed average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average improvement indices are calculated from the average effect sizes.
10 These figures represent difference-in-differences adjusted means not reported in the original study. They are based on results provided by the author(s) in response to a request by the WWC. The difference-in-differences adjustment subtracts baseline differences between the study groups from the post-intervention differences between the groups. The author query for additional information was required because the original study presented only analyses of the impact of the amount of treatment received, rather than intent-to-treat effects. The means for the Saxon and comparison groups differed by 0.07 standard deviations at baseline.
11 The original study reported only means for subtests. The value reported here is the mean across those subtests. For subtest results, see Appendix A4.2.
12 Student-level standard deviations were not available for this study. School-level standard deviations for the intervention group were 6.60 for grade 1, 6.39 for grade 2, 6.50 for grade 3, 8.51 for grade 4, and 6.94 for grade 5. School-level standard deviations for the comparison group were 6.80 for grade 1, 7.35 for grade 2, 7.15 for grade 3, 11.83 for grade 4, and 8.93 for grade 5. Because the student-level effect sizes and improvement indices could not be computed, the magnitude of the effect size was not considered for rating purposes. Note, however, that the average school-level effect size for the study is zero, and student-level effect sizes are typically smaller than school-level effect sizes. The statistical significance for this study is comparable to other studies and is included in the intervention rating. For further details, please see the WWC Procedures and Standards Handbook, Appendix B.

ns = not statistically significant
nr = not reported
ECLS-K = Early Childhood Longitudinal Study–Kindergarten
FRP = Free/reduced-price meal eligibility

¹ This appendix presents subgroup findings for measures that fall in the mathematics achievement domain. Total group scores were used for rating purposes and are presented in Appendix A3.
² The subgroup sample sizes were obtained through communication with the study authors.
³ The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes.
⁴ Subgroups were defined using school characteristics. Subgroups defined using baseline student achievement data are defined as students in schools with average math scores in the lowest, middle, and highest third of the study’s school-level distribution. Subgroups based on socioeconomic status are examined for students in schools with up to 40% of students eligible for free or reduced-price meals, compared to schools with more than 40% of students eligible for free or reduced-price meals.
⁵ The authors provided only the number of students, not the number of teachers or schools in each subgroup.
⁶ Positive effect sizes favor the intervention group; negative effect sizes favor the comparison group. For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B.
⁷ Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups.
⁸ The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between –50 and +50, with positive numbers denoting results favorable to the intervention group.
⁹ The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and Standards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D for multiple comparisons. In the case of Agodini et al. (2009), no corrections for clustering or multiple comparisons were needed.
¹⁰ The study provided effect sizes and statistical significance for subgroup outcomes produced though hierarchical linear modeling (HLM) that were calculated in accordance with WWC standards. Adjusted means were not available and are consequently omitted in this table. The table includes the effect sizes and statistical significance reported in the study, along with improvement index values calculated by the WWC based on the study-reported effect sizes.

ns = not statistically significant
na = not applicable
nr = not reported

¹ This appendix presents subscale findings for measures that fall in the mathematics achievement domain. Total scale scores were used for rating purposes and are presented in Appendix A3.
² The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes.
³ The intervention group and control group means are pretest adjusted and provided by the authors. They may differ from the means reported in the original study.
⁴ Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group.
⁵ For an explanation of the effect size calculation, see the WWC Procedures and Standards Handbook, Appendix B.
⁶ Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups.
⁷ The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between –50 and +50, with positive numbers denoting results favorable to the intervention group.
⁸ The level of statistical significance was reported by the study authors. No correction was required for clustering within classrooms or schools, or for multiple comparisons.
⁹ Student-level standard deviations and improvement indices were not available for this study. School-level standard deviations, which were requested by the WWC and provided by the first study author, ranged from 4.50 to 10.32 across grade levels and subtests in the intervention group and from 5.41 to 14.75 across grade levels and subtests in the comparison group. Because student-level standard deviations were not available, student-level effect sizes and improvement indices could not be computed. However, the statistical significance of the findings in Resendez and Manley (2005) is comparable to other studies and is reported in this appendix. For further details, see the WWC Procedures and Standards Handbook, Appendix B.

na = not applicable/not studied. Total number of students not reported in all of the relevant studies.

¹ A rating of ”medium to large“ requires at least two studies and two schools across studies in one domain and a total sample size across studies of at least 350 students or 14 classrooms. Otherwise, the rating is ”small.“ For more details on the extent of evidence categorization, see the WWC Procedures and Standards Handbook, Appendix G.