Appendix A1.1 Study characteristics: Dynarski & Wood, 1997—Stockton study (randomized controlled trial with control group crossover)
| Characteristic | Description |
|---|---|
| Study citation | Dynarski, M., & Wood, R. (1997). Helping high-risk youth: Results from the Alternative Schools Demonstration Program. Princeton, NJ: Mathematica Policy Research, Inc. |
| Participants | The Stockton High School Redirection study used a randomized controlled trial research design. Students were assigned to the two research groups using a 2: 1 random assignment ratio under which two students were assigned to the intervention group for every one student assigned to the control group . The original study sample of 924 students included two cohorts. Cohort 1 consisted of students who applied to the Model Alternative High School (the replication of High School Redirection in Stockton) prior to or during the 1991–92 school year and included 253 students in the intervention group and 121 students in the control group. Cohort 2 consisted of students who applied prior to or during the 1992–93 school year and included 363 students in the intervention group and 187 students in the control group. The Stockton school district launched a major restructuring effort in the fall of 1993 that merged all its alternative high schools with the Model Alternative High School. This substantially changed the program. It also placed many control group students who had been attending other alternative high schools in the district into the intervention school. Because this district reorganization had a greater effect on Cohort 2 than on Cohort 1, occurring just one year after program entry for the later cohort, the study's authors estimated the program's effects using data for Cohort 1 only. Because of this disruption to the study, the WWC rated this study as meeting evidence standards with reservations. Results summarized here are based on school records, which are available for all 253 intervention students and all 121 control group students in Cohort 1, as well as a follow-up survey administered three years after random assignment. Two hundred and two intervention-group students and 96 control-group students responded to the survey, for response rates of 80% and 79%, respectively. For the students in Cohort 1, researchers compared the baseline characteristics of the two research groups on 12 demographic, socio-economic, and school-related measures. A statistical test of the overall difference between the research groups on the full set of 12 baseline characteristics found that the groups were not significantly different.1 Stockton participants were, on average, just under 17 years old at the time they applied to the High School Redirection program. Almost half (45%) were Hispanic; the rest were divided between whites, African-Americans, and other racial and ethnic groups. Participants were evenly split between males and females. Over half (55%) had dropped out of school before applying to the alternative high school. |
| Setting | This study took place at the Model Alternative High School, an alternative high school in Stockton, California. |
| Intervention | The Model Alternative High School was a replication of High School Redirection and was part of the U. S. Department of Labor's Alternative Schools Demonstration Program (ASDP) evaluation. The Stockton school included most of the key features of the High School Redirection model specified by the ASDP evaluation: granting regular high school diplomas, taking students from throughout the district, offering the STAR remedial reading program to those with serious literacy problems, providing on-site child care, offering no extra-curricular activities, and operating with considerable autonomy from the local district. During the evaluation period, the program enrolled approximately 600 students and held both morning and afternoon sessions in order to accommodate more students. Accumulating credits toward graduation was the primary goal of many of the Model Alternative students. Therefore, the school's independent study option—under which students could complete assigned work on their own time and away from school—was quite popular (Rubenstein, 1995). Besides this independent study option, the school followed the district's core curriculum without modification. |
| Comparison | Control group students could attend other district high schools that did not implement the High School Redirection model. In some cases, these students attended otheralternative high schools for at-risk students operated by the district. Beginning with the 1992–93 school year, these other alternative education programs were located on the same campus as the Model Alternative High School. In 1993–94, these other programs were merged with Model Alternative. According to district records, in the first two years after random assignment (representing the 1991–92 and 1992–93 school years), 24% of control group students attended one of the alternative programs that merged with Model Alternative in the fall of 1993. |
| Primary outcomes and measurement | Five relevant outcomes from the Stockton High School Redirection study were used for rating purposes: number of days enrolled in a district high school during the first follow-up year, number of days enrolled in a district high school during the second follow-up year, dropped out at the end of the third follow-up year, cumulative credits earned by the end of the fourth follow-up year, and graduated or earned a GED certificate by the end of the third follow-up year. (See Appendices A2.1, A2.2, and A2.3 for a more detailed description of outcome measures.) |
| Teacher training | Model Alternative High School teachers were regular high school teachers employed by the Stockton Public Schools. No additional information about specific training they received was available. |
|
1 These Cohort 1 specific baseline characteristics were not included in the original report, but were provided to the WWC by the study's authors. |
|
Appendix A1.2 Study characteristics: Dynarski & Wood, 1997—Wichita study (randomized controlled trial)
| Characteristic | Description |
|---|---|
| Study citation | Dynarski, M., & Wood, R. (1997). Helping high-risk youth: Results from the Alternative Schools Demonstration Program. Princeton, NJ: Mathematica Policy Research. |
| Participants | The Wichita High School Redirection study used a randomized controlled trial research design. Students were assigned to the two research groups using a 2: 1 random assignment ratio under which two students were assigned to the intervention group for every one student assigned to the control group. The study sample of 358 students included two cohorts. Cohort 1 consisted of students who applied to Metro-Midtown Alternative High School (the replication of High School Redirection in Wichita) prior to or during the 1991–92 school year and included 162 students in the intervention group and 78 students in the control group. Cohort 2 consisted of students who applied prior to or during the 1992–93 school year and included 79 students in the intervention group and 39 students in the control group. Results summarized here are based on school records, which are available for all 241 intervention students and all 117 control group students in the two cohorts, as well as a follow-up survey administered two years after random assignment. Two hundred and two intervention-group students and 108 control-group students responded to the survey, for response rates of 84% and 88%, respectively. Researchers compared the baseline characteristics of the two research groups on 12 demographic, socio-economic, and school-related measures. A statistical test of the overall difference between the research groups on the full set of 12 baseline characteristics found that the groups were not significantly different. Wichita participants were, on average, 17 years old at the time they applied to the High School Redirection program. Almost half (45%) were white. About 4 in 10 were African-American and about 1 in 10 was Hispanic. Participants were evenly split between males and females. About half (53%) had dropped out of school before applying to the alternative high school. |
| Setting | The study was conducted at the Metro-Midtown Alternative High School, an alternative high school in Wichita, Kansas. |
| Intervention | The Metro-Midtown Alternative High School was a replication of High School Redirection and was part of the U. S. Department of Labor's Alternative Schools Demonstration Program (ASDP) evaluation. The Wichita school included most of the key features of the High School Redirection model specified by the ASDP evaluation: granting regular high school diplomas, taking students from throughout the district, offering the STAR remedial reading program to those with serious literacy problems, providing on-site child care, offering no extra-curricular activities, and operating with considerable autonomy from the local district. During the evaluation period, the program enrolled approximately 300 students and held both daytime and evening sessions. The school offered the standard district curriculum. Teachers set up individual programs of instruction for each student, and students were able to progress at their own pace (Rubenstein, 1995). |
| Comparison | Control group students could attend other district high schools that did not implement the High School Redirection model. In some cases, these students may have attended one of the two other alternative high schools operated by the district. Later in the evaluation, the district began admitting some control group students to Metro-Midtown Alternative High School if they applied again after having been originally turned away. During the first two years after random assignment, 13% of control group students spent some time enrolled in Metro-Midtown. |
| Primary outcomes and measurement | Five relevant outcomes from the Wichita High School Redirection study were used for rating purposes: number of days enrolled in a district high school during the first follow-up year, number of days enrolled in a district high school during the second follow-up year, dropped out by the end of the second follow-up year, cumulative credits earned by the end of the fourth follow-up year, and graduated or earned a GED certificate by the end of the second follow-up year. (See Appendices A2.1, A2.2, and A2.3 for a more detailed description of outcome measures.) |
| Teacher training | Metro-Midtown Alternative High School teachers were regular high school teachers employed by the Wichita Public Schools. No additional information about specific training they received was available. |
Appendix A1.3 Study characteristics: Dynarski & Wood, 1997—Cincinnati study (randomized controlled trial)
| Characteristic | Description |
|---|---|
| Study citation | Dynarski, M., & Wood, R. (1997). Helping high-risk youth: Results from the Alternative Schools Demonstration Program. Princeton, NJ: Mathematica Policy Research. |
| Participants | The Cincinnati High School Redirection study used a randomized controlled trial research design. Students were assigned to the two research groups using a 2: 1 random assignment ratio under which two students were assigned to the intervention group for every one student assigned to the control group. The study sample of 902 students included two cohorts. Cohort 1 consisted of students who applied to Clark Academy (the replication of High School Redirection in Cincinnati) prior to or during the 1993–94 school year and included 390 students in the intervention group and 185 students in the control group. Cohort 2 consisted of students who applied prior to or during the 1994–95 school year and included 222 students in the intervention group and 105 students in the control group. Results summarized here are based on school records, which are available for all 612 intervention students and all 290 control group students in the two cohorts. Researchers compared the baseline characteristics of the two research groups on 12 demographic, socio-economic, and school-related measures. A statistical test of the overall difference between the research groups on the full set of 12 baseline characteristics found that the groups were not significantly different. Cincinnati study participants were, on average, 17.6 years old at the time they applied to the High School Redirection program. Most (82%) were African-American. Just over half of participants (54%) were female. Over a third were teenage parents. About 6 in 10 had dropped out of school before applying to the alternative high school. |
| Setting | The study was conducted at the Peter H. Clark Academy, an alternative high school in Cincinnati, Ohio. |
| Intervention | The Peter H. Clark Academy was a replication of High School Redirection and was part of the U. S. Department of Labor's Alternative Schools Demonstration Program (ASDP) evaluation. The Cincinnati school included most of the key features of the High School Redirection model specified by the ASDP evaluation: granting regular high school diplomas, taking students from throughout the district, offering the STAR remedial reading program to those with serious literacy problems, offering no extra-curricular activities, and operating with considerable autonomy from the local district. The program had capacity for 250 students and operated out of a converted elementary school in a poor neighborhood of Cincinnati. Unlike the other High School Redirection replication sites summarized in this report, the program did not offer on-site child care. Instead, it offered off-site child care and van service between the school and the daycare facility. The school developed a special curriculum for ninth grade in which educational themes were integrated across the curriculum. The school also offered a modified English curriculum that allowed some students to earn graduation credits at an accelerated pace. Beyond these two modifications, the school offered the standard district curriculum (Rubenstein, 1995). |
| Comparison | Control group students could attend other district high schools that did not implement the High School Redirection model. At the time of the ASDP evaluation, the Cincinnati Public Schools offered few other alternative education programs for at-risk students besides the Peter H. Clark Academy. |
| Primary outcomes and measurement | Two relevant outcomes from the Cincinnati High School Redirection study were used for rating purposes: dropped out at the end of the second follow-up year and graduated or earned a GED certificate by the end of the second follow-up year. (See Appendices A2.1, A2.2, and A2.3 for a more detailed description of outcome measures.) |
| Teacher training | Clark Academy teachers were regular high school teachers employed by the Cincinnati Public Schools. No additional information about specific training they received was available. |
Appendix A2.1 Outcome measures for the staying in school domain
| Outcome measure | Description |
|---|---|
| Number of days enrolled in year 1 | This measure represents the number of days that students were enrolled in a district high school during the first follow-up year. These data were collected through school records for the Stockton and Wichita studies. This measure was not available in the Cincinnati study. |
| Number of days enrolled in year 2 | This measure represents the number of days that students were enrolled in a district high school during the second follow-up year. These data were collected through school records for the Stockton and Wichita studies. This measure was not available in the Cincinnati study. |
| Dropped out | This binary measure represents whether a student has dropped out of school. In the Stockton and Wichita studies, this outcome is based on follow-up survey data and is defined as not being enrolled in any school or GED program and not having earned a high school diploma or GED certificate. For the Stockton study, the outcome is measured at the end of year 3; for the Wichita study, it is measured at the end of year 2. In the Cincinnati study, this outcome is based on school records data, as well as GED testing data maintained by the Ohio Department of Education. It is measured at the end of year 2 and represents not being enrolled in a district high school and not having earned a district diploma or a GED certificate in the state. It does not include enrollment in a GED program. |
Appendix A2.2 Outcome measures for the progressing in school domain
| Outcome measure | Description |
|---|---|
| Total credits earned | This measure represents the cumulative number of credits that students earned by the end of the fourth follow-up year. For the Stockton and Wichita studies, these data were collected through school records. This measure was not available for the Cincinnati study. |
Appendix A2.3 Outcome measures for the completing school domain
| Outcome measure | Description |
|---|---|
| Earned a high school diploma or GED | This binary measure represents the percentage of students that earned a high school diploma or GED certificate at a particular point after random assignment. In the Stockton and Wichita studies, this outcome is based on follow-up survey data and is defined as having earned a diploma from any high school or a GED certificate in any state. For the Stockton study, the outcome is measured at the end of year 3; for the Wichita study, it is measured at the end of year 2. In the Cincinnati study, this outcome is measured at the end of year 2 and represents having graduated from a district high school or having earned a district diploma or a GED certificate in Ohio. The measure is based on school records data, as well as GED testing data that were provided by the Ohio Department of Education. |
Appendix A3.1 Summary of study findings included in the rating for the staying in school domain1
| Author's findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome (standard deviation2) | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (students) | High School Redirection group | Comparison group | Mean difference3 (High SchoolRedirection– comparison) | Effect size4 | Statistical significance5 (at α= 0.05) | Improvement index6 |
| Dynarski and Wood, 1997 (randomized controlled trial with control group crossover)7 —Stockton | ||||||||
| Number of days enrolled in year 1 | Cohort 1 | 374 | 110 (71) | 71 (78) | 39 | 0.53 | Statistically significant | +20 |
| Number of days enrolled in year 2 | Cohort 1 | 374 | 67 (78) | 50 (73) | 17 | 0.22 | Statistically significant | +9 |
| Dropped out at end of year 3 (%) | Cohort 1 | 298 | 43 | 53 | 10 | 0.24 | ns | +9 |
| Average8 for staying in school (Dynarski and Wood, 1997)—Stockton | 0.33 | Statistically significant | +13 | |||||
| Dynarski and Wood, 1997 (randomized controlled trial)7 —Wichita | ||||||||
| Number of days enrolled in year 1 | Cohorts 1 and 2 | 358 | 108 (64) | 94 (73) | 14 | 0.21 | ns | +8 |
| Number of days enrolled in year 2 | Cohorts 1 and 2 | 358 | 59 (77) | 60 (73) | -1 | -0.01 | ns | 0 |
| Dropped out at end of year 2 (%) | Cohorts 1 and 2 | 310 | 51 | 46 | -5 | -0.12 | ns | -5 |
| Average8 for staying in school (Dynarski and Wood, 1997)—Wichita | 0.03 | ns | +1 | |||||
| Dynarski and Wood, 1997 (randomized controlled trial)7 —Cincinnati | ||||||||
| Dropped out at end of year 2 (%) | Cohorts 1 and 2 | 902 | 80 | 83 | 3 | 0.12 | ns | +5 |
| Average8 for staying in school (Dynarski and Wood, 1997)—Cincinnati | 0.12 | ns | +5 | |||||
| Domain average8 for staying in school | 0.16 | na | +6 | |||||
|
ns = not statistically significant 2Standard deviations are reported for continuous outcomes only. The standard deviation across all students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. Standard deviations for number of days enrolled were not included in Dynarski and Wood (1997), but were reported to the WWC by the study's authors. 3 Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. For the dropout outcome, signs were reversed on the mean difference, effect size, and improvement index to demonstrate that the treatment group was favored when negative differences were reported. Means from Dynarski and Wood (1997) are estimated using regression models that control for baseline characteristics. 4 The effect sizes for dichotomous variables were computed using the Cox Index. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. 5 Statistical significance is the probability that the difference between the groups is a result of chance rather than a real difference between the groups. Although the study authors originally reported statistical significance using a p< 0.10 threshold, all analyses were redone by the WWC to test for statistical significance using a p< 0.05 threshold. 6 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting results favorable to the intervention group. 7 The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of the Dynarski and Wood (1997) Stockton, Wichita, and Cincinnati studies, no corrections for clustering were needed. 8 The WWC-computed average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average improvement indices are calculated from the average effect sizes. |
||||||||
Appendix A3.2 Summary of study findings included in the rating for the Progressing in school domain1
| Author's findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome (standard deviation2) | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (students) | High School Redirection group | Comparison group | Mean difference3 (High SchoolRedirection– comparison) | Effect size4 | Statistical significance5 (at α= 0.05) | Improvement index6 |
| Dynarski and Wood, 1997 (randomized controlled trial with control group crossover)7 —Stockton | ||||||||
| Total credits earned at end of year 4 | Cohort 1 | 374 | 10.5 (7.6) | 8.5 (7.4) | 2.0 | 0.26 | Statistically significant | +10 |
| Average8 for progressing in school (Dynarski and Wood, 1997)—Stockton | 0.26 | Statistically significant | +10 | |||||
| Dynarski and Wood, 1997 (randomized controlled trial)7 —Wichita | ||||||||
| Total credits earned at end of year 4 | Cohorts 1 and 2 | 358 | 7.6 (6.9) | 8.2 (6.8) | -0.6 | -0.09 | ns | -3 |
| Average8 for progressing in school (Dynarski and Wood, 1997)—Wichita | -0.09 | ns | -3 | |||||
| Domain average8 for progressing in school | 0.09 | na | +4 | |||||
|
ns = not statistically significant 2 The standard deviation across all students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. Standard deviations for total credits earned were not included in Dynarski and Wood (1997), but were reported to the WWC by the study's authors. 3 Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. Means from Dynarski and Wood (1997) are estimated using regression models that control for baseline characteristics. 4 For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. 5 Statistical significance is the probability that the difference between the groups is a result of chance rather than a real difference between the groups. Although the study authors originally reported statistical significance using a p< 0.10 threshold, all analyses were redone by the WWC to test for statistical significance using a p< 0.05 threshold. 6 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting results favorable to the intervention group. 7 The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of the Dynarski and Wood (1997) Stockton and Wichita studies, no corrections for clustering or multiple comparisons were needed. 8 The WWC-computed average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average improvement indices are calculated from the average effect sizes. |
||||||||
Appendix A3.3 Summary of study findings included in the rating for the completing school domain1
| Author's findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (students) | High School Redirection group | Comparison group | Mean difference2 (High SchoolRedirection– comparison) | Effect size3 | Statistical significance4 (at α= 0.05) | Improvement index5 |
| Dynarski and Wood, 1997 (randomized controlled trial with control group crossover)6 —Stockton | ||||||||
| Earned a high school diploma or GED at end of year 3 (%) | Cohort 1 | 298 | 40 | 32 | 8 | 0.21 | ns | +8 |
| Average7 for completing school (Dynarski and Wood, 1997)—Stockton | 0.21 | ns | +8 | |||||
| Dynarski and Wood, 1997 (randomized controlled trial)6 —Wichita | ||||||||
| Earned a high school diploma or GED at end of year 2 (%) | Cohorts 1 and 2 | 310 | 29 | 30 | -1 | -0.03 | ns | -1 |
| Average7 for completing school (Dynarski and Wood, 1997)—Wichita | -0.03 | ns | -1 | |||||
| Dynarski and Wood, 1997 (randomized controlled trial)6 —Cincinnati | ||||||||
| Earned a high school diploma or GED at end of year 2 (%) | Cohorts 1 and 2 | 902 | 7 | 6 | 1 | 0.10 | ns | +4 |
| Average7 for completing school (Dynarski and Wood, 1997)—Cincinnati | 0.10 | ns | +4 | |||||
| Domain average7 for completing school | 0.09 | na | +4 | |||||
|
ns = not statistically significant 2 Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. Means from Dynarski and Wood (1997) are estimated using regression models that control for baseline characteristics. 3 Effect sizes for dichotomous variables were computed using the Cox Index. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. 4 Statistical significance is the probability that the difference between the groups is a result of chance rather than a real difference between the groups. Although the study authors originally reported statistical significance using a p< 0.10 threshold, all analyses were redone by the WWC to test for statistical significance using a p< 0.05 threshold. 5 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting results favorable to the intervention group. 6 The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of the Dynarski and Wood (1997) Stockton, Wichita, and Cincinnati studies, no corrections for clustering or multiple comparisons were needed. 7 The WWC-computed average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average improvement indices are calculated from the average effect sizes. |
||||||||
Appendix A4 Summary of item-level findings for the completing school domain1
| Author's findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (students) | High School Redirection group | Comparison group | Mean difference2 (High School Redirection– comparison) | Effect size3 | Statistical significance4 (at α= 0.05) | Improvement index5 |
| Dynarski and Wood, 1997 (randomized controlled trial with control group crossover)6 —Stockton | ||||||||
| Earned a high school diploma at end of year 3 (%) | Cohort 1 | 298 | 20 | 15 | 5 | 0.21 | ns | +8 |
| Earned a GED certificate at end of year 3 (%) | Cohort 1 | 298 | 21 | 17 | 4 | 0.16 | ns | +6 |
| Dynarski and Wood, 1997 (randomized controlled trial)6 —Wichita | ||||||||
| Earned a high school diploma at end of year 2 (%) | Cohorts 1 and 2 | 310 | 9 | 14 | -5 | -0.30 | ns | -12 |
| Earned a GED certificate at end of year 2 (%) | Cohorts 1 and 2 | 310 | 20 | 17 | 3 | 0.12 | ns | +5 |
| Dynarski and Wood, 1997 (randomized controlled trial)6 —Cincinnati | ||||||||
| Earned a high school diploma at end of year 2 (%) | Cohorts 1 and 2 | 902 | 4 | 1 | 3 | 0.86 | Statistically significant | +30 |
| Earned a GED certificate at end of year 2 (%) | Cohorts 1 and 2 | 902 | 3 | 5 | -2 | -0.32 | ns | -13 |
|
ns = not statistically significant 1 This appendix presents item-level findings for measures in the completing school domain. These include the separate effects of the intervention on high school diploma receipt and on GED certificate receipt. The intervention's combined effect on high school diploma and GED receipt was used for determining the effectiveness rating and is presented in Appendix A3.3.2 Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. Means from Dynarski and Wood (1997) are estimated using regression models that control for baseline characteristics. 3 Effect sizes for dichotomous variables were computed using the Cox Index. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. 4 Statistical significance is the probability that the difference between the groups is a result of chance rather than a real difference between the groups. Although the study authors originally reported statistical significance using a p< 0.10 threshold, all analyses were redone by the WWC to test for statistical significance using a p< 0.05 threshold. 5 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting results favorable to the intervention group. 6 The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of the Dynarski and Wood (1997) Stockton, Wichita, and Cincinnati studies, no corrections for clustering or multiple comparisons were needed. |
||||||||
Appendix A5.1 High School Redirection rating for the staying in school domain
The WWC rates an intervention's effects in a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.1
For the outcome domain of staying in school, the WWC rated High School Redirection as having mixed effects. It did not meet the criteria for positive effects because only one of the three studies showed positive effects. It did not meet the criteria for potentially positive effects because more studies found indeterminate effects than found significant positive effects. The remaining ratings (no discernable effects, potentially negative effects, negative effects) were not considered because High School Redirection was assigned a higher applicable rating.
| Rating received |
|---|
|
Mixed effects: Evidence of inconsistent effects as demonstrated through EITHER of the following criteria.
|
| Other ratings considered |
|
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.
|
|
Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence.
|
|
1 For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. |
Appendix A5.2High School Redirection rating for the progressing in school domain
The WWC rates an intervention's effects in a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.1
For the outcome domain of progressing in school, the WWC rated High School Redirection as having potentially positive effects. It did not meet the criteria for having positive effects because only one study found statistically significant or substantively important positive effects in this domain. The remaining ratings (mixed effects, no discernable effects, potentially negative effects, negative effects) were not considered because High School Redirection was assigned a higher applicable rating.
| Rating received |
|---|
|
Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence.
|
| Other ratings considered |
|
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.
|
|
1 For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. |
Appendix A5.3 High School Redirection rating for the completing school domain
The WWC rates an intervention's effects in a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.1
For the outcome domain of completing school, the WWC rated High School Redirection as having no discernible effects. It did not meet the criteria for the other ratings (positive effects, potentially positive effects, mixed effects, potentially negative effects, negative effects) because no statistically significant or substantively important findings were found in this domain.
| Rating received |
|---|
|
No discernible effects: No affirmative evidence of effects.
|
| Other ratings considered |
|
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.
|
|
Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence.
|
|
Mixed effects: Evidence of inconsistent effects as demonstrated through EITHER of the following criteria.
|
|
Potentially negative effects: Evidence of a negative effect with no overriding contrary evidence.
|
|
Negative effects: Strong evidence of a negative effect with no overriding contrary evidence.
|
|
1 For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. |
Appendix A6 Extent of evidence by domain
| Sample size | ||||
|---|---|---|---|---|
| Outcome domain | Number of studies | Schools | Students | Extent of evidence1 |
| Staying in school | 3 | 3 | 1,634 | Medium to large |
| Progressing in school | 2 | 2 | 732 | Medium to large |
| Completing school | 3 | 3 | 1,510 | Medium to large |
|
na = not applicable/not studied 1 A rating of "medium to large" requires at least two studies and two schools across studies in one domain, and a total sample size across studies of at least 350 students or 14 classrooms. Otherwise, the rating is "small." |
||||