Appendix A1 Study characteristics: Kemple & Snipes, 2000 (randomized controlled trial) and Kemple, 2004 (randomized controlled trial)
| Characteristic | Description |
|---|---|
| Study citation |
Kemple, J. J., & Snipes, J. C. (2000). Career Academies: Impacts on students' engagement and performance in high school. New York: MDRC (Manpower Demonstration Research Corporation).
Additional Sources: Kemple: J. J. (2004). Career Academies: Impacts on labor market outcomes and educational attainment. New York: MDRC (Manpower Demonstration Research Corporation). |
| Participants | This review focuses on the subgroup of 474 youth in the study sample who were considered most at risk of dropping out prior to the start of the intervention. These youth represent 27% of the total study sample of 1,764.1 Among these high-risk youth, 79% were 15 years old or younger, a majority were female (57%), and many were Hispanic (52%) or African-American (38%), lived in a single-parent household (50%), and received welfare or Food Stamps (32%). In the year prior to random assignment, 33% of the high-risk youth were absent for at least 15% of the school year. In the year of random assignment, 62% of the high-risk youth earned a grade point average of 2.0 or lower and 43% were overage for their grade level. |
| Setting | The nine schools in the evaluation were in eight urban areas in six states: Pittsburgh, Pennsylvania; Baltimore, Maryland; Washington, DC; Miami Beach, Florida; Socorro, Texas; Santa Ana, California; Watsonville, California; and San Jose, California. |
| Intervention2 |
The intervention group was randomly assigned to the Career Academy to which they applied, and 86% of the high-risk youth randomly assigned to the Academy group enrolled in an Academy. The intervention group could attend the Academy until graduation. Career Academies in the evaluation had been operating several years before the study began. They had three primary components: a school-within-school organization with a career theme, academic plus vocational curricula related to the career theme, and employer partnerships.
|
| Comparison | Youth not randomly offered admission to a Career Academy constituted the comparison group. Most comparison group youth enrolled in a general education program in the host high school. Some enrolled in citywide magnet programs or specialty schools. About 3% enrolled in a Career Academy that was in the evaluation, despite being in the comparison group. |
| Primary outcomes and measurement | Staying in school domain (one outcome from a student survey, district records, and school enrollment status reports): the percentage of students who dropped out of high school before the end of their projected 12th-grade year. Progressing in school domain (two outcomes from school records data): total course credits accumulated between 9th grade and the end of the projected 12th-grade year, and the percentage of students whose accumulated credits met their school's graduation requirements. Completing school domain (one outcome from a student survey administered four years after a student's projected 12th-grade year): earned a high school diploma or GED certificate. (See Appendices A2.1–A2.3 for more detailed descriptions of outcome measures.) |
| Teacher characteristics and training | Career Academy teachers came from a variety of academic and vocational disciplines but generally were similar to other teachers in host high schools. Some of the professional development opportunities offered to Academy teachers included learning how best to support students in a small learning environment and learning strategies for coordinating career development and employer-related activities. The smaller school-within-school structure lent itself to opportunities for shared planning time among Academy teachers. |
|
1 Researchers used student background characteristics (including sibling dropped out, overage for grade, transferred schools two or more times, and attendance, GPA, and credits earned in year of random assignment) to develop a model to predict whether students in the comparison group dropped out of school, and then applied the estimated model to predict which intervention-group students were most likely to drop out. |
|
Appendix A2.1 Outcome measure for the staying in school domain
| Outcome measure | Description |
|---|---|
| Dropped out of high school | This binary measure counted a student as a dropout at the end of their scheduled 12th-grade year if they were not listed as enrolled on any one of three data sources (student survey, district records, and school enrollment status reports completed by the host high schools) and if one of the following conditions were met: student reported being a dropout on the survey or school records indicated student had dropped out with no indication of being enrolled elsewhere (as cited in Kemple & Snipes, 2000 and through further author communication). |
Appendix A2.2 Outcome measure for the progressing in school domain
| Outcome measure | Description |
|---|---|
| Total course credits earned | This continuous measure taken from school records data includes all course credits students earned from ninth grade through the end of their projected 12th-grade year (until just before they would have graduated from high school). In seven of the study sites, the Academies began in the 10th grade and ninth-grade course credits were earned prior to the intervention. The remaining two Academies began in the ninth grade and the course credits were earned during four years of exposure to the intervention (as cited in Kemple & Snipes, 2000). |
| Credits earned met graduation requirements | This binary measure taken from school records data indicates whether the credits earned from 9th grade through the end of their projected 12th-grade year (until just before they would have graduated from high school) met the requirements for graduation of each study school (as cited in Kemple & Snipes, 2000). |
Appendix A2.3 Outcome measure for the completing school domain
| Outcome measure | Description |
|---|---|
| Earned a diploma or GED certificate | This binary outcome was measured by the Career Academies Four-Year Post-High School Follow-Up Survey administered four years after a student's projected 12th-grade year. Students responded whether they had earned a high school diploma (on-time or late) or earned a GED certificate (as cited in Kemple, 2004). |
Appendix A3.1 Summary of study findings included in the rating for the staying in school domain1
| Author's findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome (standard deviation2) | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (students) | Career Academies group | Comparison group | Mean difference3 Career Academies–comparison | Effect size4 | Statistical significance5 (at α= 0.05) | Improvement index6 |
| Kemple & Snipes, 2000 (randomized controlled trial)7 | ||||||||
| Dropped out of high school (%) | High-risk youth | 345 | 21.3 (41.1) | 32.2 (46.9) | 10.9 | 0.34 | Statistically significant | +13 |
| Domain average8 for staying in school | 0.34 | Statistically significant | +13 | |||||
| 1 This appendix reports findings for the high-risk youth that were considered for the effectiveness rating and the average improvement indices. Findings from the low- and medium-risk subgroups are not included in these ratings, but are reported in Appendix A4.1. 2 The standard deviation for students in each group shows how dispersed participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. The standard deviation was derived by the WWC from a dichotomous variable for this outcome. 3 For this outcome, the mean difference was calculated so that a positive effect was found when fewer intervention youth than comparison youth dropped out of school (comparison group mean minus the intervention group mean). 4 Effect sizes for dichotomous variables are computed using the Cox index. For further explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. 5 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 6 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results. 7 The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Kemple and Snipes (2000), no corrections for clustering or multiple comparisons within this outcome domain were made. 8 This row provides the study average, which in this case is also the domain average. The WWC-computed domain average effect size is a simple average rounded to two decimal places. The domain improvement index is calculated from the average effect size. |
||||||||
Appendix A3.2 Summary of study findings included in the rating for the progressing in school domain1
| Author's findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome (standard deviation2) | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size3 (students) | Career Academies group | Comparison group | Mean difference Career Academies–comparison | Effect size4 | Statistical significance5 (at α= 0.05) | Improvement index6 |
| Kemple & Snipes, 2000 (randomized controlled trial)7 | ||||||||
| Total course credits earned | High-risk youth | 316 | 19.3 (6.9) | 17.3 (6.9) | 2.0 | 0.29 | Statistically significant | +11 |
| Credits earned met graduation requirements (%) | High-risk youth | 316 | 39.9 (49.1) | 26.2 (44.1) | 13.7 | 0.38 | Statistically significant | +15 |
| Domain average8 for progressing in school | 0.33 | Statistically significant | +13 | |||||
| 1 This appendix reports findings for the high-risk youth that were considered for the effectiveness rating and the average improvement indices. Findings from the low- and medium-risk subgroups are not included in these ratings, but are reported in Appendix A4.2. 2 The standard deviation for students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. For the total course credits earned, the standard deviation was provided by the author and is not reported in Kemple and Snipes (2000). For the credits meet graduation requirements outcome, the standard deviation was derived by the WWC from a dichotomous variable. 3 Sample size provided by the study author and differs slightly from those reported in Kemple and Snipes (2000). 4 Effect sizes for dichotomous variables are computed using the Cox index. For an explanation of the effect size calculations, see Technical Details of WWC-Conducted Computations. 5 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 6 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results. 7 The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Kemple and Snipes (2000), a correction for multiple comparisons was made, but the significance levels did not differ from those reported in the original study. 8 This row provides the study average, which in this case is also the domain average. The WWC-computed domain average effect size is a simple average rounded to two decimal places. The domain improvement index is calculated from the average effect size. |
||||||||
Appendix A3.3 Summary of study findings included in the rating for the completing school domain1
| Author's findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome (standard deviation2) | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (students) | Career Academies group | Comparison group | Mean difference Career Academies–comparison | Effect size3 | Statistical significance4 (at α= 0.05) | Improvement index5 |
| Kemple, 2004 (randomized controlled trial)6 | ||||||||
| Earned a diploma or GED certificate (%) | High-risk youth | 360 | 82.7 (37.9) | 83.2 (37.5) | -0.5 | -0.02 | ns | -0.1 |
| Domain average7 for completing school | -0.02 | ns | -0.1 | |||||
|
ns = not statistically significant 1 This appendix reports findings for the high-risk youth that were considered for the effectiveness rating and the average improvement indices. Findings from the low- and medium-risk subgroups are not included in these ratings, but are reported in Appendix A4.3.2 The standard deviation for students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. The standard deviation was derived by the WWC from a dichotomous variable for this outcome. 3 Effect sizes for dichotomous variables are computed using the Cox index. For an explanation of the effect size calculations, see Technical Details of WWC-Conducted Computations. 4 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 5 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results. 6 The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Kemple (2004), no corrections for clustering or multiple comparisons within this outcome domain were made. 7 This row provides the study average, which in this case is also the domain average. The WWC-computed domain average effect size is a simple average rounded to two decimal places. The domain improvement index is calculated from the average effect size. |
||||||||
Appendix A4.1 Summary of low-risk and medium-risk subgroup findings for the staying in school domain1
| Author's findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome (standard deviation2) | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (students) | Career Academies group | Comparison group | Mean difference3 Career Academies–comparison | Effect size4 | Statistical significance5 (at α= 0.05) | Improvement index6 |
| Kemple & Snipes, 2000 (randomized controlled trial)7 | ||||||||
| Dropped out of high school (%) | Low-risk youth | 385 | 1.9 (13.7) | 2.9 (16.8) | 1.0 | 0.26 | ns | +10 |
| Dropped out of high school (%) | Medium-risk youth | 724 | 9.0 (28.7) | 8.0 (27.2) | -1.0 | -0.08 | ns | -3 |
|
ns = not statistically significant 1 This appendix presents findings for the low-risk and medium-risk youth for measures that fall in the staying in school domain. Findings for youth who were at high risk for dropping out were used for rating purposes and are presented in Appendix A3.1.2 The standard deviation for students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. The standard deviation was derived by the WWC from a dichotomous variable for this outcome. 3 For this outcome, the mean difference is calculated so that a positive effect is found when fewer intervention youth than comparison youth drop out of school (comparison group mean minus the intervention group mean). 4 Effect sizes for dichotomous variables are computed using the Cox index. For an explanation of the effect size calculations, see Technical Details of WWC-Conducted Computations. 5 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 6 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results. 7 The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple comparisons were not done for findings not included in the overall intervention rating). For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Kemple and Snipes (2000), no correction for clustering within this outcome domain was made. |
||||||||
Appendix A4.2 Summary of low-risk and medium-risk subgroup findings for the progressing in school domain1
| Author's findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome (standard deviation2) | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (students) | Career Academies group | Comparison group | Mean difference Career Academies–comparison | Effect size3 | Statistical significance4 (at α= 0.05) | Improvement index5 |
| Kemple & Snipes, 2000 (randomized controlled trial)6 | ||||||||
| Total course credits earned | Low-risk youth | 376 | 24.4 (4.0) | 23.6 (3.3) | 0.80 | 0.22 | ns | +9 |
| Total course credits earned | Medium-risk youth | 687 | 22.6 (5.1) | 22.9 (4.9) | -0.30 | -0.06 | ns | -2 |
| Credits earned met graduation requirements (%) | Low-risk youth | 376 | 85.7 (35.1) | 74.6 (43.7) | 11.1 | 0.43 | Statistically significant | +17 |
| Credits earned met graduation requirements (%) | Medium-risk youth | 687 | 64.8 (47.8) | 65.2 (47.7) | -0.4 | -0.01 | ns | -0.4 |
|
ns = not statistically significant 1 This appendix presents findings for the low-risk and medium-risk youth for measures that fall in the progressing in school domain. Findings for youth who were at high risk for dropping out were used for rating purposes and are presented in Appendix A3.2.2 The standard deviation for students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. For the total course credits outcome, the standard deviations were provided by the study author and are not reported in Kemple and Snipes (2000). For the credits meet graduation requirements outcome, the standard deviation was derived by the WWC from a dichotomous variable. 3 Effect sizes for dichotomous variables are computed using the Cox index. For an explanation of the effect size calculations, see Technical Details of WWC-Conducted Computations. 4 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 5 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results. 6 The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple comparisons were not done for findings not included in the overall intervention rating). For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Kemple and Snipes (2000), no correction for clustering within this outcome domain was made. |
||||||||
Appendix A4.3 Summary of low-risk and medium-risk subgroup findings for the completing school domain1
| Author's findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome (standard deviation2) | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (students) | Career Academies group | Comparison group | Mean difference Career Academies–comparison | Effect size3 | Statistical significance4 (at α= 0.05) | Improvement index5 |
| Kemple, 2004 (randomized controlled trial)6 | ||||||||
| Earned a diploma or GED certificate (%) | Low-risk youth | 376 | 100.0 (0.0) | 99.4 (7.7) | 0.6 | na | ns | na |
| Earned a diploma or GED certificate (%) | Medium-risk youth | 722 | 92.9 (25.7) | 92.1 (27.0) | 0.8 | 0.07 | ns | +3 |
|
na = not applicable 2 The standard deviation for students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. The standard deviation was derived by the WWC from a dichotomous variable for this measure. 3 Effect sizes for dichotomous variables are computed using the Cox index. For an explanation of the effect size calculations, see Technical Details of WWC-Conducted Computations. An effect size cannot be calculated when there is no variance in outcomes among the groups. 4 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 5 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results. The improvement index cannot be estimated when an effect size cannot be calculated. 6 The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple comparisons were not done for findings not included in the overall intervention rating). For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Kemple (2004), no correction for clustering within this outcome domain was made. |
||||||||
Appendix A5.1 Career Academiesrating for the staying in school domain1
The WWC rates an intervention's effects for a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.2
For the outcome domain of staying in school, the WWC rated Career Academiesas having potentially positive effects. It did not meet the criteria for positive effects because it had only one study that met WWC standards. The remaining ratings (mixed effects, no discernible effects, potentially negative effects, and negative effects) were not considered because Career Academieswas assigned a higher applicable rating.
| Rating received |
|---|
|
Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence.
|
| Other ratings considered |
|
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.
|
|
1 This intervention rating was calculated based on 474 youth in the study sample who were most at risk of dropping out of high school. Researchers used student characteristics at baseline (including whether students had a sibling who dropped out, was overage for grade, had transferred schools two or more times, and their attendance, GPA, and credits earned in the year of random assignment) to develop a model to predict whether students in the comparison group dropped out of school and then applied theses predictions to the intervention group students. The full study sample included 1,764 youth. |
Appendix A5.2 Career Academiesrating for the progressing in school domain1
The WWC rates an intervention's effects for a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.2
For the outcome domain of progressing in school, the WWC rated Career Academiesas having potentially positive effects. It did not meet the criteria for positive effects because it had only one study that met WWC standards. The remaining ratings (mixed effects, no discernible effects, potentially negative effects, and negative effects) were not considered, because Career Academieswas assigned a higher applicable rating.
| Rating received |
|---|
|
Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence.
|
| Other ratings considered |
|
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.
|
|
1 This intervention rating was calculated based on 474 youth in the study sample who were most at risk of dropping out of high school. Researchers used student characteristics at baseline (including whether students had a sibling who dropped out, was overage for grade, had transferred schools two or more times, and their attendance, GPA, and credits earned in the year of random assignment) to develop a model to predict whether students in the comparison group dropped out of school and then applied theses predictions to the intervention group students. The full study sample included 1,764 youth. |
Appendix A5.3 Career Academies rating for the completing school domain1
The WWC rates an intervention's effects for a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.2
For the outcome domain of completing school, the WWC rated Career Academiesas having no discernible effects. It did not meet the criteria for other ratings (positive effects, potentially positive effects, mixed effects, potentially negative effects, and negative effects) because the one study that met WWC standards did not show statistically significant or substantively important effects.
| Rating received |
|---|
|
No discernible effects: No affirmative evidence of effects.
|
| Other ratings considered |
|
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.
|
|
Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence.
|
|
Mixed effects: Evidence of inconsistent effects as demonstrated through either of the following criteria.
|
|
Potentially negative effects: Evidence of a negative effect with no overriding contrary evidence.
|
|
Negative effects: Strong evidence of a negative effect with no overriding contrary evidence.
|
|
1 This intervention rating was calculated based on 474 youth in the study sample who were most at risk of dropping out of high school. Researchers used student characteristics at baseline (including whether students had a sibling who dropped out, was overage for grade, had transferred schools two or more times, and their attendance, GPA, and credits earned in the year of random assignment) to develop a model to predict whether students in the comparison group dropped out of school and then applied these predictions to the intervention group students. The full study sample included 1,764 youth. |
|Institute of Education Sciences
