Appendix A1 Study characteristics: Larson & Rumberger, 1995 (randomized controlled trial)
| Characteristic | Description |
|---|---|
| Study citation |
Larson, K. A., & Rumberger, R. W. (1995). ALAS: Achievement for Latinos through Academic Success. In H. Thornton (Ed.), Staying in school. A technical report of three dropout prevention projects for junior high school students with learning and emotional disabilities. Minneapolis, MN: University of Minnesota, Institute on Community Integration. Additional analysis Gándara, P., Larson, K. A., Mehan, H., & Rumberger, R. W. (1998). Capturing Latino Students in the Academic Pipeline. Berkeley, CA: Chicano/Latino Policy Project. |
| Participants |
The study focuses on a group of 94 high-risk students who entered seventh grade in 1990. Students were identified as high risk if their sixth-grade teacher rated them below the classroom average on a rating scale. Almost all the high-risk students who participated in the study were Latino (96%); most were males (65%); and almost all participated in the free or reduced-price lunch program (91%). About 23% were limited English proficient (LEP), about 33% were fluent English proficient, and the rest were English only students. On average, students were 12 years 7 months old when they entered the seventh grade. Students who spoke no English were excluded because the intervention was not designed to accommodate them. The program also included a sample of students with learning disabilities or who were classified as emotionally disturbed. The WWC does not report on this sample because that analysis did not meet WWC standards.
Additional analysis This analysis focuses on a subsample of 81 out of 94 students who had entered seventh grade in 1990 and remained in the target school (treatment group) or transferred to a junior high school in the same district (control group). |
| Setting | The study was conducted in a large junior high school in the Los Angeles Unified School District. |
| Intervention |
From the pool of 94 high-risk seventh graders, 46 students were randomly assigned to the intervention group. Treatment students received the ALAS intervention during the three years of junior high school (seventh through ninth grade) or until they left the junior high school. Each student was assigned a counselor who monitored the student continuously, worked as case manager, and ensured that all components of the intervention were provided. ALAS students received 10 weeks of problem-solving skills instruction and two years of follow-up problem-solving prompting and counseling. Student period-by-period attendance was monitored, and they were required to make up missed time. Parents were contacted about student truancy or extended absence. ALAS provided weekly and, if needed, daily feedback reports to students and parents regarding classroom comportment and missed assignments. Parents were trained in problem solving and participation in school. ALAS staff helped to directly facilitate youth and parents' use of such community services as mental health services and social services.
Additional analysis The treatment group includes only students who stayed in the ALAS junior high during all three years (36 students). |
| Comparison |
Forty-eight students were randomly assigned to the comparison group. They received the regular school program offered by the target school.
Additional analysis The comparison group for this study (45 students) includes students who were randomly assigned to be control students at the beginning of seventh grade and either did not transfer from the school or transferred to a school within the district. |
| Primary outcomes and measurement |
Two outcomes relevant for the WWC review were examined: the percentage of students enrolled at the end of the school year (staying in school domain) as measured at two points, grades 9 and 11, and the percentage of students on track to graduate from high school on time conditional on being enrolled in the district (progressing in school domain) as measured at two points, grades 9 and 11. (See Appendices A2.1 and A2.2 for more detailed descriptions of outcome measures.)
Additional analysis Three outcomes relevant for the WWC review were examined for this subgroup: the percentage of students enrolled at the end of the school year (staying in school domain) as measured at two points, grades 9 and 10; the percentage of students on track to graduate from high school on time (progressing in school domain) as measured at two points, grades 9 and 10; and the percentage of students who graduated from high school on time, at the end of grade 12 (completing school domain). (See Appendices A2.1–2.3 for more detailed descriptions of outcome measures.) |
| Teacher training | ALAS was delivered by a supervisor, counselors, and clerical staff housed full-time on the school campus. The supervisor, who was an experienced teacher, counselor, or social worker, provided on-going training to ALAS counselors and worked to coordinate services among the school, the family, and the community. ALAS staff and teachers were trained to deliver the social problem-solving skills curriculum. The supervisor may or may not have received training depending on prior experience. |
Appendix A2.1 Outcome measures in the staying in school domain
| Outcome measure | Description |
|---|---|
| Enrolled at end of ninth grade | The percentage of students who were enrolled at the end of ninth grade (spring 1993). A student was considered enrolled if he or she was enrolled in a district school no later than 20 days before the end of the semester; transferred to another school out of district or out of state as confirmed by a request for student records from the receiving district; or was institutionalized in a government or private mental health facility (as cited in Larson & Rumberger, 1995). |
| Enrolled at the end of 11th grade | The percentage of students who were enrolled at the end of 11th grade (spring 1995). A student was considered enrolled if he or she was enrolled in a district school no later than 20 days before the end of the semester; transferred to another school out of district or out of state as confirmed by a request for student records from the receiving district; or was institutionalized in a government or private mental health facility (as cited in Larson & Rumberger, 1995). |
| Enrolled at the end of ninth grade | The percentage of students who were enrolled at the end of ninth grade (spring 1993). A student was considered enrolled if he or she was enrolled in a district school no later than 20 days before the end of the semester (as cited in Gándara et al., 1998). |
| Enrolled at the end of 10th grade | The percentage of students who were enrolled at the end of 10th grade (spring 1994). A student was considered enrolled if he or she was enrolled in a district school no later than 20 days before the end of the semester (Gándara et al., 1998). |
Appendix A2.2 Outcome measures in the progressing in school domain
| Outcome measure | Description |
|---|---|
| On track to graduate on time at the end of ninth grade | The percentage of students who were on track to graduate on time. This measure was derived from the total number of credits earned by the student at the end of ninth grade summer inclusive. Students were on track to graduate on time at ninth grade if they had completed at least one quarter of their high school graduation requirements (220 credits) by the end of ninth grade (summer 1993) (as cited in Larson & Rumberger, 1995). |
| On track to graduate on time at the end of 11th grade | The percentage of students who were on track to graduate on time. This measure was derived from the total number of credits earned by the student at the end of 11th grade summer inclusive. Students were on track to graduate on time at 11th grade if they had completed at least three quarters of their high school graduation requirements (220 credits) by the end of 11th grade (summer 1995) (as cited in Larson & Rumberger, 1995). |
| On track to graduate on time at the end of ninth grade | The percentage of students who were on track to graduate on time. This measure was derived from the total number of credits earned by the student at the end of ninth grade summer inclusive. Students were on track to graduate on time at ninth grade if they had completed at least one quarter of their high school graduation requirements (220 credits) by the end of ninth grade (summer 1993) (as cited in Gándara et al., 1998). |
| On track to graduate on time at the end of 10th grade | The percentage of students who were on track to graduate on time. This measure was derived from the total number of credits earned by the student at the end of 10th grade summer inclusive. Students were on track to graduate on time at 10th grade if they had completed at least one half of their high school graduation requirements (220 credits) by the end of 10th grade (summer 1994) (as cited in Gándara et al., 1998). |
Appendix A2.3 Outcome measure in the completing school domain
| Outcome measure | Description |
|---|---|
| Completed high school on time | The percentage of students who completed high school on time by the end of 12th grade (summer 1996) (as cited in Gándara et al., 1998). |
Appendix A3.1 Summary of study findings included in the rating for the staying in school domain1
| Author's findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome2 (standard deviation3) | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (schools/students) | ALAS group | Comparison group | Mean difference4 (ALAS–comparison) | Effect size5 | Statistical significance6 (at α= 0.05) | Improvement index7 |
| Larson & Rumberger, 1995 (randomized controlled trial)8 | ||||||||
| Enrolled at end of ninth grade (%) | Full sample | 94 | 98 (14) | 83 (38) | 15 | 1.39 | Statistically significant | 42 |
| Domain average9 for staying in school | 1.39 | Statistically significant | 42 | |||||
| 1 This appendix reports findings considered for the effectiveness rating and the improvement index. Subgroup and follow-up findings from the same study are not included in these ratings, but are reported in Appendix A4.1. 2 The outcomes are binary, so means represent percentages. 3 The standard deviation across all students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. Standard deviations were derived from binary outcomes in this study. 4 Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 5 Effect sizes were calculated using the Cox index for binary outcomes. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. 6 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 7 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results. 8 The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of ALAS, no corrections were necessary. 9 This row provides the study average, which in this case is also the domain average. The WWC-computed domain average effect size is a simple average rounded to two decimal places. The domain improvement index is calculated from the average effect size. |
||||||||
Appendix A3.2 Summary of study findings included in the rating for the progressing in school domain1
| Author's findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome2 (standard deviation3) | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (schools/students) | ALAS group | Comparison group | Mean difference4 (ALAS–comparison) | Effect size5 | Statistical significance6 (at α= 0.05) | Improvement index7 |
| Larson & Rumberger, 1995 (randomized controlled trial)8 | ||||||||
| On track to graduate on time at the end of ninth grade (%) | Conditional on being in a district school | 81 | 72 (45) | 53 (51) | 19 | 0.49 | Statistically significant | 19 |
| Domain average9 for progressing in school | 0.49 | Statistically significant | 19 | |||||
| 1 This appendix reports findings considered for the effectiveness rating and the improvement index. Subgroup findings from the same study are not included in these ratings, but are reported in Appendix A4.2. 2 The outcomes are binary, so means represent percentages. 3 The standard deviation across all students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. Standard deviations were derived from binary outcomes in this study. 4 Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 5 Effect sizes were calculated using the Cox index for binary outcomes. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. 6 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 7 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results. 8 The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of ALAS, no corrections were necessary. 9 This row provides the study average, which in this case is also the domain average. The WWC-computed domain average effect size is a simple average rounded to two decimal places. The domain improvement index is calculated from the average effect size. |
||||||||
Appendix A4.1 Summary of follow-up and subgroup findings for the staying in school domain1
| Author's findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome2 (standard deviation3) | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (schools/students) | ALAS group | Comparison group | Mean difference4 (ALAS–comparison) | Effect size5 | Statistical significance6 (at α= 0.05) | Improvement index7 |
| Larson & Rumberger, 1995 (randomized controlled trial)8 | ||||||||
| Enrolled at end of 11th grade (%) | Full sample | 89 | 75 (44) | 67 (48) | 8 | 0.24 | ns | 10 |
| Gandara et al., 1998 (randomized controlled trial with non-randomized data collection)8 | ||||||||
| Enrolled at end of ninth grade (%) | Subsample | 81 | 97 (17) | 82 (39) | 15 | 1.18 | Statistically significant | 38 |
| Enrolled at end of 10th grade (%) | Subsample | 81 | 86 (35) | 69 (47) | 17 | 0.61 | ns | 23 |
|
ns = not statistically significant 1 This appendix presents subgroup and follow-up findings for outcomes related to staying in school. Outcomes related to the full sample are used for rating purposes and are presented in Appendix A3.1.2 The outcomes are binary, so means represent percentages. 3 The standard deviation across all students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. Standard deviations were derived from binary outcomes in this study. 4 Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 5 Effect sizes were calculated using the Cox index for binary outcomes. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. 6 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 7 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results. 8 The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple comparisons were not done for findings not included in the overall intervention rating). For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of ALAS, no corrections were necessary. |
||||||||
Appendix A4.2 Summary of follow-up and subgroup findings for the progressing in school domain1
| Author's findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome2 (standard deviation3) | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (schools/students) | ALAS group | Comparison group | Mean difference4 (ALAS–comparison ) | Effect size5 | Statistical significance6 (at α= 0.05) | Improvement index7 |
| Larson & Rumberger, 1995 (randomized controlled trial)8 | ||||||||
| On track to graduate on time at the end of 11th grade (%) | Conditional on being in a district school | 60 | 33 (48) | 26 (45) | 7 | 0.21 | ns | 8 |
| Gandara et al., 1998 (randomized controlled trial with non-randomized data collection)8 | ||||||||
| On track to graduate on time at the end of ninth grade (%) | Subsample | 81 | 75 (44) | 44 (50) | 31 | 0.80 | Statistically significant | 29 |
| On track to graduate on time at the end of 10th grade (%) | Subsample | 81 | 44 (50) | 22 (42) | 22 | 0.61 | Statistically significant | 23 |
|
ns = not statistically significant 1 This appendix presents subgroup and follow-up findings for outcomes related to progressing in school. Outcomes related to the full sample are used for rating purposes and are presented in Appendix A3.2.2 The outcomes are binary, so means represent percentages. 3 The standard deviation across all students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. Standard deviations were derived from binary outcomes in this study. 4 Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 5 Effect sizes were calculated using the Cox index for binary outcomes. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. 6 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 7 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results. 8 The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple comparisons were not done for findings not included in the overall intervention rating). For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of ALAS, no corrections were necessary. |
||||||||
Appendix A4.3 Summary of subgroup findings for the completing school domain1
| Author's findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome2 (standard deviation3) | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (schools/students) | ALAS group | Comparison group | Mean difference4 (ALAS–comparison ) | Effect size5 | Statistical significance6 (at α= 0.05) | Improvement index7 |
| Gándara et al., 1998 (randomized controlled trial with non-randomized data collection)8 | ||||||||
| Graduated at the end of 12th grade | Subsample | 79 | 32 (47) | 27 (45) | 5 | 0.14 | ns | 6 |
|
ns = not statistically significant 1 This appendix presents subgroup and follow-up findings for outcomes related to completing school. Outcomes related to the full sample are used for rating purposes, but no rating is given in this report because results for the full sample are not available.2 The outcomes are binary, so means represent percentages. 3 The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. Standard deviations were derived from binary outcomes in this study. 4 Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 5 Effect sizes were calculated using the Cox index for binary outcomes. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. 6 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 7 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results. 8 The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple comparisons were not done for findings not included in the overall intervention rating). For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of ALAS, no corrections were necessary. |
||||||||
Appendix A5.1 ALAS rating for the staying in school domain
The WWC rates an intervention’s effects for a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.1
For the outcome domain of staying in school, the WWC rated ALAS as having potentially positive effects. It did not meet the criteria for positive effects, because it had only one study. The remaining ratings (mixed, no discernible effects, potentially negative, and negative) were not considered because ALAS received a higher applicable rating.
| Rating received |
|---|
|
Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence.
|
| Other ratings considered |
|
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.
|
|
1 For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain level effects. The WWC also considers the size of the domain level effects for ratings of potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. |
Appendix A5.2 ALAS rating for the progressing in school domain
The WWC rates an intervention’s effects for a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.1
For the outcome domain of progressin, g in school, the WWC rated ALAS as having potentially positive effects. It did not meet the criteria for positive effects because it had only one study. The remaining ratings (mixed, no discernible effects, potentially negative, and negative) were not considered because ALAS received a higher applicable rating.
| Rating received |
|---|
|
Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence.
|
| Other ratings considered |
|
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.
|
|
1 For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain level effects. The WWC also considers the size of the domain level effects for ratings of potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. |