Appendix A1.1 Study characteristics: Dynarski, Gleason, Rangarajan, & Wood, 1998—Georgia study (randomized controlled trial with differential attrition)
| Characteristic | Description |
|---|---|
| Study citation | Dynarski, M., Gleason, P., Rangarajan, A., & Wood, R. (1998). Impacts of dropout prevention programs: Final report. A research report from the School Dropout Demonstration Assistance Program evaluation. Princeton, NJ: Mathematica Policy Research, Inc. |
| Participants |
The Georgia study used a randomized controlled trial research design. The study sample included one cohort of 160 students who entered the 7th or 8th grade in the Griffin-Spalding (Georgia) school district in the 1993/94 school year. All students had been retained in grade at least once. Eighty students were randomly assigned to the accelerated middle school group and were offered admission to Griffin-Spalding Middle School Academy. The other 80 students were randomly assigned to the control group and generally attended one of the other three traditional middle schools in the district. Participants were, on average, 14-years-old when they entered the program. About 60% of students were African-American; most others were White. More than 70% were male. About three-quarters of participants had discipline problems in the previous school year. Results summarized here are drawn from a follow-up survey administered two years after random assignment: 67 intervention-group students (84%) and 73 control-group students (91%) responded. Because the response rates represent differential attrition of more than 5 percentage points, the What Works Clearinghouse (WWC) rated this study as meeting evidence standards with reservations. Researchers compared the baseline characteristics of follow-up survey respondents in the two research groups on 13 demographic, socioeconomic, and school performance measures. A statistical test of the overall difference between the research groups on the full set of 13 baseline characteristics found that a statistically significant baseline difference did exist between the research groups (at the 0.10 significance level). Study authors report that intervention-group students were more likely to be from two-parent families, less likely to receive public assistance, and less likely to be frequently absent from school. Researchers used regression models to adjust for these differences when estimating the effects of the program. |
| Setting | The Georgia study was conducted in the Griffin-Spalding school district south of Atlanta, Georgia. |
| Intervention |
During the evaluation period the Griffin-Spalding Middle School Academy served 7th and 8th graders who had been retained in grade at least once. The school has since closed. The aim of the program was to cover two years of core curriculum content in one year so that students could “leap frog” into grade 9 and rejoin their age peers if they passed all their required courses. For example, students in the academy covered both life science and earth science in one year, whereas, in other district middle schools, a full year was devoted to each of these subjects. The program accepted about 25 students a year from each of the district’s three middle schools. The academy was located in a church building that also housed a similar program for behind-grade-level high school students. The middle and high school programs shared the same teachers, administrators, and other staff. The school emphasized hands-on instruction and downplayed traditional lecture methods. Teachers were given greater flexibility than other district teachers to decide what specific material to cover in their classes. The school used the “SUCCESS” curriculum, which focused on self-esteem and decision making skills, and integrated it into the core curriculum classes. The academy did not assign homework and did not allow students to take books home because staff members were concerned that students would not return the materials. The middle school academy offered few electives—it did not offer art, music, or foreign language—to make additional time in core academic subjects. Students could return to their home middle schools to participate in extracurricular activities. The school employed a part-time counselor to work with students and their families. Information from student follow-up surveys suggests that school staff members referred substantial numbers of students to support services outside school. According to student reports, 42% of intervention-group students were referred to a social service agency for counseling, health needs, or other assistance, compared with only 15% of control-group students (Dynarski et al. 1998). |
| Comparison | Control-group students typically remained in one of the three traditional middle schools in the Griffin-Spalding school district. |
| Primary outcomes and measurement |
Two relevant outcomes from the Georgia study are included in this summary: the dropout rate and highest grade completed. (For a more detailed description of these outcome measures, see Appendices A2.1 and A2.2.) The study also examined the program’s effects on absenteeism, English and math grades, self-esteem, and perceived likelihood of completing high school. These outcomes do not fall within the three domains examined by the WWC’s review of dropout prevention interventions (staying in school, progressing in school, and completing school) and are not included in this summary. |
| Staff training | Instructional staff members at Griffin-Spalding Middle School Academy were regular classroom teachers from the Griffin-Spalding district. According to evaluation team researchers, they did not receive additional training as part of their assignment to the academy. |
Appendix A1.2 Study Characteristics: Dynarski, Gleason, Rangarajan, & Wood, 1998—Michigan study (randomized controlled trial with differential attrition)
| Characteristic | Description |
|---|---|
| Study citation | Dynarski, M., Gleason, P., Rangarajan, A., & Wood, R. (1998). Impacts of dropout prevention programs: Final report. A research report from the School Dropout Demonstration Assistance Program evaluation. Princeton, NJ: Mathematica Policy Research, Inc. |
| Participants |
The Michigan study used a randomized controlled trial research design. The study sample included 198 students who entered the 6th grade in the Flint Community Schools district in the 1992/93 and 1993/94 school years. All students were two or more years behind grade level and were typically 13- or 14-years-old when they entered the program. One hundred twelve students were randomly assigned to the accelerated middle school group and offered admission to Accelerated Academics Academy. The other 86 students were randomly assigned to the control group and typically attended one of the four traditional middle schools in the district. District staff members identified new students for the academy each spring from 5th graders who were two or more years overage for grade. From this group they selected students they considered most likely to benefit from the accelerated program. They made these assessments based on the students’ academic performance, as well as interviews with school staff, parents, and the students themselves. About 60% of students were African American; most others were White. About 60% were male. About half the participants lived in households that received public assistance. More than two-thirds had had discipline problems in the previous school year. Results summarized here are drawn from a follow-up survey administered two years after random assignment: 100 intervention-group students (89.3%) and 72 control-group students (83.7%) responded. Because these response rates represent differential attrition of more than 5 percentage points, the WWC rated this study as meeting evidence standards with reservations. Researchers compared the baseline characteristics of follow-up survey respondents in the two research groups on 13 demographic, socioeconomic, and school performance measures. A statistical test of the overall difference between the research groups on the full set of 13 baseline characteristics found that the groups were not significantly different at the 0.10 significance level. Even so, researchers used regression models to adjust for small differences in the initial characteristics of intervention- and control-group students when estimating the effects of the program. |
| Setting | The Michigan study was conducted in the Flint Community Schools school district in Flint, Michigan. |
| Intervention |
During the evaluation period the Accelerated Academics Academy (AAA) served middle school students who were two or more years behind grade level. (The school continues to operate, but this description focuses on its operations during the evaluation period.) The goal of the program was to accelerate instruction so that behind-grade-level students could enter high school with their age peers. AAA was a self-contained program that occupied an entire floor of a former middle school. The other two floors were occupied by a private school. Enrollment in the school was limited to 100 students. The program offered smaller classes than other middle schools in Flint and placed a greater emphasis on thematic instruction and integrating the curriculum across core academic subjects. Teachers often used nontraditional approaches, such as cooperative learning groups, instructional technology, collaborative teaching, and peer tutoring. The curriculum was flexible and not driven by textbook content. To make the curriculum more relevant and engaging, instruction often centered on current issues and events. AAA offered five core subjects: language arts, mathematics, science, social studies, and art. In addition, the regular schedule was compressed each Wednesday to make room for a "Wonderful Wednesday" class that included a rotating set of topics chosen based on student interests, such as algebra, Spanish, quilting, and science club. Each school day began with a 30-minute "family period" in which a group of 10 students met with a staff member. These sessions could include a mix of activities, such as cooperative learning, tutoring, counseling, silent reading, or group discussions. During the sessions students had the opportunity to discuss issues of concern to them, such as violence in the community, substance use, and family relationships. The school employed a full-time counselor and a full-time social worker for students. This substantial in-house student support may explain why fewer intervention-group students than control-group students reported receiving referrals to outside social service agencies, 5% compared with 18% (Dynarski et al. 1998). The school also employed two paraprofessional "student advocates" who provided in-class tutoring and other supports to students. |
| Comparison | Control-group students typically attended one of the four traditional middle schools in Flint. |
| Primary outcomes and measurement |
Two relevant outcomes from the Michigan study are included in this summary: the dropout rate and highest grade completed. (For a more detailed description of these outcome measures, see Appendices A2.1 and A2.2.) The study also examined the program’s effects on absenteeism, English and math grades, self-esteem, and perceived likelihood of completing high school. These outcomes do not fall within the three domains examined by the WWC’s review of dropout prevention interventions (staying in school, progressing in school, and completing school) and are not included in this summary. |
| Staff training | AAA instructional staff members were regular classroom teachers from the Flint Community Schools school district. No additional information was available concerning their training. |
Appendix A1.3 Study Characteristics: Dynarski, Gleason, Rangarajan, & Wood, 1998—New Jersey study (randomized controlled trial)
| Characteristic | Description |
|---|---|
| Study citation | Dynarski, M., Gleason, P., Rangarajan, A., & Wood, R. (1998). Impacts of dropout prevention programs: Final report. A research report from the School Dropout Demonstration Assistance Program evaluation. Princeton, NJ: Mathematica Policy Research, Inc. |
| Participants |
The New Jersey study used a randomized controlled trial research design. The study sample included 620 students who entered the 6th or 7th grade in the Newark public schools in the 1992/93 and 1993/94 school years. Of the 620 students in the research sample, 392 were randomly assigned to the accelerated middle school group and offered admission to Project Accelerated Curriculum Classes Emphasizing Learning (ACCEL).The other 228 students were randomly assigned to the control group and did not enroll in the accelerated program. District staff hoped to select students for Project ACCEL who would succeed in an accelerated academic program. For this reason, they used fairly stringent screening criteria. Each spring letters were sent home describing the program to parents of 5th and 6th graders in the district who had been retained in grade at least once. Interested parents had to complete an application. From these applicants participants were selected based on teacher recommendations and student interviews with a team of Project ACCEL staff members. Students in the program were typically 13-years-old. Three-quarters were African-American; most others were Hispanic. About half were male. About half lived in households that received public assistance, and about half had discipline problems in the previous school year. Results summarized here are drawn from a follow-up survey administered two years after random assignment: 341 intervention-group students (87%) and 194 control-group students (85%) responded. Researchers compared the baseline characteristics of follow-up survey respondents in the two research groups on 13 demographic, socioeconomic, and school performance measures. A statistical test of the overall difference between the research groups on the full set of 13 baseline characteristics found that the groups were not significantly different at the 0.10 significance level. Even so, researchers used regression models to adjust for small differences in the initial characteristics of intervention- and control-group students when estimating the effects of the program. |
| Setting | The New Jersey study was conducted in the Newark Public Schools school district in Newark, New Jersey. |
| Intervention |
During the study period Project ACCEL served 6th and 7th graders who were retained in grade at least once. (The program is no longer in operation.) The aim of the program was to allow behind-grade-level middle school students to accelerate their studies and “catch up” with their age peers. Sixth graders typically stayed in the program for two years and covered three years of curriculum material. Seventh graders were in the program for one year and covered the 7th and 8th grade curriculum. Project ACCEL operated in five district schools in Newark, some that were organized as K–8 elementary schools and others that were organized as grades 5–8 middle schools. Project ACCEL used a school-within-a-school approach and operated out of a cluster of classrooms within these schools. Each of the five programs served about 50 students, taught by a team of four teachers who each covered one of four subjects: English, math, basic skills, and science/social studies. (In contrast, in other Newark classrooms for middle-grade students at that time teachers typically taught all subjects and worked with only one group of students throughout the day.) Project ACCEL instructional staff used team teaching strategies and collaborated to link the curriculum thematically across subjects. The program had a strong emphasis on discipline and attendance monitoring. Students who missed more than nine days of school were subject to termination from the program. Teachers assigned more homework than was typical in other Newark schools to facilitate the coverage of an accelerated curriculum. Classes were small and generally included 12 or 13 students. One full-time guidance counselor was available to the program and worked closely with ACCEL students and teachers. Consistent with the program’s emphasis on counseling and case management, on follow-up surveys more intervention-group students than control-group students reported having received counseling during the first follow-up year—74% compared with 59%. Similarly, more intervention-group students reported having received a referral to an outside social services agency during this period—27% compared with 15% (Dynarski et al. 1998). Project ACCEL staff members were supervised by the school principal. However, each Project ACCEL team had considerable autonomy in operating their program. |
| Comparison | Control-group students typically attended a district school and followed the standard curriculum offered by the district. |
| Primary outcomes and measurement |
Two relevant outcomes from the New Jersey study are included in this summary: the dropout rate and the highest grade completed. (For a more detailed description of these outcome measures, see Appendices A2.1 and A2.2.) The study also examined the program’s effects on absenteeism, English and math grades, self-esteem, and perceived likelihood of completing high school. These outcomes do not fall within the three domains examined by the WWC’s review of dropout prevention interventions (staying in school, progressing in school, and completing school) and are not included in this summary. |
| Staff training | All Project ACCEL staff members attended 1–2 weeks of training each August in preparation for the upcoming school year. The purpose of this training was to plan for the program and to ensure that all Project ACCEL staff members followed the same approach. In addition, all Project ACCEL staff members met as a group five or six times during the school year to discuss program issues. |
Appendix A2.1 Outcome measures for the staying in school domain
| Outcome measure | Description |
|---|---|
| Dropped out after two years | The percentage of students who dropped out of school by the end of second follow-up year. These data were collected from student follow-up surveys. |
Appendix A2.2 Outcome measures for the progressing in school domain
| Outcome measure | Description |
|---|---|
| Highest grade completed after two years | The grade level completed by the end of the second follow-up year. These data were collected from student follow-up surveys. |
Appendix A3.1 Summary of study findings included in the rating for the staying in school domain1
| Author's findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (students) | Accelerated middle schools group | Comparison group | Mean difference2 (accelerated middle schools – comparison) | Effect size3 | Statistical significance4 (at α= 0.05) |
Improvement index5 |
| Dynarski et al., 1998—Georgia study (randomized controlled trial with differential attrition)6 | ||||||||
| Dropped out after two years (%) | Full sample | 140 | 6 | 14 | 8 | 0.56 | ns | +21 |
| Average for staying in school (Dynarski et al., 1998—Georgia study)7 | 0.56 | ns | +21 | |||||
| Dynarski et al., 1998—Michigan study (randomized controlled trial with differential attrition)6 | ||||||||
| Dropped out after two years (%) | Full sample | 172 | 2 | 9 | 7 | 0.95 | Statistically significant | +33 |
| Average for staying in school (Dynarski et al., 1998—Michigan study)7 | 0.95 | Statistically significant | +33 | |||||
| Dynarski et al., 1998—New Jersey study (randomized controlled trial)6 | ||||||||
| Dropped out after two years (%) | Full sample | 536 | 6 | 5 | –1 | –0.12 | ns | –5 |
| Average for staying in school (Dynarski et al., 1998—New Jersey study)7 | –0.12 | ns | –5 | |||||
| Domain average for staying in school across all studies7 | 0.47 | na | +18 | |||||
|
ns = not statistically significant 1 This appendix reports second-year follow-up findings considered for the effectiveness rating and the improvement index for the staying in school domain. Third-year follow-up findings—available for an early cohort only—are not included in these ratings but are reported in Appendix A4.1. |
||||||||
Appendix A3.2 Summary of study findings included in the rating for the progressing in school domain1
| Author's findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome2 | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (students) | Accelerated middle schools group | Comparison group | Mean difference3 (accelerated middle schools – comparison) | Effect size4 | Statistical significance5 (at α= 0.05) |
Improvement index6 |
| Dynarski et al., 1998—Georgia study (randomized controlled trial with differential attrition)7 | ||||||||
|
Highest grade completed after two years |
Full sample | 140 | 8.6 (0.48) |
7.9 (0.42) |
0.7 | 1.53 | Statistically significant | +44 |
| Average for progressing in school (Dynarski et al., 1998—Georgia study)8 | 1.53 | Statistically significant | +44 | |||||
| Dynarski et al., 1998—Michigan study (randomized controlled trial with differential attrition)7 | ||||||||
| Highest grade completed after two years | Full sample | 172 | 7.3 (0.37) |
6.8 (0.44) |
0.5 | 1.25 | Statistically significant | +39 |
| Average for progressing in school (Dynarski et al., 1998—Michigan study)8 | 1.25 | Statistically significant | +39 | |||||
| Dynarski et al., 1998—New Jersey study (randomized controlled trial)7 | ||||||||
| Highest grade completed after two years | Full sample | 536 | 7.8 (0.85) |
7.5 (0.70) |
0.3 | 0.38 | Statistically significant | +15 |
| Average for progressing in school (Dynarski et al., 1998—New Jersey study)8 | 0.38 | Statistically significant | +15 | |||||
| Domain average for progressing in school across all studies8 | 1.06 | na | +35 | |||||
|
na = not applicable 1 This appendix reports second-year follow-up findings considered for the effectiveness rating and the improvement index for the progressing in school domain. Third-year follow-up findings—available for an early cohort only—are not included in these ratings but are reported in Appendix A4.2. |
||||||||
Appendix A4.1 Summary of additional findings for the staying in school domain1
| Author's findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome |
WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (students) | Accelerated middle schools group | Comparison group | Mean difference2(accelerated middle schools – comparison) | Effect size3 | Statistical significance4 (at α= 0.05) |
Improvement index5 |
| Dynarski et al., 1998-Michigan study (randomized controlled trial with differential attrition)6 | ||||||||
| Dropped out after three years (%) | Early cohort | 76 | 3 | 17 | 14 | 1.13 | Statistically significant | +37 |
| Dynarski et al., 1998—New Jersey study (randomized controlled trial)6 | ||||||||
| Dropped out after three years (%) | Early cohort | 296 | 15 | 19 | 4 | 0.17 | ns | +7 |
|
ns = not statistically significant 1 This appendix reports third-year follow-up findings for dropout rates, which were only available for an early cohort. This early cohort entered the Michigan and New Jersey programs during the 1992/93 school year. Second-year follow-up findings were available for the full sample and were used for determining the effectiveness rating. These results are reported in Appendix A3.1. |
||||||||
Appendix A4.2 additional study findings in the Progressing in school domain1
| Author's findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome2 (standard deviation) |
WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (students) | Accelerated middle schools group | Comparison group | Mean difference3 (accelerated middle schools – comparison) | Effect size4 | Statistical significance5 (at α= 0.05) |
Improvement index6 |
| Dynarski et al., 1998—Michigan study (randomized controlled trial with differential attrition)7 | ||||||||
| Highest grade completed after three years | Early cohort | 76 | 8.5 (0.32) |
7.8 (0.61) |
0.7 | 1.45 | Statistically significant | +43 |
| Dynarski et al., 1998—New Jersey study (randomized controlled trial)7 | ||||||||
| Highest grade completed after three years | Early cohort | 296 | 8.7 (0.78) |
8.4 (0.69) |
0.3 | 0.40 | Statistically significant | +16 |
|
1 This appendix reports third-year follow-up findings for the highest grade completed, which were only available for an early cohort. This early cohort entered the Michigan and New Jersey programs during the 1992/93 school year. Second-year follow-up findings were available for the full sample and were used for determining the effectiveness rating. These results are reported in Appendix A3.2. |
||||||||
Appendix A5.1 Accelerated middle schools rating for the staying in school domain
The WWC rates an intervention’s effects for a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.1
For the outcome domain of staying in school, the WWC rated accelerated middle schools as having potentially positive effects. They did not meet the criteria for positive effects because only one study showed statistically significant positive effects. The remaining ratings (mixed effects, no discernible effects, potentially negative effects, and negative effects) were not considered because accelerated middle schools were assigned the highest applicable rating.
| Rating received |
|---|
|
Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence.
AND
|
| Other ratings considered |
|
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.
AND
|
|
1 For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of potentially positive or potentially negative effects. For a complete description, see the WWC Intervention Rating Scheme. |
Appendix A5.2 Accelerated middle schools rating for the progressing in school domain
The WWC rates an intervention’s effects for a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.1
For the outcome domain of progressing in school, the WWC rated accelerated middle schools as having positive effects. The remaining ratings (potentially positive effects, mixed effects, no discernible effects, potentially negative effects, and negative effects) were not considered because accelerated middle schools were assigned the highest applicable rating.
| Rating received |
|---|
|
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.
AND
|
|
1 For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of potentially positive or potentially negative effects. For a complete description, see the WWC Intervention Rating Scheme. |
Appendix A6.1 Extent of evidence by domain
| Sample size | ||||
|---|---|---|---|---|
| Outcome domain | Number of studies | Schools | Students | Extent of evidence1 |
| Staying in school | 3 | 14 | 848 | Medium to large |
| Progressing in school | 3 | 14 | 848 | Medium to large |
| Completing school | na | na | na | na |
|
na = not applicable 1 A rating of "medium to large" requires at least two studies and two schools across studies in one domain and a total sample size across studies of at least 350 students or 14 classrooms. Otherwise, the rating is "small." |
||||
|Institute of Education Sciences