National Assessment of Title I: Interim Report. Volume II: Closing the Reading Gap: First Year Findings from a Randomized Trial of Four Reading Interventions for Striving Readers. NCEE 2006-4002

Findings

Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.

Findings

Completing school outcomes—Indeterminate effect found for the domain

Outcome measure	Comparison	Period	Sample	Intervention mean	Comparison mean	Significant?	Improvement     index	Evidence tier
Earned a high school diploma or GED by end of year 9 (%)	Quantum Opportunity Program vs. Business as usual	Posttest	Full sample; 914 students	78.00	75.00	No	--

Progressing in school outcomes—Indeterminate effect found for the domain

Outcome measure	Comparison	Period	Sample	Intervention mean	Comparison mean	Significant?	Improvement     index	Evidence tier
Total credits earned by the end of year 5	Quantum Opportunity Program vs. Business as usual	Posttest	Full sample; 766 students	16.20	15.80	No	--

Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.

Study Details

Setting

This study took place at 11 high schools in seven evaluation sites: Fort Worth, Texas; Cleveland, Ohio; Washington, DC; Houston, Texas; Memphis, Tennessee; Philadelphia, Pennsylvania; and Yakima, Washington. Three sites implemented QOP in multiple high schools: Washington, DC (2 high schools), Houston (2 high schools), and Memphis (3 high schools). All other sites implemented QOP in a single high school.

Study sample

The Quantum Opportunity Program Demonstration Project used a randomized controlled trial research design. The demonstration operated in seven sites and served a single cohort of entering ninth graders over a five-year period. In six of the seven sites, the programs served ninth graders who entered high school in the fall of 1995. In one site (Washington, DC) the program served ninth graders who entered high school in the fall of 1996. QOP served students from high schools with dropout rates of 40% or more. To be eligible for QOP, students in these high schools had to meet the following three criteria: (1) they were entering ninth graders who were not repeating the ninth grade; (2) they had a grade point average below the 67th percentile of entering ninth graders at the participating high school; and (3) they did not have severe physical and learning disabilities that would prevent them from participating in the program. A sample of students meeting these criteria were drawn from lists of entering ninth graders; more than 97% of those identified agreed to participate in the study. The participating youth were then randomly assigned to either an intervention group that was enrolled in QOP or a control group that was not. Across the seven locations, 580 students were assigned to QOP group and 489 were assigned to the control group. Researchers compared the baseline characteristics of QOP and control group students on gender, age, race/ethnicity, and grade point average and found no statistically significant differences between the research groups. Participants were typically 13 or 14 years old, about two-thirds of participants were African-American, and a quarter were Hispanic. Participants were evenly split between males and females. Results summarized in this report are based on high school transcripts and three telephone surveys. One of these surveys was conducted at the end of the five-year demonstration, another two years after the demonstration had ended, and a third four years after the demonstration’s end. There are two outcomes of interest for the WWC review of the effectiveness of QOP: total credits earned five years after program entry and high school diploma or GED certificate receipt within nine years of program entry. Total credits earned are based on transcript data and are available for 86% of QOP students and 77% of control group students. High school completion information is based on data from all three survey waves, as well as transcript data, and is available for 88% of QOP students and 83% of control group students.2 For credits earned, the rate of differential attrition exceeded the 5% threshold used for WWC dropout prevention reviews. For high school completion, the rate of differential attrition was equal to this threshold. Because one measure used to rate QOP’s effectiveness exceeded the differential attrition standard, the WWC rated this study as meeting evidence standards with reservations. To account for nonresponse, the study authors calculated impacts using weights that adjust for differences between respondents and nonrespondents in baseline characteristics. However, the WWC did not consider this statistical adjustment sufficient for overcoming the differential attrition.

Intervention Group

QOP Demonstration Project was an intensive, five-year, case management and mentoring program for high school youth that emphasized after-school supplemental education, developmental activities, and community service. Its primary goals were to increase the likelihood that enrollees would complete high school and enter a postsecondary education or training program. The program also aimed to reduce risky behaviors such as substance abuse, crime, and teenage parenting. QOP was operated by community-based organizations in seven sites. It offered a cohort of entering ninth graders services for up to five years and continued to provide services even if participants dropped out of school or moved out of the school district. The comprehensive program had four primary components: case management and mentoring, educational and developmental activities, supportive services, and financial incentives. These components are described in more detail below. 1. Case management and mentoring: Case managers typically had caseloads of 15 to 25 participants. Most case managers had office space within the school and were available to enrollees during the school day, at night, on weekends, and during school vacations. Case managers served as mentors to the enrollees and were typically assigned to the same enrollee for the full five years of the demonstration. According to an implementation study of QOP, most sites successfully implemented this component of the model (Maxfield et al., 2003). 2. Education services, community service activities, and developmental activities: QOP’s participation target was 750 hours per year per enrollee. One-third of that time was to be spent on educational activities (such as tutoring or computer-assisted learning), one-third on community service (such as visiting residents of a local nursing home), and one-third on developmental activities (such as life skills and employment-readiness training). In most sites, based on program staff’s assessment of participant need, resources were reallocated from community service activities toward case management and education activities. According to an implementation study of QOP, many sites did not fully implement QOP’s education component, such as sustained course-based tutoring and computer-assisted instruction in basic reading and math skills (Maxfield et al. 2003). In addition, in many cases developmental activities designed to teach life skills were primarily recreational activities. Overall, enrollees spent an average of 174 hours per year on education, community service, and developmental activities—23% of the annual goal of 750 hours. Enrollees who spent little time participating in QOP activities frequently reported that they thought the program was too much like school or that they had a barrier to participation, such as a job or a child care or transportation problem (Maxfield et al., 2003). 3. Supportive services: QOP provided transportation assistance to facilitate attendance at program activities, as well as referrals to other resources in the community (such as mental health services and summer jobs programs). According to an implementation study of QOP, most sites successfully provided transportation services; however, most did not provide adequate child care support and did not consistently offer health screenings and referrals (Maxfield et al., 2003). 4. Financial Incentives: QOP provided enrollees with a stipend of approximately $1.25 for every hour devoted to program activities other than mentoring or recreation. A matching amount was deposited in an accrual account, to be used by enrollees after they completed high school and enrolled in college, vocational training, or the military. Enrollees could also earn bonuses for achieving major milestones, such as a grade point average above a set benchmark. According to an implementation study of QOP, sites generally implemented this program component successfully (Maxfield et al., 2003).

Comparison Group

Control group members were not eligible to participate in QOP but could participate in other services available in the community. Based on responses to follow-up surveys, 16% of control group members participated in a program for disadvantaged youth other than QOP (Schirm et al., 2003). According to study authors, these other programs were generally substantially less intensive than QOP.

Outcome descriptions

Two relevant outcomes from the QOP study are included in this summary and were used for rating purposes: the number of credits earned five years after program entry and high school diploma or GED certificate receipt within nine years of program entry. (See Appendices A2.1 and A2.2 for a more detailed description of outcome measures.) The study also examined the program’s effects on academic outcomes, postsecondary outcomes, risky behaviors, resiliency factors, and attitudes. However, these outcomes do not fall within the three domains examined by the WWC’s review of dropout prevention interventions (staying in school, progressing in school, and completing school). Therefore, these additional outcomes are not included in this report.

Support for implementation

QOP staff attended annual training conferences provided by Opportunities Industrialization Centers of America (OICA) during the demonstration period. The initial training lasted seven days with annual four-day sessions in subsequent years. The Ford Foundation funded technical assistance for all seven QOP demonstration sites to be delivered by OICA. Technical assistance activities included helping sites set up and maintain QOP management information systems, conducting site visits, helping resolve case management issues, and providing sites with developmental curriculum material and computer-assisted instruction (CAI) CD-ROMs. In addition, the U.S. Department of Labor provided technical assistance on selecting computer-assisted instruction (CAI) software, guidelines for setting up and operating accrual accounts, and quarterly calls with each site to discuss service delivery strategies.

Additional Sources