Skip Navigation

What Works Clearinghouse


Search

Appendices


Appendix A1 Study Characteristics: Cave, Bos, Doolittle, & Toussaint, 1993 (randomized controlled trial)

Characteristic Description
Study citation Cave, G., Bos, H., Doolittle, F., & Toussaint, C. (1993). JOBSTART: Final report on a program for school dropouts. New York, NY: MDRC.
Participants

JOBSTART served youth who were: (1) 17 to 21 years old, (2) lacking a high school diploma or GED certificate, (3) reading below an 8th-grade level, and (4) economically disadvantaged. To meet enrollment targets, sites could waive the poor-reading-skills criterion for 20% of their enrollees. Applicants were considered economically disadvantaged if they: (1) received public assistance, (2) had a family income at or below the poverty line, or (3) were homeless.

From 1985 to 1987, JOBSTART participants were recruited and enrolled by 13 sites in nine states. All participants were high school dropouts, and half had not worked at all in the year prior to enrolling in the program. Most JOBSTART recruits were non-white, 44% African-American, and 44% Hispanic. Just over half enrollees (54%) were women, and about half the women were young mothers. At baseline, 27% of sample members were receiving Aid to Families with Dependent Children or general assistance, while 38% were receiving food stamps.

In the JOBSTART demonstration, 2,312 youth were randomly assigned to either JOBSTART or a control condition. The analyses in this report are based on data collected in the 48-month follow-up survey, to which 85% of JOBSTART youth and 83% of control group youth responded. Among the 1,941 respondents, the study authors compared the baseline characteristics of JOBSTART and control group youth, including their gender, ethnicity, family structure, employment history, and whether they received public assistance. They found no statistically significant differences between the research groups on these characteristics.

Setting JOBSTART was evaluated in 13 study sites in nine states: Arizona (Phoenix), California (Los Angeles, Monterey Park, San Jose), Colorado (Denver), Connecticut (Hartford), Illinois (Chicago), Georgia (Atlanta), New York (Buffalo, New York City), Pennsylvania (Pittsburgh), and Texas (Corpus Christi, Dallas). The sites differed in their organization and structure: four of the sites were adult schools (three vocational, one community college), six were community-based organizations, and three were nonresidential Job Corps programs.
Intervention JOBSTART offered a combination of education and occupational preparation services to improve the employment prospects of high school dropouts. Participation was voluntary. JOBSTART was modeled after Job Corps, which according to the study authors, was one of the few programs that had documented success in improving the outcomes of high school dropouts. But Job Corps was relatively expensive and required a residential commitment (a considerable barrier to participation for many youth). JOBSTART offered many of the same components as Job Corps, but was nonresidential. In addition, in an effort to reduce the program cost, JOBSTART did not include some of Job Corps’ most intensive services, such as extensive support services, paid work experience, and financial compensation.

JOBSTART emphasized education and occupational preparation. It had four key components: (1) instruction in basic academic skills, (2) occupational skills training, (3) support services to facilitate participation, and (4) job placement assistance. Although all sites were required to offer the four components, implementation of the components varied. Some sites, for instance, offered all components in-house, while others linked participants with outside agencies that provided these services. The four key components of JOBSTART are described in more detail below:
  1. Instruction in basic academic skills. Sites offered participants a minimum of 200 hours of self-paced instruction on reading, math, and other subjects covered on the GED test. Sites were not required to use a particular curriculum or materials. During basic academic skills classes, participants worked on exercises using workbooks or, less commonly, computer-assisted instruction. Teachers reviewed their progress and provided individualized instruction. Sites had flexibility in how they implemented this component of JOBSTART. For example, some sites offered group instruction, while others did not. Sites also selected the curricula and the number of hours per week that were focused on basic skills. Typically, basic academic skills classes met two to three hours per day three to five days per week. According to study authors, JOBSTART youth spent, on average, 125 hours in the program's basic academic skills classes (Cave et al., 1993). This average includes the 11% of youth who were assigned to JOBSTART but did not participate in the program.

  2. Occupational skills training. Sites offered participants a minimum of 500 hours of occupational skills training. Occupational skills instruction was classroom based and designed to prepare youth for high-demand occupations. Classes met three to four hours a day, three to five days a week. In 8 of the 13 sites, occupational skills training was offered concurrently with basic academic skills instruction. In the other sites, participants received occupational skills training after completing their basic academic skills instruction. Sites were instructed to develop training materials with the assistance of private-sector partners to meet the entry-level requirements of local employers. Sites developed courses to cover a range of occupations, including clerical and service jobs, machine trades, and other skilled trades. According to study authors, JOBSTART youth completed, on average, 238 hours of occupational skills instruction (Cave et al., 1993). This average includes the 11% of youth who were assigned to JOBSTART but did not participate in the program.

  3. Support services. To facilitate participation in the program, JOBSTART offered transportation and child care assistance. Transportation assistance could take the form of bus passes or small allowances. Child care was mainly provided by other agencies off-site, with JOBSTART covering the costs. Most sites also offered participants small amounts of money for emergencies (to cover the cost of rent or clothing, for example), as well as on-site meals. About half the sites offered life-skills training on such topics as health, personal finances, and workplace expectations.

  4. Job placement assistance. Sites assisted participants in finding jobs upon completing JOBSTART's education and occupational skills components. Most sites offered participants instruction in employers' expectations and job search techniques. About half the sites arranged internships with employers for participants. Assistance in finding permanent employment was often not offered until participants completed the program's education and occupational skills components. Because many participants dropped out of JOBSTART before completing the education and skills training, many did not receive help with job placement. On follow-up surveys, only one-quarter of JOBSTART participants reported that staff had informed them of job openings and opportunities.
Comparison Control group youth were not eligible to participate in JOBSTART but could voluntarily participate in other services available in the community. Based on survey responses, many control group youth participated in education and training programs during the follow-up period—but their rate of participation was substantially lower than it was among JOBSTART youth (Cave et al., 1993). During the four years after random assignment, 56% of control group youth reported receiving remedial or occupational instruction, compared with 94% of JOBSTART youth. Over this period, the average amount of participation in education and training was 432 hours for control group youth, compared with 800 hours for the JOBSTART youth (including their participation in JOBSTART as well as other education and training activities).
Primary outcomes and measurement One relevant outcome from the JOBSTART study is included in this summary and used for rating purposes: receiving a high school diploma or GED certificate within 48 months of random assignment. For a more detailed description of the outcome measure, see Appendix A2. The study also examined the program’s effects on employment, hours worked, total earnings, welfare receipt, pregnancy, criminal activity, and drug use. These outcomes, however, do not fall within the three domains examined by the WWC’s review of dropout prevention interventions (staying in school, progressing in school, and completing school). So, they are not included in this report.
Staff training Many JOBSTART staff were former teachers from public schools or community colleges. In many instances, these teachers had experience working with disadvantaged youth or adults. In some instances, they had taught GED preparation classes or remedial education prior to working for JOBSTART. Other JOBSTART staff had previously worked in other employment programs for disadvantaged youth (Auspos, Cave, Doolittle, & Hoerz, 1989).

Top

Appendix A2 Outcome measures for the completing school domain

Outcome measure Description
Earned a high school diploma or GED certificate within 48 months of random assignment This binary measure represents the percentage of students who either received a high school diploma or GED certificate within 48 months of random assignment. This measure was based on the sample member’s response to the 48-month survey.

Top

Appendix A3 Summary of study findings included in the rating for the completing school domain1

  Author's findings from the study  
  Mean outcome WWC calculations
Outcome measure Study sample Sample size (students) JOBSTART group Comparison group Mean difference2 (JOBSTART – comparison) Effect size3 Statistical significance4
(at α= 0.05)
Improvement index5
Cave, Bos, Doolittle, & Toussaint, 1993 (randomized controlled trial)6
Earned a high school diploma or GED certificate within 48 months of random assignment (%) Full sample 1,941 42.0 28.6 13.4 0.36 Statistically significant +14
Domain average for completing school7 0.36 Statistically significant +14

1 This appendix reports findings considered for the effectiveness rating and the average improvement index for the completing school domain. Subgroup findings are not included in these ratings but are in Appendix A4.1. Appendix A4.2 reports the separate effects of JOBSTART on earning a GED certificate or high school diploma, which were not used in JOBSTART’s effectiveness rating.
2 Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group.
3 Effect sizes for dichotomous variables were computed using the Cox Index. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations.
4 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between groups.
5 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and the average student in the comparison condition. The improvement index can take on values between –50 and +50, with positive numbers denoting results favorable to the intervention group.
6 The level of statistical significance was reported by the study authors, or where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. For the formulas the WWC used to calculate statistical significance, see Technical Details of WWC-Conducted Computations. In this case, no corrections for clustering or multiple comparisons were needed.
7 The domain improvement index is calculated from the average effect size.

Top

Appendix A4.1 Summary of subgroup findings by gender and parenthood status for the completing school domain1

  Author's findings from the study  
  Mean outcome WWC calculations
Outcome measure Study sample Sample size (students) JOBSTART group Comparison group Mean difference2 (JOBSTART – comparison) Effect size3 Statistical significance4
(at α= 0.05)
Improvement index5
Cave, Bos, Doolittle, & Toussaint, 1993 (randomized controlled trial)6
Earned a high school diploma or GED certificate within 48 months of random assignment (%) Men 900 42.0 28.3 13.7 0.37 Statistically significant +14
Earned a high school diploma or GED certificate within 48 months of random assignment (%) Women living with own children at baseline 508 42.0 26.7 15.3 0.42 Statistically significant +16
Earned a high school diploma or GED certificate within 48 months of random assignment (%) Women not living with own children at baseline 533 41.6 31.3 10.4 0.27 Statistically significant +11

1 This appendix presents subgroup findings by gender and parenthood status for the intervention’s effects on receiving a high school diploma or GED certificate. The full sample was used for determining the effectiveness rating. These findings are in Appendix A3.
2 Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group.
3 Effect sizes for dichotomous variables were computed using the Cox Index. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations.
4 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between groups.
5 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and the average student in the comparison condition. The improvement index can take on values between –50 and +50, with positive numbers denoting results favorable to the intervention group.
6 The level of statistical significance was reported by the study authors, or where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. For the formulas the WWC used to calculate statistical significance, see Technical Details of WWC-Conducted Computations. In this case, no corrections for clustering or multiple comparisons were needed.

Top

Appendix A4.2 Summary of additional findings for the completing school domain1

  Author's findings from the study  
  Mean outcome WWC calculations
Outcome measure Study sample Sample size (students) JOBSTART group Comparison group Mean difference2 (JOBSTART – comparison) Effect size3 Statistical significance4
(at α= 0.05)
Improvement index5
Cave, Bos, Doolittle, & Toussaint, 1993 (randomized controlled trial)6
Earned a GED certificate within 48 months of random assignment (%) Full sample 1,941 37.6 21.1 16.5 0.49 Statistically significant +19
Earned a high school diploma or GED certificate within 48 months of random assignment (%) Full sample 1,941 4.4 7.5 –3.1 –0.34 Statistically significant –13

1 This appendix presents separate effects of JOBSTART on receiving a high school diploma or GED certificate. The intervention’s combined effect on receiving a high school diploma or GED certificate was used for determining the effectiveness rating and is in Appendix A3.
2 Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group.
3 Effect sizes for dichotomous variables were computed using the Cox Index. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations.
4 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between groups.
5 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and the average student in the comparison condition. The improvement index can take on values between –50 and +50, with positive numbers denoting results favorable to the intervention group.
6 The level of statistical significance was reported by the study authors, or where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. For the formulas the WWC used to calculate statistical significance, see Technical Details of WWC-Conducted Computations. In this case, no corrections for clustering or multiple comparisons were needed.

Top

Appendix A5 JOBSTART rating for the completing school domain

The WWC rates an intervention’s effects in a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.1

For the outcome domain of completing school, the WWC rated JOBSTART as having potentially positive effects. It did not meet the criteria for positive effects because only one study showed a statistically significant or substantively important effect in this domain. The remaining ratings (mixed effects, no discernable effects, potentially negative effects, negative effects) were not considered because JOBSTART was assigned the highest applicable rating.

Rating received

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence.

  • Criterion 1: At least one study showing a statistically significant or substantively important positive effect.

    Met. One study of JOBSTART meeting WWC evidence standards found a statistically significant effect in this domain.

           AND

  • Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing indeterminate effects than showing statistically significant or substantively important positive effects.

    Met. No study of JOBSTART found statistically significant or substantively important negative effects in the completing school domain.

Other ratings considered

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.

  • Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design.

    Not met. JOBSTART had only one study that met WWC evidence standards.

           AND

  • Criterion 2: No studies showing statistically significant or substantively important negative effects.

    Met. No study found statistically significant or substantively important negative effects in this domain.

1 For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of potentially positive or potentially negative effects. For a complete description, see the WWC Intervention Rating Scheme.

Top

Appendix A6 Extent of evidence by domain

  Sample size
Outcome domain Number of studies Schools Students Extent of evidence1
Staying in school 0 na na na
Progressing in school 0 na na na
Completing school 1 13 1,941 Small

na = not applicable/not studied

1 A rating of "medium to large" requires at least two studies and two schools across studies in one domain and a total sample size across studies of at least 350 students or 14 classrooms. Otherwise, the rating is "small."

Top