Grossman, J. B., & Sipe, C. L. (1992). Philadelphia, PA: Public/Private Ventures.
Reviewed: May 2009
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Please see the WWC summary of evidence for Summer Training and Education Program (STEP).
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Highest grade completed |
Summer Training and Education Program (STEP) vs. Work only |
Posttest |
Cohort 2;
|
11.20 |
11.20 |
No |
-- | |
|
Highest grade completed |
Summer Training and Education Program (STEP) vs. Work only |
Posttest |
Cohort 3;
|
10.80 |
10.90 |
No |
-- |
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Dropped out |
Summer Training and Education Program (STEP) vs. Work only |
Posttest |
Cohort 3;
|
21.60 |
22.60 |
No |
-- | |
|
Dropped out |
Summer Training and Education Program (STEP) vs. Work only |
Posttest |
Cohort 2;
|
29.20 |
25.10 |
No |
-- |
Characteristics of study sample as reported by study author.
| Asian | 18% | |
| Black | 47% | |
| Other or unknown | 15% |
| Hispanic | 19% | |
| Not Hispanic or Latino | 81% |
The five study sites were Job Training and Partnership Act local employment and training agencies that operated both STEP and SYETP. These sites were located in Boston, MA; Fresno, CA; Portland, OR; San Diego, CA; and Seattle, WA. Remedial education and life skills learning classes were typically held at a community school or college. Work experience was conducted in the community near where the classes were held.
The study used a randomized controlled trial research design to examine the effects of STEP in five sites located in four states. The sample included youth who had applied for STEP in 1986 and 1987 in five research sites. To be eligible, youth had to be 14 or 15 years old and from low-income families and had to have tested below grade level. The study included three cohorts of youth. The authors analyzed each of these cohorts separately. The Cohort 1 analysis did not meet WWC evidence standards because of high attrition and a failure to establish baseline equivalence. This analysis is therefore not included in this report. The Cohort 2 and Cohort 3 analyses meet evidence standards. Since these analyses were conducted in the same sites, the WWC treated these analyses as one study and combined the results when rating the effectiveness of STEP. The Cohort 2 analysis included 1,635 eligible youth who were randomly assigned to either a treatment group, which was offered the opportunity of participating in the STEP program during the summers of 1986 and 1987, or to a control group, which was offered a summer job in the federally funded Summer Youth Employment and Training Program (SYETP). The study lost 382 students to attrition or survey nonresponse, leaving an analysis sample of 1,253 youth. The Cohort 3 analysis included 1,591 eligible youth who were randomly assigned to either a treatment group, which was offered the opportunity of participating in the STEP program during the summers of 1987 and 1988, or to a control group, which was offered a summer job in the federally funded SYETP. The study lost 256 students to attrition or survey nonresponse, leaving an analysis sample of 1,335 youth. Across these two cohorts, slightly more than half of the study sample was female. About half of the sample was African-American, one-fifth was Asian, and one-fifth was Hispanic. All students came from low-income households, and about half of the sample came from female-headed households. About one-third of students reported having repeated a grade, and average test scores indicated that sample students were performing substantially below their grade level in math.
The STEP program focused on four areas: remediation, life skills and opportunities (LSO), work experience, and school-year support. During each of two summers, participants were offered approximately 90 hours of remedial instruction in basic reading and math skills, 18 hours of LSO instruction, and at least 80 hours of work experience. The remediation component provided a minimum of 90 hours of skill-based group and individually paced instruction. The LSO component stressed responsible social and sexual attitudes and behaviors. In the work experience component, participants were usually assigned to jobs near their remediation site. These jobs were typically in maintenance, recreation, clerical, and child care aide positions. During the intervening school year, program youth interacted with a designated counselor/advocate. These counselors helped monitor school attendance and performance of students and referred them to social services as needed.
The control group members were offered either a one-summer or two-summer job in the federally funded SYETP. The counterfactual condition varied across sites. In general, members of the control group were provided a part-time job during the first summer for which they were eligible for the program. In two sites, San Diego and Seattle, control group youth also were provided a job in the second summer. In general, control group youth spent more time working than did the treatment group youth, since treatment group youth were receiving remedial education and life skills training in addition to work. Overall numbers of hours engaged in study-related activity appear to be roughly the same for treatment and control group youth, with control group youth participating in employment for an average of 190 hours.
The relevant study outcomes included in this review are whether students dropped out of school and their highest grade completed, based on student follow-up interviews. For a more detailed description of these outcome measures, see Appendices A2.1 and A2.2.
Each site hired a lead teacher who was primarily responsible for overseeing the remediation component, hiring teachers, planning and providing teacher training, and assembling a curriculum development team of reading and math teachers to develop curriculum modules for site use. The lead teacher also assumed responsibility for seeing that life skills staff were supported and successfully integrated into program operations. Remediation teachers also received in-service training over the course of the summer program, an average of 4 to 6 hours total. In addition, before this cohort’s second summer of participation, the evaluator’s consultants and staff delivered preservice training to all remediation teachers at participating sites. STEP teachers received an average of between 15 and 20 hours of preservice training (the hours varied by site). In addition to specific topics outlined in the training guides, teacher training included an orientation to the program, discussions on the integration of the life skills and work experience components, and a review of modules and program logistics. For the life skills component, all life skills instructors attended a two-day comprehensive training workshop. This training included a review of curriculum content, an analysis of instructional techniques, a discussion of classroom management issues, and a series of role-plays and other sensitizing activities. They received minimal in-service training.
In the case of multiple manuscripts that report on one study, the WWC selects one manuscript as the primary citation and lists other manuscripts that describe the study as additional sources.
Sipe, C. L. (1988). Summer Training and Education Program (STEP): Report on the 1987 experience. Philadelphia, PA: Public/Private Ventures.
Branch, A. Y., Milliner, J. A., & Bumbaugh, J. (1986). Summer Training and Education Program (STEP): Report on the 1985 summer experience. Philadelphia, PA: Public/Private Ventures.
Sipe, C. L., Grossman, J. B., & Milliner, J. A. (1987). Summer Training and Education Program (STEP): Report on the 1986 experience. Philadelphia, PA: Public/Private Ventures.
An indicator of the effect of the intervention, the improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention.
For more, please see the WWC Glossary entry for improvement index.
An outcome is the knowledge, skills, and attitudes that are attained as a result of an activity. An outcome measures is an instrument, device, or method that provides data on the outcome.
A finding that is included in the effectiveness rating. Excluded findings may include subgroups and subscales.
The sample on which the analysis was conducted.
The group to which the intervention group is compared, which may include a different intervention, business as usual, or no services.
The timing of the post-intervention outcome measure.
The number of students included in the analysis.
The mean score of students in the intervention group.
The mean score of students in the comparison group.
The WWC considers a finding to be statistically significant if the likelihood that the finding is due to chance alone, rather than a real difference, is less than five percent.
The WWC reviews studies for WWC products, Department of Education grant competitions, and IES performance measures.
The name and version of the document used to guide the review of the study.
The version of the WWC design standards used to guide the review of the study.
The result of the WWC assessment of the study. The rating is based on the strength of evidence of the effectiveness of the intervention. Studies are given a rating of Meets WWC Design Standards without Reservations, Meets WWC Design Standards with Reservations, or >Does Not Meet WWC Design Standards.
A related publication that was reviewed alongside the main study of interest.
Study findings for this report.
Based on the direction, magnitude, and statistical significance of the findings within a domain, the WWC characterizes the findings from a study as one of the following: statistically significant positive effects, substantively important positive effects, indeterminate effects, substantively important negative effects, and statistically significant negative effects. For more, please see the WWC Handbook.
The WWC may review studies for multiple purposes, including different reports and re-reviews using updated standards. Each WWC review of this study is listed in the dropdown. Details on any review may be accessed by making a selection from the drop down list.
Tier 1 Strong indicates strong evidence of effectiveness,
Tier 2 Moderate indicates moderate evidence of effectiveness, and
Tier 3 Promising indicates promising evidence of effectiveness,
as defined in the
non-regulatory guidance for ESSA
and the regulations for ED discretionary grants (EDGAR Part 77).
Resources
Newsflash
Contact Us
Facebook
Twitter
YouTube