|Title:||Development of College- and Employer-based Career Pathways Models that Build on the Year Up Program Logic|
|Principal Investigator:||Fein, David||Awardee:||Abt Associates, Inc.|
|Program:||Postsecondary and Adult Education [Program Details]|
|Award Period:||4 years (07/01/2015-06/30/2019)||Award Amount:||$1,499,246|
|Type:||Development and Innovation||Award Number:||R305A150214|
Co-Principal Investigator: Maynard, Rebecca
Purpose: In this study, the researchers developed and piloted cost-effective, scalable versions of the Year Up's original "core" program for young adults aged 18–24 with a high school diploma or equivalent.
Project Activities: Researchers identified design and implementation challenges, developed and tried solutions, and revised the design of Year Up's PTC program to help it meet key performance goals. The team's approach included quick-turnaround studies of critical links in the program logic model. The research team used iterative, evidence-based improvement processes informed by substantial, authentic collaboration with practitioners. The study team used administrative and survey data to assess PTC's overall usability and implementation, implemented a small randomized controlled trial of programs at three local sites, and conducted a final round of in-depth interviews to assess program operations at the end of the grant period (mid-2019).
Key Outcomes: Key findings include:
Setting: The research team conducted two RCT tests, each involving three local PTC sites (with one site participating in both RCTs), in cities across the U.S. and gathered and analyzed data on the (then) full sample of fifteen sites nationally. The sites represent a mix of college environments and labor market conditions in urban and suburban settings.
Sample: The target population is low-income youth ages 18–24 with a high school degree or equivalent. Overall enrollment in the PTC programs increased from 500 to 1,926 between 2015 and 2018, chiefly due to expansion in the number of colleges. A majority of enrollees were males (59 percent) from racial or ethnic minority groups (e.g., 53 percent African American, 27 percent Hispanic). The first of two RCTs conducted for this study enrolled a total sample of 317 youth. The second RCT tested PTC's overall impacts in three sites (including one site participating in the earlier test), enrolling a total sample of 552 youth.
Intervention: The Professional Training Corps (PTC) is a lower-cost version of Year Up's original standalone "core" program. Briefly, PTC includes the same basic components.: (1) non-credit and credit bearing courses to build basic and professional competencies; (2) occupationally relevant work-based learning experiences provided through internships to build vocational competencies and links to the workforce; and (3) a high-support and high feedback culture created through behavior contracts and stipends, advising and life/career navigation, learning communities, and support services. The key modifications for PTC include operating on college campuses and using college instruction and facilities to reduce costs and reduced weekly stipends. Compared to the core program's $35,000 per participant cost, PTC targets costs of $20,000 per participant. Two of the improvements developed in this project address challenges in optimizing modifications for college settings—specifically, the increased challenge of monitoring academic outcomes and promoting longer-term college persistence. The third—increasing consistency in internship set-up—applies to both the core and PTC programs.
Research Design and Methods: Researchers worked with Year Up staff to prioritize and select focal problems for research and development. The development process employed iterative plan, do, study and act (PDSA) cycles. Program components that could be varied in response to identified focal problems included, for example, the length of time for the internship, the mode of course delivery, or the relative focus on different outcomes. The primary sources of data were document review, administrative data, telephone interviews, and targeted site visits. Once a model was ready for testing, the selected site recruited more applicants than could be accommodated, and then randomly selected applicants to the program or the comparison group. Data were collected on both groups using web-based surveys at 6-month intervals. A separate RCT is measuring the overall impacts of PTC (assigning eligible young adults to either a program or control group, with the latter not allowed to enroll in PTC). The project also makes extensive use of Year Up administrative data and stakeholder interviews to assess program implementation.
Control Condition: This project involved two RCTs. The first RCT (conducted during this project's development phase) tested improved academic monitoring and supports compared with usual PTC supports (which do not focus on academic outcomes). The second RCT (conducted in the pilot testing phase) compared outcomes for PTC compared with usual services available to young adults in the community.
Key Measures: Quantitative analyses drew on data from several administrative information systems. The two RCTs–conducted as part of the project's development and pilot testing work—utilized similar measures. These involved a series of college and employment-related outcomes, measured for successive calendar quarters following random assignment and drawing on records in the National Student Clearinghouse and National Directory of New Hires, respectively. Analyses of implementation fidelity involved a series of outcomes measured largely using Year Up administrative data and financial reports, as well as a small survey of local PTC offices conducted in mid-2019.
Data Analytic Strategy: Analyses for both RCTs involved comparing mean outcomes between randomly assigned groups, with the difference in means representing the estimated impacts. The analyses regression-adjust these differences in mean outcomes to improve precision. Analyses of implementation fidelity provided simple summary measures (e.g., means, percentages) to characterize performance outcomes.
Publications and Products
Fein, D., Maynard, R., Baelen, R, Shivji, A., & Souvanna, P. (2020). To Improve and to Prove: A Development and Innovation Study of Year Up's Professional Training Corps. Rockville, MD: Abt Associates Inc.
Maynard, R. A., Baelen, R. N., Fein, D., & Souvanna, P. (2020). Using Iterative Experimentation to Accelerate Program Improvement: A Case Example. Evaluation Review. https://doi.org/10.1177/0193841X20923199