Purpose: This project will evaluate the SUCCESS intervention, which provides evidence-based, mutually reinforcing support services to help community college students stay on track through graduation at an affordable cost for colleges. The intervention design seeks to address two problems that have emerged in the postsecondary research literature: (1) very few interventions have causal evidence of substantially increasing graduation rates at community colleges; and (2) the few interventions with such causal evidence tend to be comprehensive, and expensive. These problems limit widespread adoption and sustainability of completion-increasing interventions.

The project will evaluate SUCCESS's effectiveness overall, for students that entered the program during the COVID-19 pandemic, and for cohorts that enter the program  after the COVID-19 pandemic. Researchers will also assess the intervention's effectiveness for relevant subpopulations of students, variation in its impact across colleges, and measure its implementation, including fidelity and service contrast, and cost, including cost-effectiveness and revenue generated.

Project Activities: The research team will carry out an RCT evaluation of SUCCESS at 10 colleges including over 4,000 students. They will work with the participating colleges to implement the intervention for three years, and will track academic outcomes including persistence, credit accumulation, and degree completion during this period. The team will also conduct implementation research to document the intervention's implementation during and after the COVID-19 pandemic, fidelity to the model, and service contrast. They will use finding from the implementation research to interpret effect estimates, facilitate continuous improvement of the intervention, and informing its future scaling. Cost analyses will be conducted to estimate the total cost and cost-effectiveness of the program, as well as inform questions about the financial sustainability and scalability of SUCCESS.

Products: The SUCCESS RCT will produce three published reports and a peer-reviewed journal article detailing the impact, implementation, and cost findings. MDRC will also produce five state-specific 2-page memos for policymakers in the participating states. Release of the memos will be timed to incorporate one-year academic outcomes for the study cohorts and will include early considerations surrounding scaling and expansion in each state. MDRC will create a restricted access file in the final year of the grant. The findings will be disseminated at conferences and presentations.

Structured Abstract

Setting: The research will take place in 10 diverse community colleges operating in five states (California, Indiana, Minnesota, New Jersey, and Ohio). The setting includes large and small colleges and urban, suburban, and rural settings.

Population/Sample: The sample will include over 4,000 students. The target population is low-income students who are willing to enroll full time, seek a degree or certificate, and new or continuing with limited credits earned. Four of the colleges are minority serving institutions, and several of the colleges run programs that exclusively serve students of color or first-generation students.

Intervention/Factors/Assessment: SUCCESS is a multifaceted, integrated three-year program with four core components: (1) proactive coaching: coaches reach out actively to students and meet with them frequently; (2) enrollment intensity messaging: students receive messaging to enroll full time and are strongly encouraged to enroll in summer; (3) monthly financial incentives: financial incentives encourage students to meet with coaches and to meet enrollment intensity benchmarks; they also reduce financial burden; and (4) management information systems: MDRC works with the colleges to set up systems that provide real-time data to support efficient program management for staff and student accountability, and for continuous improvement.

Research Design and Methods: A student-level random assignment design will be used to estimate the causal effects of SUCCESS. Over the course of the project, over 4,000 students will be assigned to a program group or a control group during the class registration process. The sample size will enable very precise impact estimates for the full sample, sufficiently precise subgroup impact estimates for key subgroups, and estimation of how much the impacts vary across colleges.

Control Condition: Control group members are eligible to receive all their college's standard services, but not the enhanced services available to students in SUCCESS.

Key Measures: Measures of academic progress and completion, including enrollment, credit accumulation, transfer, and degree or certificate completion will be tracked for three years after students enter the study. Data will be obtained through college transcript records and the National Student Clearinghouse. Measures of implementation fidelity and the service contrast will be obtained through a student survey, the program's MIS, and interviews and focus groups with students and staff. Cost data will come from budgets and staff interviews.

Data Analytic Strategy: Researchers will compute intent-to-treat estimates by comparing average outcomes of the program group and control group members using standard statistical tests and multivariate regression. Subgroup analyses will be conducted based on students' race, college, and status as a traditional or non-traditional student. A key confirmatory subgroup will be whether a student joined the study during the COVID-19 pandemic or afterward, given the pandemic's influence on relevant context (e.g., students taking class online instead of in-person) and SUCCESS's services (e.g., virtual advising rather than primarily in-person advising).

Cost Analysis: The net cost per student of SUCCESS will be estimated from the societal perspective. Costs will be disaggregated for multiple perspectives, with emphasis placed on the college perspective. To understand cost effectiveness, the cost per additional confirmatory outcome will be calculated and compared to other postsecondary interventions.

Related IES Projects: Assessing the Long-Term Efficacy and Costs of the City University of New York's (CUNY'S) Accelerated Study in Associate Programs (ASAP) (R305A160273)