Principal Investigator: Weiss, Michael J.

Purpose: This project synthesized evidence from 15+ years of postsecondary randomized controlled trials (RCTs) conducted by MDRC to draw conclusions about effective intervention strategies for boosting rates of college success. Drawing on detailed data from 30 large-scale postsecondary RCTs of 39 interventions, including 65,000 students, this project

• explored the predictive relationship between intervention features (for example, increased financial support, increased advising, learning communities) and the magnitude of program effects on postsecondary outcomes
• examined what happens to program effects on credit accumulation once a program is over (e.g., do effects grow, fade-out, or are they maintained
• created a large individual-participant database from higher education RCTs, known as The Higher Education Randomized Controlled Trials (THE-RCT) Restricted Access File (RAF), which is available to researchers

Project Activities: First, the research team assembled the comprehensive database, including aligning outcome measures across the component studies, coding the features of each intervention, and merging the 30 studies into a unified database for analyses. Next, the research team carried out the research agenda (goals 1 and 2 above), disseminating findings to researchers, practitioners, and funders. Finally, the research team made the project dataset available to outside researchers on a restricted-use (for legal/IRB reasons) basis.

Key Outcomes:

Findings reported in Weiss, Bloom, and Singh (2022) indicate that the impacts of community college interventions on credit accumulation during the first two semesters of college and persistence to the third semester of college increase with

• the comprehensiveness of the intervention, as measured by its number of intervention components
• promotion of full-time enrollment (during fall and spring) and summer enrollment

Structured Abstract

Setting: The studies took place at over 45 broad-access postsecondary institutions (mostly community colleges) throughout the United States.

Sample: The sample includes over 60,000 students in 30 postsecondary education experiments conducted by MDRC from 2003 to 2019. The sample includes low-income individuals, first-time college students, and students in need of remediation.

Factors: The synthesis examined predictive relationships between intervention features and intervention impacts for community college students. Features included increased financial supports, increased advising, increased tutoring, learning communities, student success courses, promoting full-time and summer enrollment, and instructional reforms. In addition, the research team examined the relationship between intervention comprehensiveness and impacts.

Research Design and Methods: This project was a secondary analysis of individual-level data from a series of multi-site RCTs. The design built on two methodological strategies for analyzing data from multiple evaluations: (1) the multi-site random-effects model developed by Bloom, Hill, and Riccio (2003) for their secondary synthesis of data from MDRC's welfare/work experiments; and (2) the fixed-intercept, random-coefficient (FIRC) model developed by Bloom, Raudenbush, Weiss, & Porter (2017) for studying cross-site variation in program effects. The intent of the design is to find the crucial components of intervention strategies and assess their predictive relationship to impacts on postsecondary outcomes.

Control Condition: Most control group students in the component studies were offered all their college's business-as-usual services but not the program services.

Key Measures: Analyses focused on a parsimonious set of postsecondary academic outcomes including enrollment and credit accumulation, maximizing the number of studies and sample members included in the analyses.

Data Analytic Strategy: The research team used 2-level random-effects models to quantify and predict impact variation across the 39 interventions in THE-RCT database (students at level 1, studied interventions at level 2). This model first supports estimation of how much variation in impacts exists across the 39 interventions. Next, this model supports estimation of the relationship between the intensity of a given feature of an intervention (e.g., increased financial support) and the impact of the intervention on a given student outcome (e.g., credit accumulation).

Related IES Projects: The Learning Communities evaluation within the National Center for Postsecondary Research (R305A060010), Evaluating the Long-Term Effects and the Costs of Two Community College Interventions (R305A100066), Performance-Based Scholarship Demonstration — An Alternative Financial Aid Program to Incentivize Academic Success (R305A110204), Using Computer-Assisted Instruction to Accelerate Students through Developmental Math: An Impact Study of Modularization and Compression (R305A130125), Evaluating the Impact of CUNY Start through a Researcher (MDRC) – Local Education Agency (City University of New York) Partnership (R305H140065), Assessing the Long-Term Efficacy and Costs of the City University of New York's (CUNY'S) Accelerated Study in Associate Programs (ASAP) (R305A160273), An Empirical Analysis of Two Methodological Issues for Education Research Caused by Variation in Program Impacts (R305D140012), An Intervention Return on Investment (ROI) Tool for Community Colleges (R305U200007), Planning Randomized Controlled Trials in Community Colleges (R305D190025)

Products and Publications

ERIC Citations: Find available citations in ERIC for this award here. Publicly Available Data: https://www.icpsr.umich.edu/web/ICPSR/studies/37932

Project Website: https://www.mdrc.org/work/projects/higher-education-randomized-controlled-trial. Additional Online Resources and Information: This grant also produced a unique and valuable set of Open Science resources:

• Student-level data, a study-level database, a codebook, and a user guide are available through the Inter-university Consortium for Political and Social Research (ICPSR): https://www.icpsr.umich.edu/web/ICPSR/studies/37932#.

Select Publications:

Journal articles Weiss, M. J., Bloom, H. S., & Singh, K. (2022). What 20 Years of MDRC RCTs Suggest About Predictive Relationships Between Intervention Features and Intervention Impacts for Community College Students. Educational Evaluation and Policy Analysis. doi:10.3102/01623737221139493