|Title:||The Higher Education Randomized Controlled Trial (THE-RCT) Project: Synthesizing Evidence from 15+ Years of RCTs in Postsecondary Education|
|Principal Investigator:||Weiss, Michael||Awardee:||MDRC|
|Program:||Postsecondary and Adult Education [Program Details]|
|Award Period:||2 years (07/01/2019 - 06/30/2021)||Award Amount:||$599,698|
Purpose: This project will synthesize evidence from 15+ years of postsecondary randomized controlled trials (RCTs) conducted by MDRC, to draw conclusions about effective intervention strategies for boosting rates of college success. By drawing on detailed data from 25 large-scale RCTs conducted in postsecondary settings, this project will: (1) explore the relationship between malleable factors (for example, financial incentives, enhanced advising, academic supports) and the magnitude of program effects on key postsecondary outcomes; (2) examine what happens to program effects once a program is over; and (3) create and make publicly available a comprehensive, student-level database including baseline and academic outcomes data from the 25 component evaluations.
Project Activities: During the first year of the project, researchers will assemble the comprehensive database. They will align outcome measures across the component studies; recode the intervention measures for the intensity of malleable factors; and merge the 25 component studies into a unified database for analysis. During year two, the team will carry out its research agenda for all malleable factors included in the study, disseminate its findings to researchers and practitioners, and make the project dataset available to outside researchers on a restricted-use basis.
Products: The team will generate rigorous evidence on the effectiveness of common components within postsecondary intervention strategies. The team will produce a synthesis of its findings for a peer-reviewed publication, and an online summary of its findings for policymakers and practitioners; the team will also host a colloquium to share its findings.
Setting: The studies took place at broad-access postsecondary institutions (mostly community colleges) throughout the United States.
Sample: The sample includes over 55,000 students from over 50 colleges in 25 postsecondary education experiments conducted by MDRC since 2000; each experiment contributes at least 300 students to the sample. The sample includes low-income individuals, first-time college students, and students in need of remediation.
Intervention: The meta-analysis comprises all strategies included in MDRC-conducted RCTs over the last 15 years, including enhanced advising, financial supports, learning communities, academic supports, student-centered pedagogy, and comprehensive interventions.
Research Design and Methods: This project is a secondary meta-analysis of individual-level data from a series of multi-site RCTs. The design builds on two path-breaking methodological strategies for analyzing data from multiple evaluations: (1) the multi-site random-effects model developed by Bloom, Hill and Riccio (2003) for their secondary meta-analysis of data from MDRC's welfare/work experiments; and (2) the fixed-intercept, random-coefficient (FIRC) model developed by Bloom, Raudenbush, Weiss & Porter (2017) for studying cross-site variation in program effects. The intent of the design is to find the crucial components of intervention strategies and precisely assess their effects on key postsecondary outcomes.
Control Condition: Most control group students in the component studies were offered all their college's business-as-usual services, but not the program services.
Key Measures: Analyses will include a full set of postsecondary academic outcomes including enrollment, persistence in college, GPA, credits earned, and degree or certificate completion. The analyses will include these main intervention components: financial support, success courses, learning communities, pedagogical reforms, curricular reforms, enhanced advising, and enhanced tutoring. Given enough data within intervention types, the researchers will create continuous measures of the intervention components (the malleable factors).
Data Analytic Strategy: Researchers will use a two-level statistical model with students at level one and research sites (colleges) at level two. Fixed site-specific intercepts at level one will ensure that all treatment and control group outcome comparisons take place within sites. Interaction terms at level one for treatment assignment-by-student characteristics will facilitate cross-group comparison of program component effects. Level two will include the malleable factors, which vary at the site level and aid estimation of the key parameter estimates for the effectiveness of program components.
Related IES Projects: The Learning Communities evaluation within the National Center for Postsecondary Research (R305A060010); Evaluating the Long-Term Effects and the Costs of Two Community College Interventions (R305A100066); Performance-Based Scholarship Demonstration — An Alternative Financial Aid Program to Incentivize Academic Success (R305A110204); Using Computer-Assisted Instruction to Accelerate Students through Developmental Math: An Impact Study of Modularization and Compression (R305A130125); Evaluating the Impact of CUNY Start through a Researcher (MDRC) – Local Education Agency (City University of New York) Partnership (R305H140065); Assessing the Long-Term Efficacy and Costs of the City University of New York's (CUNY'S) Accelerated Study in Associate Programs (ASAP) (R305A160273);