Co-Principal Investigators: Tipton, Elizabeth; Somers, Marie-Andree

Purpose: The project team developed products to support community college researchers with the same type of empirically based benchmarks, design parameters, and tools currently available for K–12 researchers planning and interpreting randomized controlled trials (RCTs).

Project Activities: To achieve its aims, the project team did the following:

• Used data from 14 RCTs of community college interventions to create empirical estimates of 2 key design parameters for calculating the minimum detectable true effect (MDTE) for a planned student-level blocked RCT: (1) the within-block outcome standard deviation and (2) the within-block outcome variance explained by baseline covariates like student characteristics. The cross-study distribution of parameter estimates was provided by outcome (enrollment, credits earned, credential attainment, and grade point average) and by semester.
• Used data from RCTs of 39 community college interventions to examine the distribution of effect sizes across these interventions. The cross-study distribution of effect sizes was estimated by outcome (enrollment, credits earned, and credential attainment) and by semester.
• Extended an existing software tool, known as The Generalizer, to aid applied community college researchers in identifying sites to be representative of a specified inference population.

Key Outcomes: Key findings and products from this project include the following:

• Design parameters for planning community college RCTs are reported in an open access journal (Somers, Weiss and Hill, 2022).
• Empirical benchmarks for contextualizing effect sizes in community college RCTs are reported in an open access journal (Weiss, Somers, and Hill, 2023; and a supplemental app is available for researchers to compare the effect size for their study (planned or completed) to the effect sizes from prior evaluations.
• A recruitment planning tool for generalizable community college RCTs is available to aid applied community college researchers in recruiting sites to be representative of a specified inference population (The Generalizer).
• A student-level database of community college RCTs used to estimate the empirical benchmarks and design parameters for this project is available at the Inter-university Consortium for Political and Social Research.

Structured Abstract

Statistical/Methodological Product: The project team provided community college researchers with empirically based assumptions for power calculations highlights factors that researchers should consider when planning their studies. The project team also compared cross-study effect size distributions based on three estimators: ordinary least square (OLS) estimates, empirical Bayes estimates, and adjusted empirical Bayes estimates.

Development/Refinement Process: The project was based on data from more than 30 RCTs of community college interventions conducted by MDRC. The researchers cleaned and standardized the student-level data from these RCTs (including baseline data and outcomes data) across RCTs. For the empirical benchmarks analysis, they used a fixed intercept random coefficient (FIRC) model to estimate the mean and variance of the distribution of true effects across interventions and to derive adjusted empirical Bayes impact estimates for each intervention that are shrunken to account for estimation error. For the calculation of design parameters, the within-block outcome standard deviation and the within-block variance explained by baseline covariates were estimated for each study based on the residual outcome variance from an OLS regression model that controls for blocks alone, and then blocks and baseline characteristics. All analyses were conducted by outcome and by semester.

Products and Publications

ERIC Citations: Find available citations in ERIC for this award here.

Publicly Available Data: MDRC's The Higher Education Randomized Controlled Trials Restricted Access File (THE-RCT RAF), United States, 2003–2019 (ICPSR 37932): https://www.icpsr.umich.edu/web/ICPSR/studies/37932

Additional Online Resources and Information:

• The Generalizer
• Lachanski, Michael, Michael Weiss, and Colin Hill. 2023. THE-RCT Empirical Benchmarks App. New York: MDRC. THE-RCT Empirical Benchmarks | MDRC

Select Publications:

Weiss, M. J., Somers, M.-A., & Hill, C. (2023). Empirical benchmarks for planning and interpreting causal effects of community college interventions. Journal of Postsecondary Student Success, 3(1), 14–59.