Project Activities
To achieve its aims, the project team did the following:
- Used data from 14 RCTs of community college interventions to create empirical estimates of 2 key design parameters for calculating the minimum detectable true effect (MDTE) for a planned student-level blocked RCT: (1) the within-block outcome standard deviation and (2) the within-block outcome variance explained by baseline covariates like student characteristics. The cross-study distribution of parameter estimates was provided by outcome (enrollment, credits earned, credential attainment, and grade point average) and by semester.
- Used data from RCTs of 39 community college interventions to examine the distribution of effect sizes across these interventions. The cross-study distribution of effect sizes was estimated by outcome (enrollment, credits earned, and credential attainment) and by semester.
- Extended an existing software tool, known as The Generalizer, to aid applied community college researchers in identifying sites to be representative of a specified inference population.
Structured Abstract
Statistical/Methodological Product: The project team provided community college researchers with empirically based assumptions for power calculations highlights factors that researchers should consider when planning their studies. The project team also compared cross-study effect size distributions based on three estimators: ordinary least square (OLS) estimates, empirical Bayes estimates, and adjusted empirical Bayes estimates.
Development/Refinement Process: The project was based on data from more than 30 RCTs of community college interventions conducted by MDRC. The researchers cleaned and standardized the student-level data from these RCTs (including baseline data and outcomes data) across RCTs. For the empirical benchmarks analysis, they used a fixed intercept random coefficient (FIRC) model to estimate the mean and variance of the distribution of true effects across interventions and to derive adjusted empirical Bayes impact estimates for each intervention that are shrunken to account for estimation error. For the calculation of design parameters, the within-block outcome standard deviation and the within-block variance explained by baseline covariates were estimated for each study based on the residual outcome variance from an OLS regression model that controls for blocks alone, and then blocks and baseline characteristics. All analyses were conducted by outcome and by semester.
Key outcomes
Key findings and products from this project include the following:
- Design parameters for planning community college RCTs are reported in an open access journal (Somers, Weiss and Hill, 2022).
- Empirical benchmarks for contextualizing effect sizes in community college RCTs are reported in an open access journal (Weiss, Somers, and Hill, 2023; and a supplemental app is available for researchers to compare the effect size for their study (planned or completed) to the effect sizes from prior evaluations.
- A recruitment planning tool for generalizable community college RCTs is available to aid applied community college researchers in recruiting sites to be representative of a specified inference population (The Generalizer).
- A student-level database of community college RCTs used to estimate the empirical benchmarks and design parameters for this project is available at the Inter-university Consortium for Political and Social Research.
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
ERIC Citations: Find available citations in ERIC for this award here.
Publicly Available Data: MDRC's The Higher Education Randomized Controlled Trials Restricted Access File (THE-RCT RAF), United States, 2003–2019 (ICPSR 37932): https://www.icpsr.umich.edu/web/ICPSR/studies/37932
Additional Online Resources and Information:
- The Generalizer
- Lachanski, Michael, Michael Weiss, and Colin Hill. 2023. THE-RCT Empirical Benchmarks App. New York: MDRC. THE-RCT Empirical Benchmarks | MDRC
Select Publications:
Somers, M.-A., Weiss, M. J., & Hill, C. (2023). Design parameters for planning the sample size of individual-level randomized controlled trials in community colleges. Evaluation Review, 47(4), 599–629.
Weiss, M. J., Somers, M.-A., & Hill, C. (2023). Empirical benchmarks for planning and interpreting causal effects of community college interventions. Journal of Postsecondary Student Success, 3(1), 14–59.
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
