Advancing State-specific Design Parameters for Designing Better Evaluation Studies

Co-Principal Investigator: Eric Hedberg (NORC)

In a multilevel model, the statistical power, precision of estimates of treatment effects, the most efficient allocation of sample between levels, and the minimum detectable effect size all depend on the intraclass correlation structure and the effectiveness of any covariates in explaining variation at each level where they are used. In a randomized block design, researchers depend on the variation of treatment effects to draw causal conclusions about the effectiveness of an intervention. Accurate information about these design parameters is essential for well-designed experiments. Previous experiments and data from single districts provide estimates of design parameters that are typically imprecise and poorly matched to the current experiment. National surveys provide estimates of design parameters that are too general and not useful.

In this study, researchers seek to use entire state data systems to estimate intraclass correlations, R² values representing covariate effectiveness, and variance components representing heterogeneity of effects for use in designing multi-level studies in education. This project advances previously funded IES grant work (State-specific Design Parameters for Designing Better Evaluation Studies, R305D110032) in that access to comprehensive data from states allows for estimates of heterogeneity of important effects across district, school, and classes. This provides the research community with a better sense of the heterogeneity of program effects that might be reasonable for specific grades and subjects. An important product of the research will be a website that will expand on the current site developed under the previous grant. The new website will feature an interface that will allow users to select states and other parameters, such as research contexts, and receive tables of effect sizes and required sample sizes in return.

Products and Publications

Book

Hedberg, E. C. (2017). Introduction to Power Analysis: Two-group Studies (Vol. 176). SAGE Publications.

Journal article, monograph, or newsletter

Hedberg, E.C. (2016). Academic and Behavioral Design Parameters for Cluster Randomized Trials in Kindergarten: An Analysis of the Early Childhood Longitudinal Study 2011 Kindergarten Cohort (ECLS-K 2011). Evaluation Review, 40(4), 279–313.

Hedges, L.V. (2018). Challenges in Building Usable Knowledge in Education. Journal of Research on Educational Effectiveness, 11(1), 1–21.