State-specific Design Parameters for Designing Better Evaluation Studies

Many educational experiments use designs that involve the random assignment of entire pre-existing groups (e.g., classrooms, schools, districts) to treatments. The groups typically used in education experiments are not themselves composed at random. As a result, individuals in the same group tend to be more alike than individuals in different groups implying a kind of dependence among individuals in groups called intraclass correlation (ICC). As these experiments include multiple levels of clustering (e.g., students in classrooms, classrooms in schools, schools in districts), they have multiple ICCs (one for each cluster). The statistical power, precision of estimates for treatment effects, efficient allocation of sample between levels, and the minimum detectable effect size all depend on the intraclass correlation structure and (if covariates are used) the effectiveness of the covariates at explaining variation at each of the levels that they are used. ICCs obtained from single district data may be too imprecise or too poorly matched to a current experiment to provide adequate guidance and those obtained from national surveys may be too general to be optimally useful. Using mixed linear models, the project decomposed the total variation of state achievement test scores (in reading and math) into district, school, classroom, and student level variance components. These variance components were then used to estimate ICCs at the district, school, and classroom levels.