State-specific Design Parameters for Designing Better Evaluation Studies

Many educational experiments use designs that involve the random assignment of entire pre-existing groups (e.g., classrooms, schools, districts) to treatments. The groups typically used in education experiments are not themselves composed at random. As a result, individuals in the same group tend to be more alike than individuals in different groups implying a kind of dependence among individuals in groups called intraclass correlation (ICC). As these experiments include multiple levels of clustering (e.g., students in classrooms, classrooms in schools, schools in districts), they have multiple ICCs (one for each cluster). The statistical power, precision of estimates for treatment effects, efficient allocation of sample between levels, and the minimum detectable effect size all depend on the intraclass correlation structure and (if covariates are used) the effectiveness of the covariates at explaining variation at each of the levels that they are used. ICCs obtained from single district data may be too imprecise or too poorly matched to a current experiment to provide adequate guidance and those obtained from national surveys may be too general to be optimally useful. Using mixed linear models, the project decomposed the total variation of state achievement test scores (in reading and math) into district, school, classroom, and student level variance components. These variance components were then used to estimate ICCs at the district, school, and classroom levels.

The project estimated design parameters for students in particular grades in each state subdivided by three dimensions. The first dimension is an outcome metric that will include achievement status (achievement at a single point in time) and growth (the difference between achievement in a particular year and those in a previous year) in reading and in mathematics in a particular grade. The second dimension involves analyses involving covariates and will focus on four models: (1) no covariates; (2) demographic covariates; (3) pre-test covariates; and (4) combinations of demographic and pretest covariates. The third dimension involves how design parameters differ across different school contexts with a focus on low-performing schools, schools serving low-income populations, or schools with large minority populations. A secondary objective of the project was to use the longitudinal state data to estimate single year increases in achievement in effect-size units for particular grades and subjects. This included estimating effect sizes for participating states controlling for each of the covariate sets and for the different school contexts.