|Title:||State-specific Design Parameters for Designing Better Evaluation Studies|
|Principal Investigator:||Hedges, Larry||Awardee:||National Opinion Research Center (NORC)|
|Program:||Statistical and Research Methodology in Education [Program Details]|
|Award Period:||3 years||Award Amount:||$1,128,562|
|Goal:||Methodological Innovation||Award Number:||R305D110032|
Co-Principal Investigator: Eric Hedberg
This project will use five state data systems to compute design parameters (including intraclass correlations and variance accounted for by pretest data) at the state, district, school, and classroom levels. Additionally, meaningful subsets of the states defined geographically, in terms of achievement levels (e.g., low achieving schools or districts), and in terms of socioeconomic status (e.g., low SES schools or districts) will also be collected.
Many educational experiments use designs that involve the random assignment of entire pre-existing groups (e.g., classrooms, schools, districts) to treatments. The groups typically used in education experiments are not themselves composed at random. As a result, individuals in the same group tend to be more alike than individuals in different groups implying a kind of dependence among individuals in groups called intraclass correlation (ICC). As these experiments include multiple levels of clustering (e.g., students in classrooms, classrooms in schools, schools in districts), they have multiple ICCs (one for each cluster). The statistical power, precision of estimates for treatment effects, efficient allocation of sample between levels, and the minimum detectable effect size all depend on the intraclass correlation structure and (if covariates are used) the effectiveness of the covariates at explaining variation at each of the levels that they are used. ICCs obtained from single district data may be too imprecise or too poorly matched to a current experiment to provide adequate guidance and those obtained from national surveys may be too general to be optimally useful. Using mixed linear models, the project will decompose the total variation of state achievement test scores (in reading and math) into district, school, classroom, and student level variance components. These variance components will then be used to estimate ICCs at the district, school, and classroom levels.
The project will estimate design parameters for students in particular grades in each state subdivided by three dimensions. The first dimension is an outcome metric that will include achievement status (achievement at a single point in time) and growth (the difference between achievement in a particular year and those in a previous year) in reading and in mathematics in a particular grade. The second dimension involves analyses involving covariates and will focus on four models: (1) no covariates; (2) demographic covariates; (3) pre-test covariates; and (4) combinations of demographic and pretest covariates. The third dimension involves how design parameters differ across different school contexts with a focus on low-performing schools, schools serving low income populations, or schools with large minority populations. A secondary objective of the project will be to use the longitudinal state data to estimate single year increases in achievement in effect-size units for particular grades and subjects. This will include estimating effect sizes for participating states controlling for each of the covariate sets and for the different school contexts.
Journal article, monograph, or newsletter
Hedberg, E.C., and Hedges, L.V. (2014). Reference Values of Within-District Intraclass Correlations of Academic Achievement by District Characteristics: Results From a Meta-Analysis of District-Specific Values. Evaluation Review, 38(6), 546–582.
Hedges, L.V., and Hedberg, E.C. (2013). Intraclass Correlations and Covariate Outcome Correlations for Planning Two- and Three-Level Cluster-Randomized Experiments in Education. Evaluation Review, 37(6), 445–489.
Hedges, L.V., Hedberg, E.C., and Kuyper, A.M. (2012). The Variance of Intraclass Correlations in Three- and Four-Level Models. Educational and Psychological Measurement, 72(6): 893–909.