Skip Navigation
Funding Opportunities | Search Funded Research Grants and Contracts

IES Grant

Title: State-specific Design Parameters for Designing Better Evaluation Studies
Center: NCER Year: 2011
Principal Investigator: Hedges, Larry Awardee: National Opinion Research Center (NORC)
Program: Statistical and Research Methodology in Education      [Program Details]
Award Period: 3 years Award Amount: $1,128,562
Type: Methodological Innovation Award Number: R305D110032
Description:

Co-Principal Investigator: Hedberg, Eric

Purpose: This project used five state data systems to compute design parameters (including intraclass correlations and variance accounted for by pretest data) at the state, district, school, and classroom levels. Additionally, meaningful subsets of the states defined geographically, in terms of achievement levels (e.g., low achieving schools or districts), and in terms of socioeconomic status (e.g., low SES schools or districts) were also collected.

Project Activities: Many educational experiments use designs that involve the random assignment of entire pre-existing groups (e.g., classrooms, schools, districts) to treatments. The groups typically used in education experiments are not themselves composed at random. As a result, individuals in the same group tend to be more alike than individuals in different groups implying a kind of dependence among individuals in groups called intraclass correlation (ICC). As these experiments include multiple levels of clustering (e.g., students in classrooms, classrooms in schools, schools in districts), they have multiple ICCs (one for each cluster). The statistical power, precision of estimates for treatment effects, efficient allocation of sample between levels, and the minimum detectable effect size all depend on the intraclass correlation structure and (if covariates are used) the effectiveness of the covariates at explaining variation at each of the levels that they are used. ICCs obtained from single district data may be too imprecise or too poorly matched to a current experiment to provide adequate guidance and those obtained from national surveys may be too general to be optimally useful. Using mixed linear models, the project decomposed the total variation of state achievement test scores (in reading and math) into district, school, classroom, and student level variance components. These variance components were then used to estimate ICCs at the district, school, and classroom levels.

The project estimated design parameters for students in particular grades in each state subdivided by three dimensions. The first dimension is an outcome metric that will include achievement status (achievement at a single point in time) and growth (the difference between achievement in a particular year and those in a previous year) in reading and in mathematics in a particular grade. The second dimension involves analyses involving covariates and will focus on four models: (1) no covariates; (2) demographic covariates; (3) pre-test covariates; and (4) combinations of demographic and pretest covariates. The third dimension involves how design parameters differ across different school contexts with a focus on low-performing schools, schools serving low-income populations, or schools with large minority populations. A secondary objective of the project was to use the longitudinal state data to estimate single year increases in achievement in effect-size units for particular grades and subjects. This included estimating effect sizes for participating states controlling for each of the covariate sets and for the different school contexts.

Products and Publications

Journal article, monograph, or newsletter

Hedberg, E.C., and Hedges, L.V. (2014). Reference Values of Within-District Intraclass Correlations of Academic Achievement by District Characteristics: Results From a Meta-Analysis of District-Specific Values. Evaluation Review, 38(6), 546–582.

Hedges, L.V., and Hedberg, E.C. (2013). Intraclass Correlations and Covariate Outcome Correlations for Planning Two- and Three-Level Cluster-Randomized Experiments in Education. Evaluation Review, 37(6), 445–489.

Hedges, L.V., Hedberg, E.C., and Kuyper, A.M. (2012). The Variance of Intraclass Correlations in Three- and Four-Level Models. Educational and Psychological Measurement, 72(6): 893–909.


Back