Skip Navigation
Funding Opportunities | Search Funded Research Grants and Contracts

IES Grant

Title: Using Imperfect Fidelity Measures to Improve Statistical Inferences about Educational Interventions
Center: NCER Year: 2009
Principal Investigator: Stokes, S. Lynne Awardee: Southern Methodist University
Program: Statistical and Research Methodology in Education      [Program Details]
Award Period: 2 years Award Amount: $431,823
Type: Methodological Innovation Award Number: R305D090020
Description:

Co-Principal Investigators: Allor, Jill; Harris, Ian

Purpose: The purpose of this project was to develop statistical methods to conduct inference for impact studies of education interventions that take into account the variability of intervention fidelity in the experimental treatment group.

The inclusion of fidelity of implementation measures in evaluations of education programs and policies can both increase the power of a study to detect an intervention's effects and allow researchers to determine whether a failure to find impacts stems from the ineffectiveness of the intervention or its implementation. Because most measures of fidelity are either made from occasional observations or are based on subjective assessment, they may be vulnerable to measurement error. If fidelity measures contain measurement error, then analyses will not provide unbiased estimators of the relationship between fidelity and outcomes. Instead, the regression coefficient estimates become biased and there is a loss of power for detecting a relationship between fidelity and outcome. A noisy measure of the explanatory variable in simple linear regression causes the estimated regression coefficient to be attenuated, or biased toward zero, so that any effect looks smaller than it actually is. In multiple regression, the bias can be in either direction depending on the relationship between the noisy variable and others in the model. Power for detecting significance of regression coefficients is also reduced because the noise in the explanatory variable blurs any relationships between explanatory and response variable that might exist.

Project Activities: The project was composed of three components. The primary component was the development of regression calibration estimators of regression coefficients of noisy covariates in multilevel linear and logistic regression for four measurement error models (as well as the classical unbiased, homogeneous variance, additive model): (1) biased additive models with heterogeneous variance; (2) models including observer variance components; (3) mixture of zero mass and classical error model; and (4) Berkson error models. The models considered will allow for fidelity measurements made at level 2 (classroom) and, in some cases, at level 1 (student). The models will allow for additional covariates at both levels that do not contain errors.

The second component used two ongoing longitudinal quasi-experimental studies on similar interventions to teach basic reading skills to K–3 students who are English language learners or children with mild to moderate mental disabilities. The datasets will have replicate and/or validate data for selected fidelity measures along with individual student standardized test scores. Measurement error models will be estimated using a selection of fidelity measures. Following this, the appropriate adjusted estimation procedures developed under the project will be applied and estimates of the relationship between outcome(s) and fidelity will be compared between models using the adjustments and those without them.

The third component was the development of improved sample designs for collecting fidelity data to enable efficient measurement error model estimation. The project investigated the optimal design of fidelity measure data collection drawing on methods from the literature on surveys. The goal for this work was to make practical recommendations to researchers about the number of fidelity measurements needed to achieve adequate precision in the estimation of measurement error.

Publications and Products

Publications

Journal article, monograph, or newsletter

Stokes, L., and Allor, J.H. (2016). A Power Analysis for Fidelity Measurement Sample Size Determination. Psychological Methods, 21(1), 35.


Back