Grant Closed

Using Imperfect Fidelity Measures to Improve Statistical Inferences about Educational Interventions

NCER

Program: Statistical and Research Methodology in Education

Program topic(s): Core

Award amount: $431,823

Principal investigator: S. Lynne Stokes

Awardee:

Southern Methodist University

Year: 2009

Award period: 3 years (06/01/2009 - 05/31/2012)

Project type:

Methodological Innovation

Award number: R305D090020

Purpose

The purpose of this project was to develop statistical methods to conduct inference for impact studies of education interventions that take into account the variability of intervention fidelity in the experimental treatment group.

The inclusion of fidelity of implementation measures in evaluations of education programs and policies can both increase the power of a study to detect an intervention's effects and allow researchers to determine whether a failure to find impacts stems from the ineffectiveness of the intervention or its implementation. Because most measures of fidelity are either made from occasional observations or are based on subjective assessment, they may be vulnerable to measurement error. If fidelity measures contain measurement error, then analyses will not provide unbiased estimators of the relationship between fidelity and outcomes. Instead, the regression coefficient estimates become biased and there is a loss of power for detecting a relationship between fidelity and outcome. A noisy measure of the explanatory variable in simple linear regression causes the estimated regression coefficient to be attenuated, or biased toward zero, so that any effect looks smaller than it actually is. In multiple regression, the bias can be in either direction depending on the relationship between the noisy variable and others in the model. Power for detecting significance of regression coefficients is also reduced because the noise in the explanatory variable blurs any relationships between explanatory and response variable that might exist.

Project Activities

The project was composed of three components. The primary component was the development of regression calibration estimators of regression coefficients of noisy covariates in multilevel linear and logistic regression for four measurement error models (as well as the classical unbiased, homogeneous variance, additive model): (1) biased additive models with heterogeneous variance; (2) models including observer variance components; (3) mixture of zero mass and classical error model; and (4) Berkson error models. The models considered will allow for fidelity measurements made at level 2 (classroom) and, in some cases, at level 1 (student). The models will allow for additional covariates at both levels that do not contain errors.

The second component used two ongoing longitudinal quasi-experimental studies on similar interventions to teach basic reading skills to K–3 students who are English language learners or children with mild to moderate mental disabilities. The datasets will have replicate and/or validate data for selected fidelity measures along with individual student standardized test scores. Measurement error models will be estimated using a selection of fidelity measures. Following this, the appropriate adjusted estimation procedures developed under the project will be applied and estimates of the relationship between outcome(s) and fidelity will be compared between models using the adjustments and those without them.

The third component was the development of improved sample designs for collecting fidelity data to enable efficient measurement error model estimation. The project investigated the optimal design of fidelity measure data collection drawing on methods from the literature on surveys. The goal for this work was to make practical recommendations to researchers about the number of fidelity measurements needed to achieve adequate precision in the estimation of measurement error.