Using Imperfect Fidelity Measures to Improve Statistical Inferences about Educational Interventions

Co-Principal Investigators: Allor, Jill; Harris, Ian

The inclusion of fidelity of implementation measures in evaluations of education programs and policies can both increase the power of a study to detect an intervention's effects and allow researchers to determine whether a failure to find impacts stems from the ineffectiveness of the intervention or its implementation. Because most measures of fidelity are either made from occasional observations or are based on subjective assessment, they may be vulnerable to measurement error. If fidelity measures contain measurement error, then analyses will not provide unbiased estimators of the relationship between fidelity and outcomes. Instead, the regression coefficient estimates become biased and there is a loss of power for detecting a relationship between fidelity and outcome. A noisy measure of the explanatory variable in simple linear regression causes the estimated regression coefficient to be attenuated, or biased toward zero, so that any effect looks smaller than it actually is. In multiple regression, the bias can be in either direction depending on the relationship between the noisy variable and others in the model. Power for detecting significance of regression coefficients is also reduced because the noise in the explanatory variable blurs any relationships between explanatory and response variable that might exist.

The second component used two ongoing longitudinal quasi-experimental studies on similar interventions to teach basic reading skills to K–3 students who are English language learners or children with mild to moderate mental disabilities. The datasets will have replicate and/or validate data for selected fidelity measures along with individual student standardized test scores. Measurement error models will be estimated using a selection of fidelity measures. Following this, the appropriate adjusted estimation procedures developed under the project will be applied and estimates of the relationship between outcome(s) and fidelity will be compared between models using the adjustments and those without them.

The third component was the development of improved sample designs for collecting fidelity data to enable efficient measurement error model estimation. The project investigated the optimal design of fidelity measure data collection drawing on methods from the literature on surveys. The goal for this work was to make practical recommendations to researchers about the number of fidelity measurements needed to achieve adequate precision in the estimation of measurement error.