|Title:||Assessing Intervention Fidelity in Randomized Field Experiments (RFTs)|
|Principal Investigator:||Cordray, David S.||Awardee:||Vanderbilt University|
|Program:||Statistical and Research Methodology in Education [Program Details]|
|Award Period:||3 years||Award Amount:||$1,495,133|
|Type:||Methodological Innovation||Award Number:||R305U060002|
Purpose: This project seeks to develop a framework to identify all the components of an education intervention and gauge both the degree and extent to which each component is in place and operating as intended. To this end, the project will develop, modify, test and combine (a) intervention theory and structure with (b) measures of intervention delivery and receipt, and (c) methods for making use of the measures of intervention fidelity in analysis in order to better understand the effects of the intervention on the outcome measures.
Project Activities: The research team will use three general sources of information: (1) the literature on assessing intervention fidelity and on constructing measures of the reliability and validity of constructs such as fidelity of implementation; (2) three existing research studies that are sufficiently documented to permit reanalysis and evaluation of some fidelity measures; and (3) an on-going empirical study of fidelity embedded in an existing randomized field experiment (RFT). Reanalysis of the three existing data sets is being carried out in cooperation with the original research teams.
The existing projects are: (1) the Preschool Curriculum Evaluation Research (PCER) project; (2) the Early Reading First (ERF) program, Wayne County, TN; and (3) the Chicago Public Schools Striving Readers (CPS-SR) program. The ongoing project is Nashville's Technology-enhanced, Research-based, Instruction, Assessment, and professional Development project (TRIAD).
Products: The products of this project will include a fully developed framework for measuring the fidelity of implementation of intervention and methods for incorporating fidelity measures in the analyses of RFTs. In addition, the project will produce a collection of specific measures and methods including examples of theoretical and corresponding operational measures, component-specific fidelity measures, distinctions between delivery and receipt of an intervention, measures of the degree of delivery of an intervention, and extent/consistency of intervention delivery (or receipt) across sites. The results will be submitted to peer-reviewed journals for publication.
Setting: The PCER project provided 36 pre-K classrooms; the ERF project provided 9 preschool classrooms; the CPS-SR program provided 64 schools of 6th through 8th graders; and the TRIAD project provided 33 pre-K classrooms in 16 schools and 28 classrooms in four Head Start Centers.
Population: Participants are staff and students in these schools.
Intervention: Varies across the four studies
Research Design and Methods: Fidelity is specified within the context of RFTs and specified for the individual studies used in this project.
Control Condition: Fidelity of intervention of the study intervention is measured in control as well as treatment condition.
Key Measures: Fidelity of implementation of components and overall intervention; reliability and validity of fidelity measures; and variations of fidelity across sites within conditions and difference across conditions.
Data Analytic Strategy: Analysis will include: (1) calculating and presenting an array of fidelity measures across studies, across sites within conditions and across conditions; (2) weighting implementation measures across intervention components; and (3) developing models for incorporating fidelity measures into analyses of the effects of interventions on study outcome measures, and examining the behavior of the models when loaded with study data.
Journal article, monograph, or newsletter
Nelson, M.C., Cordray, D.S., Hulleman, C.S., Darrow, C.L., and Sommer, E.C. (2012). A Procedure for Assessing Intervention Fidelity in Experiments Testing Educational and Behavioral Interventions. Journal of Behavioral Health Services and Research, 39(4): 374–396.
** This project was submitted to and funded as an Unsolicited application in FY 2006.