Purpose: This project developed a framework to identify all the components of an education intervention and gauge both the degree and extent to which each component is in place and operating as intended. To this end, the project developed, modified, tested and combined (a) intervention theory and structure with (b) measures of intervention delivery and receipt, and (c) methods for making use of the measures of intervention fidelity in analysis in order to better understand the effects of the intervention on the outcome measures.

Project Activities: The research team used three general sources of information: (1) the literature on assessing intervention fidelity and on constructing measures of the reliability and validity of constructs such as fidelity of implementation; (2) three existing research studies that are sufficiently documented to permit reanalysis and evaluation of some fidelity measures; and (3) an on-going empirical study of fidelity embedded in an existing randomized field experiment (RFT). Reanalysis of the three existing data sets was carried out in cooperation with the original research teams.

The existing projects are: (1) the Preschool Curriculum Evaluation Research (PCER) project; (2) the Early Reading First (ERF) program, Wayne County, TN; and (3) the Chicago Public Schools Striving Readers (CPS-SR) program. The ongoing project was Nashville's Technology-enhanced, Research-based, Instruction, Assessment, and professional Development project (TRIAD).

Products: The products of this project include a fully developed framework for measuring the fidelity of implementation of intervention and methods for incorporating fidelity measures in the analyses of RFTs. In addition, the project produced a collection of specific measures and methods including examples of theoretical and corresponding operational measures, component-specific fidelity measures, distinctions between delivery and receipt of an intervention, measures of the degree of delivery of an intervention, and extent/consistency of intervention delivery (or receipt) across sites. The results were submitted to peer-reviewed journals for publication.

Structured Abstract

Setting: The PCER project provided 36 pre-K classrooms; the ERF project provided 9 preschool classrooms; the CPS-SR program provided 64 schools of 6th through 8th graders; and the TRIAD project provided 33 pre-K classrooms in 16 schools and 28 classrooms in four Head Start Centers.

Population: Participants are staff and students in these schools.

Intervention: Varies across the four studies.

Research Design and Methods: Fidelity is specified within the context of RFTs and specified for the individual studies used in this project.

Control Condition: Fidelity of intervention of the study intervention is measured in control as well as treatment condition.

Key Measures: Fidelity of implementation of components and overall intervention; reliability and validity of fidelity measures; and variations of fidelity across sites within conditions and difference across conditions.

Data Analytic Strategy: Analysis included: (1) calculating and presenting an array of fidelity measures across studies, across sites within conditions and across conditions; (2) weighting implementation measures across intervention components; and (3) developing models for incorporating fidelity measures into analyses of the effects of interventions on study outcome measures, and examining the behavior of the models when loaded with study data.

Products and Publications

Publications:

Hulleman, C. S., & Cordray, D. S. (2009). Moving from the lab to the field: The role of fidelity and achieved relative intervention strength. Journal of Research on Educational Effectiveness, 2(1), 88–110.

Nelson, M.C., Cordray, D.S., Hulleman, C.S., Darrow, C.L., and Sommer, E.C. (2012). A Procedure for Assessing Intervention Fidelity in Experiments Testing Educational and Behavioral Interventions. Journal of Behavioral Health Services and Research, 39(4): 374–396.

Yun, C., Havard, A., Farran, D., Lipsey, M., Bilbrey, C., & Hofer, K. (2011). Subitizing and mathematics performance in early childhood. In Proceedings of the Annual Meeting of the Cognitive Science Society (Vol. 33, No. 33).

** This project was submitted to and funded as an Unsolicited application in FY 2006.