Project Activities
The research team used three general sources of information: (1) the literature on assessing intervention fidelity and on constructing measures of the reliability and validity of constructs such as fidelity of implementation; (2) three existing research studies that are sufficiently documented to permit reanalysis and evaluation of some fidelity measures; and (3) an on-going empirical study of fidelity embedded in an existing randomized field experiment (RFT). Reanalysis of the three existing data sets was carried out in cooperation with the original research teams.
The existing projects are: (1) the Preschool Curriculum Evaluation Research (PCER) project; (2) the Early Reading First (ERF) program, Wayne County, TN; and (3) the Chicago Public Schools Striving Readers (CPS-SR) program. The ongoing project was Nashville's Technology-enhanced, Research-based, Instruction, Assessment, and professional Development project (TRIAD).
Structured Abstract
Setting
The PCER project provided 36 pre-K classrooms; the ERF project provided 9 preschool classrooms; the CPS-SR program provided 64 schools of 6th through 8th graders; and the TRIAD project provided 33 pre-K classrooms in 16 schools and 28 classrooms in four Head Start Centers.
Sample
Participants are staff and students in these schools.
Varies across the four studies.
Research design and methods
Fidelity is specified within the context of RFTs and specified for the individual studies used in this project.
Control condition
Fidelity of intervention of the study intervention is measured in control as well as treatment condition.
Key measures
Fidelity of implementation of components and overall intervention; reliability and validity of fidelity measures; and variations of fidelity across sites within conditions and difference across conditions.
Data analytic strategy
Analysis included: (1) calculating and presenting an array of fidelity measures across studies, across sites within conditions and across conditions; (2) weighting implementation measures across intervention components; and (3) developing models for incorporating fidelity measures into analyses of the effects of interventions on study outcome measures, and examining the behavior of the models when loaded with study data.
People and institutions involved
IES program contact(s)
Products and publications
The products of this project include a fully developed framework for measuring the fidelity of implementation of intervention and methods for incorporating fidelity measures in the analyses of RFTs. In addition, the project produced a collection of specific measures and methods including examples of theoretical and corresponding operational measures, component-specific fidelity measures, distinctions between delivery and receipt of an intervention, measures of the degree of delivery of an intervention, and extent/consistency of intervention delivery (or receipt) across sites. The results were submitted to peer-reviewed journals for publication.
Publications:
Hulleman, C. S., & Cordray, D. S. (2009). Moving from the lab to the field: The role of fidelity and achieved relative intervention strength. Journal of Research on Educational Effectiveness, 2(1), 88-110.
Nelson, M.C., Cordray, D.S., Hulleman, C.S., Darrow, C.L., and Sommer, E.C. (2012). A Procedure for Assessing Intervention Fidelity in Experiments Testing Educational and Behavioral Interventions. Journal of Behavioral Health Services and Research, 39(4): 374-396.
Yun, C., Havard, A., Farran, D., Lipsey, M., Bilbrey, C., & Hofer, K. (2011). Subitizing and mathematics performance in early childhood. In Proceedings of the Annual Meeting of the Cognitive Science Society (Vol. 33, No. 33).
Additional project information
** This project was submitted to and funded as an Unsolicited application in FY 2006.
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
