Skip Navigation
Funding Opportunities | Search Funded Research Grants and Contracts

IES Grant

Title: Examining the Changes in Methodology that Occur Between the Design and Implementation of Field Trials in Education
Center: NCER Year: 2009
Principal Investigator: Spybrook, Jessaca Awardee: Western Michigan University
Program: Statistical and Research Methodology in Education      [Program Details]
Award Period: 2 years Award Amount: $300,841
Type: Methodological Innovation Award Number: R305D090013
Description:

Purpose: The purpose of this project was to gather evidence about the changes in methodology that occur between the design and implementation of cluster randomized trials (when groups are randomly assigned to treatment or control conditions) used in education research. Specifically, the researchers examined the changes from what is planned to what is implemented in: (1) the research design; (2) sample sizes; (3) outcome measures; (4) the intervention itself; and (5) expected rates of attrition. The researchers attempted to uncover why the changes occurred and the impact of the changes on the evidence produced in the study.

Project Activities: The sample for this study consisted of the cluster randomized trials funded between 2002 and 2006 by the National Center for Education Research and the National Center for Education Evaluation and Regional Assistance, two branches of the Institute of Education Sciences. In a previous study, the researchers examined the elements of the proposed research design using the funded proposals for each of the studies in the sample. The proposed study builds on this work by examining how the methodology changed when the studies were implemented in the field.

The primary mode of collecting data was through interviews with the principal investigators of the studies, and other knowledgeable project staff as needed. Study reports were also be used to fill in gaps and check data obtained through the interviews. Open coding was used with approximately the first 25 percent of the interviews in order to develop more specific coding that will be used with the remainder of the interviews. Inter-rater reliability of at least 90 percent was established. The percentage of studies that exhibited changes within each of the five topics was calculated, the type of changes were catalogued, and the most common reasons for the changes were identified. In addition, the researchers assessed the impacts of the changes on the validity of each study including statistical conclusion validity, internal validity, eternal validity, or construct validity.

Products and Publications

Publications

Journal article, monograph, or newsletter

Spybrook, J., Puente, A.C., and Lininger, M. (2011). An Examination of the Impact of Changes in Federal Policies on the Landscape of Educational Research in the USA. Effective Education, 3(2): 83–88.


Back