Examining the Changes in Methodology that Occur Between the Design and Implementation of Field Trials in Education

The sample for this study consisted of the cluster randomized trials funded between 2002 and 2006 by the National Center for Education Research and the National Center for Education Evaluation and Regional Assistance, two branches of the Institute of Education Sciences. In a previous study, the researchers examined the elements of the proposed research design using the funded proposals for each of the studies in the sample. The proposed study builds on this work by examining how the methodology changed when the studies were implemented in the field.

The primary mode of collecting data was through interviews with the principal investigators of the studies, and other knowledgeable project staff as needed. Study reports were also be used to fill in gaps and check data obtained through the interviews. Open coding was used with approximately the first 25 percent of the interviews in order to develop more specific coding that will be used with the remainder of the interviews. Inter-rater reliability of at least 90 percent was established. The percentage of studies that exhibited changes within each of the five topics was calculated, the type of changes were catalogued, and the most common reasons for the changes were identified. In addition, the researchers assessed the impacts of the changes on the validity of each study including statistical conclusion validity, internal validity, eternal validity, or construct validity.