Grant Closed

Examining the Changes in Methodology that Occur Between the Design and Implementation of Field Trials in Education

NCER

Program: Statistical and Research Methodology in Education

Program topic(s): Core

Award amount: $300,841

Principal investigator: Jessaca Spybrook

Awardee:

Western Michigan University

Year: 2009

Project type:

Methodological Innovation

Award number: R305D090013

Purpose

The purpose of this project was to gather evidence about the changes in methodology that occur between the design and implementation of cluster randomized trials (when groups are randomly assigned to treatment or control conditions) used in education research. Specifically, the researchers examined the changes from what is planned to what is implemented in: (1) the research design; (2) sample sizes; (3) outcome measures; (4) the intervention itself; and (5) expected rates of attrition. The researchers attempted to uncover why the changes occurred and the impact of the changes on the evidence produced in the study.

Project Activities

The sample for this study consisted of the cluster randomized trials funded between 2002 and 2006 by the National Center for Education Research and the National Center for Education Evaluation and Regional Assistance, two branches of the Institute of Education Sciences. In a previous study, the researchers examined the elements of the proposed research design using the funded proposals for each of the studies in the sample. The proposed study builds on this work by examining how the methodology changed when the studies were implemented in the field.

People and institutions involved

IES program contact(s)

Charles Laurin

Education Research Analyst

NCER

Products and publications

Journal article, monograph, or newsletter

Spybrook, J., Puente, A.C., and Lininger, M. (2011). An Examination of the Impact of Changes in Federal Policies on the Landscape of Educational Research in the USA. Effective Education, 3(2): 83-88.

Supplemental information

The primary mode of collecting data was through interviews with the principal investigators of the studies, and other knowledgeable project staff as needed. Study reports were also be used to fill in gaps and check data obtained through the interviews. Open coding was used with approximately the first 25 percent of the interviews in order to develop more specific coding that will be used with the remainder of the interviews. Inter-rater reliability of at least 90 percent was established. The percentage of studies that exhibited changes within each of the five topics was calculated, the type of changes were catalogued, and the most common reasons for the changes were identified. In addition, the researchers assessed the impacts of the changes on the validity of each study including statistical conclusion validity, internal validity, eternal validity, or construct validity.