- Disclosure
- List of Tables & Figures
- Chapter 1. Introduction
- Chapter 2. The Late Pretest Problem
- Chapter 3. Measuring the Variance-Bias Tradeoff
- Chapter 4. Considered Designs
- Chapter 5. Theoretical Framework
- Chapter 6. The Variance-Bias Tradeoff for Various ATE Estimators
- Chapter 7: Empirical Analysis
- Chapter 8: Summary and Conclusions
- References
Appendix A: Proof of Asymptotic Results for the ANCOVA Estimator (174 KB)- PDF & Related Info
Chapter 2: The Late Pretest Problem
Pretests on individual students are typically collected after the start of the school year for several reasons. First, school administrators and teachers typically prefer that baseline testing occur after students and teachers settle into a routine. Second, researchers often want to delay testing until a large percentage of signed study consent forms are returned by parents (many studies in a school setting require active parental consent). Finally, for cost reasons, studies often employ a small number of interviewer teams per site to administer baseline testing in the study schools. Thus, it usually takes time for these teams to set up visiting schedules and to travel to schools that are geographically dispersed. Hence, in many RCTs, baseline testing is not completed until several months after school begins. For example, in the Head Start Impact Study (Puma et al. 2005) baseline testing occurred over a three-month period from October 2002 through December 2002.
The inclusion of late pretests in the posttest impact models could lead to biased impact estimates for several reasons. First, in most evaluations, the tested interventions are implemented in the treatment schools and classrooms prior to the start of the school year. For example, in evaluations testing the effects of a new math or reading curriculum, teacher professional development typically occurs during the summer. Thus, with late pretests, students in the treatment group have already been exposed to the intervention.
A second reason that pretests could be contaminated is if the distribution of baseline testing dates differs across the treatment and control groups. Student test scores tend to increase over time naturally. Thus, pretests could be contaminated if they are administered later for one research group than the other, even if there are no early intervention effects. Well-designed evaluations attempt to evenly disperse testing dates across the treatment and control groups. However, it is sometimes more difficult to schedule testing dates in control schools (who are denied the intervention) than in treatment schools (who are offered intervention services). For example, in the National Evaluation of Early Reading First (Jackson et al. 2007) baseline testing occurred about one month later, on average, in control sites than treatment sites.