Skip to main content

Breadcrumb

Home arrow_forward_ios Information on ... arrow_forward_ios Deriving and De ...
Home arrow_forward_ios ... arrow_forward_ios Deriving and De ...
Information on ...
Grant Closed

Deriving and Developing Tools to Estimate Optimal Data Points for Quasi-Experimental Designs

NCER
Program: Statistical and Research Methodology in Education
Program topic(s): Core
Award amount: $648,791
Principal investigator: Eric Hedberg
Awardee:
Abt Associates, Inc.
Year: 2020
Award period: 5 years (07/01/2020 - 06/30/2025)
Project type:
Methodological Innovation
Award number: R305D200045

Purpose

Education policies and interventions are often implemented in ways that do not render it feasible to use a randomized control trial to test their effects. The purposes of this study were to derive theory for computing exact statistical power for three common quasi-experimental designs (QEDs) - nonequivalent control group designs, difference-in-differences, and interrupted time series - and to develop software for computing statistical power using the derived approaches for those QEDs.

Project Activities

The software was programmed in R and is freely available to applied researchers. In addition to publishing in journals and presenting at conferences, the research team used social media, press releases through NORC, briefs for nontechnical publications, and workshops at conferences, such as AERA and SREE, for applied education researchers.

Key outcomes

The main finding of this project is as follows:  

  • In anticipating sample size requirements for quasi-experiments, the number of observations required for such studies typically exceed those required for expected impacts from randomized trials. For propensity score-based studies, the sample requirements are often 30 percent larger, and for time-series studies, the requirements depend on the level of autocorrelation. If the autocorrelation between time points is about .5, the sample size requirements range from needing 4 to 16 times more timepoints relative to data without autocorrelation, depending on the model used. (Hedberg, 2023)

People and institutions involved

IES program contact(s)

Charles Laurin

Project contributors

Larry Hedges

Co-principal investigator
Board of Trustees Professor of Statistics; Faculty Fellow, Institute for Policy Research
Northwestern University

Products and publications

Publications:

Hedberg, E. C. (2023). How Many Cases per Cluster? Operationalizing the Number of Units per Cluster Relative to Minimum Detectable Effects in Two-Level Cluster Randomized Evaluations with Linear Outcomes. American Journal of Evaluation, 44(1), 153-168.

Hedberg, E. C., & Hedges, L. V. (2026). Computing Statistical Power for the Difference in Differences Design. Evaluation Review, 50(1), 149-180.

Additional project information

The code for the software is available at: https://github.com/hedbergec/plannerapp.

The app is available at: https://steppcenter.shinyapps.io/Planner/

Previous award details:

Previous award number:
R305D200023
Previous awardee:
NORC at the University of Chicago

Questions about this project?

To answer additional questions about this project or provide feedback, please contact the program officer.

 

Tags

Data and Assessments

Share

Icon to link to Facebook social media siteIcon to link to X social media siteIcon to link to LinkedIn social media siteIcon to copy link value

Questions about this project?

To answer additional questions about this project or provide feedback, please contact the program officer.

 

You may also like

Zoomed in IES logo
Workshop/Training

Summer Research Training Institute on Cluster-Rand...

July 06, 2026
Read More
Zoomed in IES logo
Workshop/Training

Data Science for Education (DS4EDU)

July 01, 2026
Read More
A group of teachers working together around a table.
Thought leadership

An Idea Whose Time Has Come: IES Welcomes New Prog...

June 04, 2026 by Matthew Soldner
Read More
icon-dot-govicon-https icon-quote