Co-Principal Investigator: Finn, Jeremy

Purpose: The project developed academic growth references for K–12 reading and math achievement based on nationally representative longitudinal datasets and time-referenced effect size metrics, based on those national academic data, that can be used to assess the effectiveness of educational interventions.

Conventional effect size metrics such as Cohen's d are standardized group mean differences based on the distributions of student outcome variables at one particular age or grade level. They do not take into account the time dimension (i.e., the time needed to learn at that age/grade level). This study is based on the premise that time-indexed effect size metrics can estimate how long it would take for an "untreated" control group to reach the treatment group outcome in terms familiar to educators—months of schooling. These "months of schooling" effect-size metrics will differ from conventional grade equivalent (GE) metrics as strength-of-effect measures, which suffer from several limitations. For instance, GEs are drawn from test publishers' norms derived from cross-sectional data of different cohort groups at a single year to estimate growth curves. Moreover, the assumption under GE that the study sample would grow at the same rate as the national norms could be erroneous. The new measures adjust the growth trajectory based on national longitudinal data using vertical scales of achievement along with information regarding the demographic profiles of the study sample and settings.

Project Activities: Primary data sources for developing the national growth references in K–12 reading and math included existing norms from standardized achievement tests such as the Metropolitan Achievement Tests (MAT), the Comprehensive Tests of Basic Skills/Terra Nova (CTBS/TN), and the Stanford Achievement Test Series (SAT). National longitudinal datasets included the elementary school-level dataset (K to 8th grade) Early Childhood Longitudinal Study, Kindergarten Class of 1998–99 (ECLS-K) and the high school-level dataset (8th to 12th grade) National Education Longitudinal Study of 1988 (NELS:88). These national datasets, together with advanced psychometric and statistical tools such as item response theory (IRT), developmental (vertical) scaling, and hierarchal linear modeling (HLM), offer a new way to measure and examine academic growth. In particular, meta-analytic synthesis of existing norms from test publishers and new norms derived from longitudinal datasets can lead to the development of more valid and reliable references for time-indexed effect size metrics. These metrics provide developmentally appropriate evaluations of educational interventions.

Prior evidence from selected experimental research (Project STAR) and quasi-experimental research (Prospects Title I) will be reevaluated using this growth curve analysis framework, and the time-indexed effect size measures will be compared to those traditional effect size measures that have been computed previously. This research contributes to enhancing our capacity to understand or provide a context for interpreting the size of an effect, a step toward bridging the gap between educational research and practice.

Products and Publications

Project Website: http://gse.buffalo.edu/faculty/centers/ties

Journal article, monograph, or newsletter

Lee, J., Finn, J., and Liu, X. (2019). Time-Indexed Effect Size for Educational Research and Evaluation: Reinterpreting Program Effects and Achievement Gaps in K–12 Reading and Math. The Journal of Experimental Education, 87(2), 193–213.