Purpose: In the era of intervention-based experimental studies, it is not uncommon to encounter higher level, hierarchically nested data. Furthermore, a higher level data structure is, in fact, rather typical in areas of teacher and school accountability. The primary purpose of this project was to examine new extensions of commonly used hierarchical models to provide researchers with new sets of statistical tools that can help to answer important questions that could not be previously studied due to a lack of appropriate statistical methods.

Project Activities: The researchers extended current hierarchical models (HMs) to include higher levels of nesting (e.g., four or five levels), such as occur when students are nested within classrooms, within schools, within districts. In addition, the researchers incorporated a latent variable regression (LVR) feature into the higher level HMs (i.e., LVR-HM4s, LVR-HM5s) that can be used for experimental and quasi-experimental studies, evaluation studies, and growth modeling studies, potentially providing a new approach to estimating "value added" for purposes of evaluating teacher and school accountability. To illustrate the LVR-HM4s/HM5s models, secondary data analysis of two longitudinal data sets from large urban schools districts were conducted. The researchers also developed two illustrative statistical programs: annotated statistical programs of LVR-HM4s/HM5s and a user-friendly power analysis module customized to four-level/five-level HMs.

Structured Abstract

Research Design and Methods: The researchers illustrated LVR-HM4, the Multi-Site Multi-Cohort Change Model, using a multi-site, multiple-cohort longitudinal dataset from an urban district school located in a northwestern state. This dataset includes five different cohorts, and each cohort consists of longitudinal two-time points measures between the third and fifth grade in 74 elementary schools. The outcomes of interest are reading scale scores in the Iowa Test of Basic Skills (ITBS) test. These scale scores are vertically equated developmental scores. For each student in each cohort, ITBS reading scores for third and fifth grades provide the basis of estimating gains in reading achievement between two grades. The total number of students in the sample is 11,530, and the average number of students per cohort is approximately 2,306. The researchers illustrated the LVR-HM5, Teacher Effect Change Model, using a large-scale district dataset from an urban school district in Southern California. This data set contains five years of data from 1999 to 2003 for five different cohorts from 77 elementary schools. The students are from first to fifth grades. The total number of students and teachers in this sample are 14,974 and 1,037, respectively. The outcomes of interest are Stanford Achievement Test-9 reading scores.

Data Analytic Strategy: A fully Bayesian approach using the Markov Chain Monte Carlo method was employed to estimate the LVR-HM4 and LVR-HM5 models. Results from both analyses were compared with those from current three-level HMs and four-level HMs, respectively, in order to directly explore the implications and value of modeling, rather than ignoring key aspects of existing data structures.

Products and Publications

Journal article, monograph, or newsletter

Choi, K., and Kim, J. (2019). Latent Variable Regression Four-Level Hierarchical Model Using Multisite Multiple-Cohort Longitudinal Data. Journal of Educational and Behavioral Statistics, 44(5): 597–624.

Choi, K., and Seltzer, M. (2010). Modeling Heterogeneity in Relationships Between Initial Status and Rates of Change: Treating Latent Variable Regression Coefficients as Random Coefficients in a Three-Level Hierarchical Model. Journal of Educational and Behavioral Statistics, 35(1): 54–91.

Goldschmidt, P., Choi, K., Martinez, F., and Novak, J. (2010). Using Growth Models to Monitor School Performance: Comparing the Effect of the Metric and the Assessment. School Effectiveness and School Improvement, 21(3): 337–357.

** This project was submitted to and funded as an Unsolicited application in FY 2007.