Practical Tools for Multilevel Hierarchical Modeling in Education Research

In this project, the researchers developed and tested new approaches to support applied researchers' use of Bayesian modal estimation (also called maximum a posteriori estimation) for multilevel models. The goal of this methodological innovation was to make it easier for applied researchers to avoid nonsensical results when they use multilevel models.

Multilevel data structures are common in education research in studies that range from descriptive analyses of children nested within classrooms and schools to more formal cluster randomized field experiments. A problem frequently encountered when fitting such models using standard software (such as HLM, Stata, R, or SPSS) is that the (restricted) maximum likelihood estimates are on the boundary of parameter space or that convergence fails. In either case, this can result in estimates that do not make sense. Examples include variance components estimated as zero, correlations between random intercepts and random slopes estimated as 1 or -1, and non-convergence due to sparseness or separation (when a linear combination of the predictors is perfectly predictive of the outcome). These problems are particularly common in small studies of educational interventions that often involve fewer than 10 or 15 classrooms. In larger studies, these concerns arise with grouping factors such as school district or study site that typically have only a small number of levels or when specifying random slopes for variables that do not always vary substantially within groups. Consequences include understated standard errors of regression coefficients and unrealistic inferences regarding the random effects.

In these situations, researchers may use Bayesian methods to guarantee sensible estimates. Bayesian inference methods use prior distributions to provide such guarantees. In this project, as the researchers were developing a Bayesian estimation algorithm, they created weakly informative priors to test their approach and ensure the algorithm avoided estimating correlations of 1 or variances of 0. "Weakly informative" priors means that when the data provide inadequate information on a parameter to be estimated, meaningless estimates will be avoided, but when the data provide sufficient information on that parameter, the priors will have essentially no effect on the estimates. The prior distributions developed by this project were designed to meet three other quality conditions. First, they are friendly for computation by having (i) a density function that is easy to evaluate and (ii) a functional form (and its derivatives) that is "analytic" enough so that iterative optimization algorithms can be applied. The second condition they meet is that they are parameterized and have enough flexibility to be tuned to meet various a priori beliefs. The third condition is that the tuning parameters have direct practical interpretations so that it is clear to applied modelers that they will have minimal influence on parameters that are well identified from data.