Differential effects of English language learner training and materials—On Our Way to English (OWE) and Responsive Instruction for Success (RISE)
Analysis plan
The primary emphasis of data analysis is to estimate the effects of the combined RISE and OWE programs on the academic achievement of students in year 2 of the study. Additional analyses will be conducted to estimate the effects of the two interventions on teacher outcomes. The proposed cluster-randomized trial involves randomization at the school level and collection of outcome data at the student and teacher level. Estimation of treatment effects at the level of the cluster (school) is considered the most appropriate methodology, given the nesting of the data.
This nested design necessitates analysis through multilevel modeling. Two separate multilevel models will be estimated: one to address the primary research question about student achievement and one to address the secondary research question about changes in teacher pedagogy. The effects of RISE and OWE on student outcomes will be analyzed through a three-level hierarchical model. Level 1 will nest students within classrooms and include the students' ELL status as a predictor. Level 1 will also include a student level covariate to account for baseline achievement. Level 2 will include the percent of students in each school on free or reduced-price lunch and a measure of teaching experience with English language learner students as covariates. Level 3 will include an indicator for assignment to intervention or control as a predictor of mean school achievement to estimate the effect of the intervention on student achievement. It will also include a covariate for school district to improve the power of the estimation of the intervention's effect.
Effects of RISE and OWE on teacher outcomes (changes in teacher pedagogical practices, as determined by the quarterly online implementation log) will be analyzed through a two-level hierarchical model. For this model, level 1 nests teachers within schools and uses teacher practices (a function of general teaching experience) and experience teaching English language learner students as covariates. Level 2 will include a school-level covariate at the teacher level from baseline measures of teacher knowledge and experience as well as a variable to indicate group assignment. In addition, the district in which the school is located will be included as a covariate.
As recommended by Shadish, Cook, and Campbell (2002), the researchers will analyze attrition—descriptive information on the overall rate of attrition, the different rate for treatment and control conditions, and whether completers differ from noncompleters. The percentage of students and teachers in each group for whom outcome data could not be obtained and the number of schools in each group that did not complete the study will be reported. Analyses will compare baseline results for the initial sample in each group with baseline results for the final sample to determine whether the final sample differs from the initial sample. Model-based multiple imputation methods will be used to handle missing data.