Skip Navigation

The Effects of Success in Sight as a School Improvement InterventionThe Effects of Success in Sight as a School Improvement Intervention

Analysis plan

Data will be analyzed to determine the effects of Success in Sight on student achievement in participating schools after two years of the intervention. Further analyses are conducted to estimate the effects of Success in Sight on school-level reform practices. Outcome data are collected at the student and teacher levels. No subgroup analyses—race/ethnicity or socioeconomic status—will occur. Intervention effects will be estimated at the school level using multilevel modeling to account for the random assignment of schools and the sources of variability in the data that result from the nested structure of the school environment.

Effects of Success in Sight on student outcomes will be analyzed using a three-level hierarchical model. The level 1 model nests students within schools and includes grade level (3 or 4) as a predictor. This model also includes a student level covariate to account for baseline achievement. The level 2 model includes an indicator for assignment to intervention or control as a predictor of mean school achievement to estimate the effect of the intervention on student achievement. This model also includes a school-level covariate (school mean on achievement) to explain additional between-school variance not explained in the level 1 model, to control for prior achievement, and to improve the power of the estimation of the intervention's effect. This model is run on two student samples—students who remained in the school for the entire two years ("stayers") and the complete school population that includes all students ("stayers" and "in-movers"). Because the achievement data come from two different state assessments, they will be analyzed by state, and the estimates are combined using a meta-analytic approach.

Effects of Success in Sight on the four school-level reform practices are also analyzed using three-level hierarchical models. Four separate models are run, one for each reform practice (data-based decisionmaking, purposeful community, shared leadership, and effective school practices). The level 1 model nests teachers within schools and includes a teacher-level covariate to account for baseline reform practices. The level 2 model includes an indicator for assignment to treatment or control condition; district and school size are treated as predictors of mean engagement in the reform practice to estimate the effect of the intervention on these practices. The level 2 model also includes a school-level covariate (school mean on the reform practice) to explain additional between-school variance not explained in the level 1 model, to control for baseline engagement in the reform practice, and to improve the power of the estimation of the intervention's effect. The level 3 model includes indicators for each of the 26 matched pairs.

Analyses are conducted to examine how attrition affects the findings. Results are presented that show the percentage of teachers in each group for whom outcome data could not be obtained and the number of schools in each group that did not complete the study. Analyses compare baseline results for the initial sample in each group with baseline results for the final sample, to determine whether the final sample differs from the initial sample.

Return to Index