Exhibit III.b.2. Scenario III, Missing Data Dependent on Post-test: Data Missing for 40% of Schools

Technical Methods Report: What to Do When Data Are Missing in Group Randomized Controlled Trials

NCEE 2009-0049
October 2009

List of Tables

Data	Pre-test Data Available?	Estimate	True Imact	Bias	Bias Level	Standard Estimate	Unbiased Estimate	Bias	Bias Level	% of Samples in which 90% Cl Contains .20
		Imact Estimate				Standard Error of Imact Est.				90% Cl
No Missing Data	No	0.203	0.200	0.003	Low Bias	0.085	0.088	-0.002	Low Bias 0.892 0.892
	Yes	0.203	0.200	0.003	Low Bias	0.062	0.062	0.000	Low Bias	0.886
A. Pre-Test (X) Data Missing
Case Deletion	No	0.176	0.200	-0.024	Low Bias	0.080	0.081	0.000	Low Bias	0.876
	Yes
Dummy Variable Method	No	0.207	0.200	0.007	Low Bias	0.072	0.073	0.000	Low Bias	0.904
	Yes
Mean Value Imputation	No	0.225	0.200	0.025	Low Bias	0.073	0.081	-0.008	Low Bias	0.851
	Yes
Single, Non-stochastic	No	0.203	0.200	0.003	Low Bias	0.055	0.081	-0.026	High Bias	0.734
Regression Imputation	Yes
Single, Stochastic	No	0.203	0.200	0.003	Low Bias	0.062	0.078	-0.017	High Bias	0.790
Regression Imputation	Yes
Multiple Stochastic	No	0.197	0.200	-0.003	Low Bias	0.067	0.073	-0.006	Low Bias	0.865
Regrssion Imputation (n=5)	Yes
EM Algorithm with Multiple Imputation (n=5)	No	0.202	0.200	0.002	Low Bias	0.074	0.075	-0.001	Low Bias	0.897
	Yes
B. Post-Test (Y) Data Missing
Case Deletion	No	0.152	0.200	-0.048	Low Bias	0.110	0.112	-0.003	Low Bias	0.850
	Yes	0.176	0.200	-0.024	Low Bias	0.080	0.081	0.000	Low Bias	0.876
Mean Value Imputation	No	0.152	0.200	-0.048	Low Bias	0.064	0.112	-0.048	Low Bias	0.590
	Yes	0.151	0.200	-0.049	Low Bias	0.055	0.099	-0.044	Low Bias	0.573
Single, Non-stocastic	No	0.152	0.200	-0.048	Low Bias	0.064	0.112	-0.048	Low Bias	0.590
Regression Imputation	Yes	0.177	0.200	-0.023	Low Bias	0.048	0.083	-0.036	Low Bias	0.632
Single, Stochastic	No	0.150	0.200	-0.050	Low Bias	0.082	0.126	-0.048	Low Bias	0.655
Regression Imputation	Yes	0.176	0.200	-0.024	Low Bias	0.060	0.092	-0.032	Low Bias	0.697
Multiple, Stochastic	No	0.166	0.200	-0.034	Low Bias	0.101	0.116	-0.015	Low Bias	0.807
Regression Imputation (n=5)	Yes	0.187	0.200	-0.013	Low Bias	0.074	0.085	-0.011	Low Bias	0.894
EM Algorithm with Multiple Imputation (n=5)	No	0.177	0.200	-0.023	Low Bias	0.086	0.085	0.001	Low Bias	0.876
	Yes
Weighting - Simple	No
	Yes
Weighting - Sophisticated	No
	Yes	0.177	0.200	-0.060	Low Bias	0.080	0.063	0.000	Low Bias	0.879
Fully Specified Regression Models	No	0.178	0.200	-0.062	Low Bias	0.080	0.063	0.000	Low Bias	0.877
w/ Treatment-Covariate Interactions	Yes
Note: When pre-test scores are available, they are used as a covariate in the analysis model. In addition, we used pre-test scores to impute values and create weights. Bias estimates were computed as described in Chapter 4 and repeated at the beginning of this appendix. The level of the bias is characterized as "High Bias" or "Low Bias" based on the criteria established in Chapter 4. 90% CI refers to the 90-percent confidence interval around the impact estimate. For more details on the simulations, see Chapter 4 and Appendix C.