Exhibit III.b.1. Scenario III, Missing Data Dependent on Post-test: Data Missing for 40% of Students

Technical Methods Report: What to Do When Data Are Missing in Group Randomized Controlled Trials

NCEE 2009-0049
October 2009

List of Tables

Data	Pre-test Data Available?	Estimate	True Imact	Bias	Bias Level	Standard Estimate	Unbiased Estimate	Bias	Bias Level	% of Samples in which 90% Cl Contains .20
		Imact Estimate				Standard Error of Imact Est.				90% Cl
No Missing Data	No	0.203	0.200	0.003	Low Bias	0.085	0.088	-0.002	Low Bias 0.892 0.892
	Yes	0.203	0.200	0.003	Low Bias	0.062	0.062	0.000	Low Bias	0.886
A. Pre-Test (X) Data Missing
Case Deletion	No	0.133	0.200	-0.067	High Bias	0.064	0.063	0.001	Low Bias	0.723
	Yes
Dummy Variable Method	No	0.215	0.200	0.015	Low Bias	0.067	0.067	0.000	Low Bias	0.886
	Yes
Mean Value Imputation	No	0.267	0.200	0.067	High Bias	0.069	0.065	0.003	Low Bias	0.749
	Yes
Single, Non-stochastic	No	0.200	0.200	0.000	Low Bias	0.065	0.064	0.001	Low Bias	0.899
Regression Imputation	Yes
Single, Stochastic	No	0.201	0.200	0.001	Low Bias	0.064	0.063	0.001	Low Bias	0.890
Regression Imputation	Yes
Multiple Stochastic	No	0.201	0.200	0.001	Low Bias	0.065	0.062	0.003	Low Bias	0.903
Regrssion Imputation (n=5)	Yes
EM Algorithm with Multiple Imputation (n=5)	No	0.202	0.200	0.002	Low Bias	0.066	0.063	0.003	Low Bias	0.899
	Yes
B. Post-Test (Y) Data Missing
Case Deletion	No	0.076	0.200	-0.124	High Bias	0.087	0.087	0.000	Low Bias	0.574
	Yes	0.113	0.200	-0.067	High Bias	0.064	0.063	0.001	Low Bias	0.723
Mean Value Imputation	No	0.059	0.200	-0.142	High Bias	0.053	0.087	-0.035	Low Bias	0.261
	Yes	0.057	0.200	-0.143	High Bias	0.040	0.069	-0.030	Low Bias	0.127
Single, Non-stocastic	No	0.079	0.200	-0.121	High Bias	0.088	0.087	0.000	Low Bias	0.592
Regression Imputation	Yes	0.140	0.200	-0.060	High Bias	0.064	0.063	0.000	Low Bias	0.758
Single, Stochastic	No	0.080	0.200	-0.120	High Bias	0.090	0.089	0.001	Low Bias	0.600
Regression Imputation	Yes	0.141	0.200	-0.059	High Bias	0.065	0.065	0.001	Low Bias	0.763
Multiple, Stochastic	No	0.078	0.200	-0.122	High Bias	0.095	0.087	0.007	Low Bias	0.632
Regression Imputation (n=5)	Yes	0.139	0.200	-0.061	High Bias	0.068	0.063	0.005	Low Bias	0.797
EM Algorithm with Multiple Imputation (n=5)	No	0.080	0.200	-0.120	High Bias	0.095	0.088	0.007	Low Bias	0.636
	Yes	0.138	0.200	-0.062	High Bias	0.069	0.063	0.006	Low Bias	0.800
Weighting - Simple	No	0.079	0.200	-0.121	High Bias	0.088	0.087	0.000	Low Bias	0.588
	Yes	0.135	0.200	-0.065	High Bias	0.064	0.063	0.001	Low Bias	0.735
Weighting - Sophisticated	No	0.079	0.200	-0.121	High Bias	0.088	0.087	0.000	Low Bias	0.593
	Yes	0.140	0.200	-0.060	High Bias	0.064	0.063	0.001	Low Bias	0.761
Fully Specified Regression Models	No	0.138	0.200	-0.062	High Bias	0.064	0.063	0.001	Low Bias	0.752
w/ Treatment-Covariate Interactions	Yes
Note: When pre-test scores are available, they are used as a covariate in the analysis model. In addition, we used pre-test scores to impute values and create weights. Bias estimates were computed as described in Chapter 4 and repeated at the beginning of this appendix. The level of the bias is characterized as "High Bias" or "Low Bias" based on the criteria established in Chapter 4. 90% CI refers to the 90-percent confidence interval around the impact estimate. For more details on the simulations, see Chapter 4 and Appendix C.

Institute of Education Sciences
National Center for Education Evaluation and Regional Assistance

U.S. Department of Education