Skip Navigation
Technical Methods Report: What to Do When Data Are Missing in Group Randomized Controlled Trials

NCEE 2009-0049
October 2009

References

Agodini, R., & M. Dynarski (2001). "Are Experiments the Only Option? A look at Dropout Prevention programs." The Review of Economics and Statistics, 86(1), 190-194.

Ahn, H. & J. L. Powell (1993). "Semiparametric Estimation of Censored Selection Models with a Nonparametric Selection Mechanism." Journal of Econometrics, 58, 3-19.

Allison, P.D. (2002). Missing Data. Thousand Oaks, CA: Sage University Paper No. 136.

Altonji, J. G., T. E. Elder, & C. R. Taber (2005). "Selection on Observed and Unobserved Variables: Assessing the Effectiveness of Catholic Schools." Journal of Political Economy. Vol. 113, February 2005.

Baker, S.G., G.M. Fitzmaurice, L.S. Freedman, & B.S. Kramer (2006) "Simple Adjustments For Randomized Trials With Nonramdomly Missing Or Censored Outcomes Arising From Informative Covariates." Biostatistics, 7(1), 29-40.

Barnard, J., & X.L. Meng (1999), "Applications of Multiple Imputation in Medical Studies: From AIDS to NHANES." Statistical Methods in Medical Research, 8, 17-36.

Battaglia, M. P., M.R. Frankel, & M. W. Link (2008). "Improving Standard Poststratification Techniques for Random-Digit-Dialing Telephone Surveys." Survey Research Methods, 2(1), 11-19.

Bell, S.H., & L.L. Orr (1994). "Is subsidized employment cost effective for welfare recipients? Experimental evidence from seven state demonstrations." The Journal of Human Resources, 29 (1), 42-61.

Bernstein, L., Dun Rappaport, C., Olsho, L., Hunt, D., & Levin, M. (2009). Impact Evaluation of the U.S. Department of Education's Student Mentoring Program (NCEE 2009-4047). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

Bloom, H. (2005). "Randomizing Groups to Evaluate Place-Based Programs," in Learning More from Social Experiments, New York: Russell Sage Foundation.

Bloom, H. (1984). "Accounting for no-shows in experimental evaluation designs." Evaluation Review, 8(2), 225-246.

Bloom, H., et al., (2002). Can Non-Experimental Comparison Group Methods Match The Findings From A Random Assignment Evaluation Of Mandatory Work-to-Welfare Programs? New York, NY: MDRC.

Borman, G.D., R.E. Slavin, R.E., A.C.K. Cheung, A.M. Chamberlain, N.A. Madden, & B. Chambers (2007). "Final reading outcomes of the national randomized field trial of Success for All." American Educational Research Journal, 44(3), 701-731.

Bryk, A. S. & S. W. Raudenbush (1992), Hierarchical Linear Models: Applications and Data Analysis Methods. Newbury Park, CA: SAGE Publications.

Bullock, J. C. (2005). "Effects of the Accelerated Reader on reading performance of third, fourth, and fifth-grade students in one western Oregon elementary school." University of Oregon; 0171 Advisor: Gerald Tindal. DAI, 66 (07A), 56-2529.

Burton, A. & D. G. Altman (2004). "Missing covariate data within cancer prognostic studies: a review of current reporting and proposed guidelines." British Journal of Cancer, 91, 4-8.

Campuzano, L., Dynarski, M., Agodini, R., & Rall, K. (2009). Effectiveness of Reading and Mathematics Software Products: Findings From Two Student Cohorts (NCEE 2009-4041). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

Carpenter, J. R., & M. G. Kenward (2007). Missing Data in Randomised Control Trials—A Practical Guide, unpublished monograph, http://www.missingdata.org.uk.

Chamberlain, G. (1986). "Asymptotic Efficiency in Semiparametric Models with Censoring." Journal of Econometrics, 32, 189-218.

Constantine, J., Player D., Silva, T., Hallgren, K., Grider, M., & Deke, J. (2009). An Evaluation of Teachers Trained Through Different Routes to Certification, Final Report (NCEE 2009-4043). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

Cook, T.D., F. Habib, M. Phillips, R.A. Settersten, S.C. Shagle, & S.M. Degirmencioglu (1999). "Comer's School Development Program in Prince George's County, Maryland: A Theory-Based Evaluation." American Educational Research Journal, Vol. 36, No. 3, 543-597.

Corrin, W., Somers, M.-A., Kemple, J., Nelson, E., & Sepanik, S. (2008). The Enhanced Reading Opportunities Study: Findings from the Second Year of Implementation (NCEE 2009-4036). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

Cox, B. (1991). "Weighting Survey Data for Analysis." Paper presented at the American Statistical Association's Continuing Education Program, Joint Statistical Meetings.

Das, M, W. Newey, & F Vella (2003). "Nonparametric Estimation of Sample Selection Models." Review of Economic Studies, 70, 33-58.

Dehejia, R., & S. Wahba (1999). "Causal Effects In Nonexperimental Studies: Reevaluating The Evaluation Of Training Programs." Journal of the American Statistical Association, 94 (448), 1053-1062.

Dehejia, R., & S. Wahba (2002). "Propensity Score Matching Methods For Nonexperimental Causal Studies." Review of Economics and Statistics, 84(1), 151-161.

DiNardo, J, J. McCrary, & L. Sanbonmatsu (2006). "Constructive Proposals for Dealing with Attrition: An Empirical Example." NBER working paper, July 21, 2006. http://wwwpersonal.umich.edu/~jdinardo/DMS_v9.pdf.

Food and Drug Administration (2006). "Guidance for Industry Patient-reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims (Draft Guidance)." Rockville, MD: U.S. Department of Health and Human Services, Food and Drug Administration.

Gamse, B.C., Jacob, R.T., Horst, M., Boulay, B., & Unlu, F. (2008). Reading First Impact Study Final Report (NCEE 2009-4038). Washington, DC: National Center for Education Evaluation.

Garet, M. S., S. Cronen, M. Eaton, A. Kurki, M. Ludwig, W. Jones, K. Uekawa, A. Falk, H. Bloom, F. Doolittle, P. Zhu, & L. Sztejnberg (2008). The Impact of Two Professional Development Interventions on Early Reading Instruction and Achievement (NCEE 2008-4030). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

Gennetian, L.A., P.A. Morris, J.M. Bos, & H.S. Bloom (2005). "Constructing Instrumental Variables From Experimental Data To Explore How Treatments Produce Effects." In Learning More from Social Experiments: Evolving Analytic Approaches, Russell Sage Foundation, New York, H. Bloom editor.

Graham, J.W. (2009). "Missing Data Analysis: Making it Work in the Real World." Annual Review of Psychology, 60, 549-576.

Haitovsky, Y. (1968). "Missing Data in Regression Analysis." Journal of the Royal Statistical Society, Series B, 30, 67-82.

Heckman, J.J. (1976). "The Common Structure Of Statistical Models Of Truncated, Sample Selection And Limited Dependent Variables, And A Simple Estimator Of Such Models." Annals of Economic and Social Measurement, 5, 475-492.

Heckman, J.J., & V.J. Hotz, V.J (1989). Choosing Among Alternative Nonexperimental Methods For Estimating The Impact Of Social Programs: The Case Of Manpower Training. University of Chicago, Economics Research Center.

Horovitz, J.L. & C. F. Manski (1998). "Censoring of Outcomes and Regressors Due to Survey Nonresponse: Identification and Estimation Using Weights and Imputations." Journal of Econometrics, 84, 37-58.

Horovitz, J.L. & C. F. Manski (2000). "Nonparametric Analysis of Randomized Experiments with Missing Covariate and Outcome Data." Journal of the American Statistical Association, 95, 77-84.

Horton, N.J. & K.P. Kleinman (2007). "Much Ado About Nothing: A Comparison Of Missing Data Methods And Software To Fit Incomplete Data Regression Models." The American Statistician, 61(1), 79-90.

Horvitz, D.G. & Thompson, D.J. (1952). "A generalization of sampling without replacement from a finite universe." Journal of the American Statistical Association, 47, 663-685.

Imbens, G.W. & J.M. Wooldridge (2009). "Recent Development in the Econometrics of Program Evaluation." Journal of Economic Literature, 47(1), 5-86.

Jones, M. (1996). "Indicator and Stratification Methods for Missing Explanatory Variables in Multiple Linear Regression." Journal of the American Statistical Association, 91, 222-230.

Judkins, D., Hao, H., Barrett, B., & Adhikari, P. (2005). "Modeling and Polishing of Nonresponse Propensity." ASA Section on Survey Research Methods.

Klar, N., & A. Donner (2001). "Current and future challenges in the design and analysis of cluster randomized trials." Statistics in Medicine, 20, 3729-3740.

Lessler, J.T. & W.D. Kalsbeek (1992). Nonsampling Errors in Surveys. New York: Wiley.

Little R.J.A. (1986). "Survey Nonresponse Adjustments for Estimates of Means." International Statistical Review, 54: 139-157.

Little, R.J.A. (1993). "Pattern-Mixture Models For Multivariate Incomplete Data." Journal of the American Statistical Association, 88, 125-134.

Little, R.J.A. (1994). "A Class Of Pattern-Mixture Models For Normal Incomplete Data." Biometrika, 81, 471-483.

Little, R. & T. Raghunathan (2004). This reference is to a course named "Statistical Analysis with Missing Data," presented by Roderick Little and Trivellore Raghunathan, May 4-5, 2004, Arlington,Virginia.

Little, R.J.A., & D.B. Rubin (2002). Statistical Analysis with Missing Data, 2nd Edition. Hoboken, NJ: John W. Wiley and Sons.

Lohr, S. L. (1999). Sampling: Design and Analysis. Pacific Grove, CA: Duxbury Press.

Manski, C. F. (1990). "Nonparametric Bounds on Treatment Effects." American Economic Review, Papers and Proceedings. 80: 319-323.

Mid-Atlantic Regional Education Laboratory (2008). "Research Design: The Effect of Connected Mathematics 2 (CM2) on the Math Achievement of Middle School Students in Selected Schools in the Mid-Atlantic Region: A Randomized Controlled Trial."

Moons, K.G.M., R.A.R.T. Donders, T. Stijnen, & F.E. Harrell (2006). "Using the outcome for imputation of missing predictor values was preferred." Journal of Clinical Epidemiology, 59, 1092–1101.

Murray, D.M. (1998). Design and Analysis of Group-Randomized Trials. New York: Oxford University Press.

Peugh, J.L., & C.K. Enders (2004), "Missing Data in Educational Research: A Review of Reporting Practices and Suggestions for Improvement." Review of Educational Research, 74 (4), 525-556.

Powell, J. L. (1994). "Estimation of Semiparametric Models." Handbook of Econometrics, Volume IV (edited by R. F. Engle & D. McFadden). New York: North-Holland.

Price, C., B. Goodson, & G. Stewart (2007). Technical Report, Volume I: Infant Environmental Exposures and Neuropsychological Outcomes at Ages 7 to 10 Years. Prepared for National Immunization Program, Centers for Disease Control and Prevention, Atlanta GA.

Puma, M., S. Bell, R. Cook, C. Heid, & M. Lopez (2005). Head Start Impact Study: First Year Findings. US Department of Health and Human Service, Administration for Children and Families: Washington, DC.

Reiter, J.P. & Raghunathan, T.E. and Kinney, S.K. (2006). "The Importance of Modeling the Sampling Design in Multiple Imputation for Missing Data," Survey Methodology, 32(2), p. 143.

Rosenbaum P.R. & D.B. Rubin (1983). "The Central Role of the Propensity Score in Observational Studies for Causal Effects." Biometrika, 70, 41-55

Rosenbaum P.R. & D.B. Rubin (1984). "Reducing Bias in Observational Studies Using Subclassification on the Propensity Score." Journal of the American Statistical Association, 79(387), 516-524.

Ross, S. M., J. Nunnery, & E. Goldfeder (2004). A randomized experiment on the effects of Accelerated Reader/Reading Renaissance in an urban school district: Preliminary evaluation report. Memphis, TN: The University of Memphis, Center for Research in Educational Policy.

Rubin, D.B. (1976), "Inference and Missing Data." Biometrika, 63, 581-92.

Rubin, D.B. (1987). Multiple Imputation for Nonresponse in Surveys. New York, NY: Wiley.

Rubin, D.B. (1996), "Multiple Imputation After 18+ Years," Journal of the American Statistical Association, 91, 473-489.

SAS9 Documentation: http://support.sas.com/documentation/onlinedoc/91pdf/index.html.

SAS Institute Inc. (2003). SAS/STAT 9, 9.1 User's Guide, Cary NC: SAS Institute, http://support.sas.com/onlinedoc/913/docMainpage.jsp.)

Schafer, J.L. (1999), "Multiple Imputation: A Primer," Statistical Methods in Medical Research, 8, 3-15.

Schafer, J.L. & J.W. Graham (2002). "Missing Data: Our View of the State of the Art." Psychological Methods, 7(2), 147-177.

Schochet, P.Z. (2005). Statistical Power for Random Assignment Evaluation of Education Programs. Mathematica Policy Research, Report Submitted to the Institute of Education Sciences, U.S. Department of Education.

Seftor, N.S., A. Mamun, & A. Schirm (2009). The Impacts of Regular Upward Bound on Postsecondary Outcomes 7-9 Years After Scheduled High School Graduation. Submitted by Mathematica Policy Research to the U.S. Department of Education, Policy and Program Studies Service.

Shafer, J.L. (1997). Analysis of Incomplete Multivariate Data. London: Chapman & Hall.

Smith, J. P. Todd (2005). "Does Matching Overcome LaLonde's Critique of Nonexperimental Estimators?" Journal of Econometrics. 125(1-2), 305-353.

Stolzenberg, R.M. & D.A. Relles (1990). "Theory Testing In A World Of Constrained Research Design: The Significance Of Heckmann's Censored Sampling Bias Correction For Nonexperimental Research." Sociological Methods & Research, 18, 395-415.

Stolzenberg, R.M. & D.A. Relles (1997). "Tools For Intuition About Sample Selection Bias And Its Correction." American Sociological Review, 62, 494-507.

U.S. Department of Education (2009), "Identifying and Implementing Educational Practices Supported By Rigorous Evidence: A User Friendly Guide" http://www.ed.gov/rschstat/research/pubs/rigorousevid/guide_pg5.html

U.S. Department of Education (2008), What Works Clearinghouse, Procedures and Standards Handbook Version 2, Washington: DC, December

van Buuren, S., H.C. Boshuizen, & D.L. Knook (1999), "Multiple Imputation of Missing Blood Pressure Covariates in Survival Analysis," Statistics in Medicine, 18, 681 -694.

Wilde, E.T. & R. Hollister (2002). "How Close Is Close Enough? Testing Nonexperimental Estimates of Impact against Experimental Estimates of Impact with Education Test Scores as Outcomes." Department of Economics, Swarthmore College.

Wolf, P., B. Gutmann, M. Puma, B. Kisida, L. Rizzo, & N. Eissa (2009). Evaluation of the DC Opportunity Scholarship Program: Impacts After Three Years (NCEE 2009-4050). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

Wood, A.M., I.R. White, & S.G. Thompson (2004), "Are Missing Outcome Data Adequatelt Handled? A Review of Published Randomized Controlled Trials in Major Medical Journals." Clinical Trials, 1, 368-76.

Top