The reporting of results from assessment surveys relies on accurate standard error estimates to make statistical inferences, based on an official set of plausible values that nearly all data users rely on for calculating standard errors used in their reports. When analysts sample their own sets of plausible values (in order, for example, to condition on new variables that had been left out of the official conditioning model), they find that they cannot reproduce many of the same statistical results found using the official sample of plausible values. This phenomenon is due to reliance on the Rubin-Mislevy method of calculating standard errors, which is based on sufficient statistics from each of a modest number of sets of plausible values. The method is unbiased, but the variance of standard errors produced by this method can be reduced with a newly developed alternative method that bases the calculation of standard errors on sufficient statistics for examinees, rather than for sets of plausible values. The purpose of this project is to further the development, and assess the quality, of this alternative statistical method for the use of posterior distributions in assessment surveys—a method that is more stable across different samples of plausible values and across different population groups and jurisdictions than the current Rubin-Mislevy method.
For the proposed project, the PI will apply and extend his preliminary work on developing an examinee-based, hierarchical-data method for calculating standard errors of means. The main approach will be secondary analyses of existing national assessment data bases, including the National Assessment of Educational Progress (NAEP) mathematics, reading, science, and writing assessments. The PI will conduct additional analyses with data from the 2003 National Assessment of Adult Literacy. The key outcomes will be development of a hierarchical-data method for calculating standard errors of percentages above cut scores, and the comparisons of standard error estimates derived from the completed-data method with those from the hierarchical data method. The latter part of the grant period will be devoted to dissemination and vetting of the new method with the National Center for Education Statistics, NAEP technical staff, NAEP advisory committees, and NAEP contractors.
