Skip Navigation
Technical Methods Report: Using State Tests in Education Experiments

NCEE 2009-013
November 2009

Assessing the Feasibility of Collecting State Test Data

The single most-expensive aspect of an RCT is typically data collection. This is especially true when the evaluation calls for the administration of an external student achievement test, which can be expensive in terms of test materials and scoring, school participation incentives, and the effort required to recruit schools willing to administer yet another test. Therefore, the feasibility and cost of gaining access to state test data should be evaluated relative to the feasibility and cost of collecting external assessment data. In general, the cost of obtaining state test data can be expected to be far lower. In this section, we nevertheless highlight factors that can make collecting state test score data somewhat difficult and costly.

Gaining access to student-level test data is not an easy task, nor should it be given the need to protect students' personal information. The Family Educational Rights and Privacy Act of 1974 (FERPA) prohibits educational agencies from disclosing students' personal information to other agencies, except under specific conditions (U.S. Department of Education 2008). In some cases, written permission from the parent or guardian of each student is required to disclose personally identifiable data including individual test scores. The difficulty of accessing existing test scores depends on several factors, including (1) whether a state or school district sponsors the study, (2) the dispersion and mobility of students in the study sample, (3) the extent to which the data can be obtained from state education agencies rather than individual districts, and (4) the complexity of the state or district research application process.

Study Sponsorship. Securing school records without parental consent is sometimes possible when the study is sponsored by a state or district. FERPA allows data to be disclosed without written permission from a parent so long as the evaluator is "conducting studies for, or on behalf of, educational agencies or institutions" (FERPA, 20 U.S.C. § 1232g; 34 CFR Part 99). The educational agencies covered by this provision include school districts, state education agencies, postsecondary institutions, and the U.S. Department of Education. However, states and districts have varying interpretations of this FERPA provision. Hence, even after addressing FERPA requirements, it is often necessary to satisfy additional state or district rules restricting the release of school records.

Dispersion and Mobility of the Study Population. For studies involving a highly dispersed or mobile student population it might be necessary to secure records from a larger number of districts and states, which can increase the costs of data collection. The dispersion of the study sample hinges on the intervention and study design. For example, an evaluation of charter schools in a suburban area that involves random assignment of applicants might result in a highly dispersed control group spread over multiple districts. By contrast, an experiment that randomly assigns an intervention to classrooms within a set of schools, or even schools within a district, is likely to have a more-concentrated sample. The mobility of the sample following the point of random assignment and the length of the study's follow-up period will also affect the dispersion of the sample.

Contact Point for Data Collection. It is generally easier and less costly to obtain records for a high proportion of the sample when these are available at the state level rather than from individual school districts; this is especially the case when the study sample is dispersed and mobile. Although many states are currently developing statewide databases with individual-level data, some states still cannot provide such data. In cases in which state databases do not include the data required for the study, a researcher will likely need to submit data requests to individual districts or schools participating in the research. However, even under this scenario, the process of obtaining state test score data is likely to be less costly than administering an external assessment to individual students in an expanded sample of schools.

Research Application Process and Requirements. The cost and time involved in securing data from any state or district also depends on the research application process. Typically, districts and states require a written data request in which the researcher states what data are being requested, what research questions will be addressed through analyses of the data, and how data security and confidentiality will be ensured. Some states and/or districts require compensation to cover the costs of processing and filling the data requests. The length of time required for review and approval of the request and transfer of the data varies, but typically ranges between one and six months. In general, researchers should expect negotiations with individual states or districts to take at least three months. These negotiations often require the establishment of formal data use/sharing agreements and the establishment of mechanisms for the transfer and storage/backup of sensitive data.

Top