Skip Navigation
Effectiveness of Reading and Mathematics Software Products

NCEE 2009-4041
February 2009

Study Design

The second year of the study was shaped by the structure of its first year. For the first year, the study team identified 16 products for the study, as noted above, and recruited 33 school districts to implement the products. In turn, districts identified a total of 132 schools to implement the selected products, and the study randomly assigned 428 volunteering teachers in the schools to either use or not use the products in their classrooms. Students were allocated to classrooms by their schools in whatever manner schools conventionally used. Students were tested in these classrooms in both the fall and spring of the 2004-2005 school year (a total of 9,458 students). The study also observed classrooms at three points during the school year, and supplemented the test scores and observational data with data about students from school records, a questionnaire completed by teachers in the study, and school data from the Common Core of Data maintained by the National Center for Education Statistics (NCES).

Collecting a second year of student data, while staying within resource constraints, required modifying the study in five ways compared to the first year. Products that had been implemented in only a few schools were dropped, classrooms were not observed in the second year, one treatment classroom and one control classroom were randomly sampled within schools that had more than one, some districts provided their test scores rather than having the study team test students, and some items were collected from school records. These changes in the data collection strategy reduced the amount of data collected in the 2005-2006 school year, and precluded the study from exploring the same range of questions it explored in the first year. The second year of the study included 10 products, 23 districts, 77 schools, 176 teachers, and 3,280 students.

The second-year study also should be understood as two different but related sub-studies. One objective of the second-year study is to assess whether the experience of a second year of use of software products increased the effects products had on student test scores. Another objective is to report on the effectiveness of individual software products in raising student test scores. Addressing the first objective requires restricting the sample to teachers who participated in both the first and second years of the study. Addressing the second objective requires data from teachers who participated in either the first or second year. Because the samples of teachers and students differ between the two substudies, estimates of sample characteristics and product effects also differ.