Setting
The study was conducted in three Rocketship Education charter schools located in San Jose,
California.
Study sample
The study sample included all kindergarten and first-grade students at the three schools that
participated in the study, a total of 557 students after attrition from a sample of 583 who were
randomly assigned. The number of classrooms included in the study is not specified. Within
grade levels, students were randomly assigned to either the intervention or comparison groups
at a 4 to 1 ratio. In the baseline sample, 53% of students were female, 87% were Hispanic,
81% were English language learners, 88% were eligible for free or reduced-price meals, 4%
were classified as special education, and 10% participated in Response to Intervention (RtI)
services.
Intervention Group
The experiment was conducted from mid-October through mid-February during the 2010–11
school year. Intervention students were scheduled to receive 20 to 40 minutes of DreamBox
Learning mathematics instruction per day; usage statistics show that students averaged 21.8
hours of usage over the course of the study, or approximately 16 minutes per day. Instructional
sessions were conducted in a computer lab. The authors noted that the low-achieving
students who were assigned to receive RtI services were scheduled to receive 45 minutes of
DreamBox Learning instruction in their after-school RtI programming, regardless of intervention
status. For the 42 intervention group students who were assigned to RtI services, this 45
minutes was in addition to the DreamBox Learning instruction provided during the school day,
for a total of 26.5 hours of usage over the course of the study on average. Progress and use
information provided by the DreamBox Learning software was not used to modify face-to-face
mathematics instruction for either the intervention or comparison group.
Comparison Group
Students in the comparison condition received no additional mathematics instruction. However,
they received additional literacy instruction via an online program during the time and in
the same location as intervention group students using the DreamBox Learning software. The
11 students in the comparison condition who were assigned to RtI services were scheduled
to receive 45 minutes of DreamBox Learning instruction in their after-school RtI programming;
the authors found that these comparison condition students averaged 5.1 hours of program
usage over the course of the study.
Outcome descriptions
The study used math test scores from the MAP assessment developed by the Northwest Evaluation
Association (NWEA). The study reports the overall math score, as well as five subtest
scores, for problem solving, number sense, computation, measurement and geometry, and
statistics and probability. Scores were scaled using the RIT scale, “which is scaled using the
Item Response Theory (IRT) and has the same meaning regardless of the grade of the student”
(as cited in Wang & Woodworth, 2011, p. 3). The schools administered the assessment in September
2010 (pretest) and January/February 2011 (posttest). For a more detailed description of
this outcome measure, see Appendix B.
Support for implementation
DreamBox Learning “does not prescribe a specific role for teachers” (Wang & Woodworth,
2011, p. 3). The computer labs in which students received DreamBox Learning instruction
were run by lab coordinators, noncredentialed hourly staff who played a minimal role in
instruction. The authors noted that lab coordinators sometimes may have been out of the
computer lab, at which times the students would be supervised by support staff.