Co-Principal Investigators: Nicole Shechtman, Mingyu Feng, Corinne Singleton

Purpose: The purpose of this project was to evaluate the efficacy of a fully developed digital mathematics curriculum (RM-CC5). In particular, the project evaluated a full-year, grade 5 curriculum that strongly implements blended and adaptive learning. RM-CC5 uses technology to give students and teachers instant feedback, implement a differentiated instruction model, provide supports for learning, and offer engaging activities. The curriculum is oriented to developing students' understanding of the key topics on the pathway to algebra; topics include place value system, fractions, rational numbers, geometric measurement, and graphing points in a coordinate plane, and is aligned with recently-adopted state standards. Researchers in this project contrasted RM-CC5's digital approach to business-as-usual curricula, which leave to teachers' discretion differentiated instruction, integration of technology, and selection of additional student supports – and do not prioritize blended and adaptive learning.

Project Activities: In this study, researchers carried out a randomized field trial in the state of West Virginia to test whether RM-CC5, a full-year digital core curriculum improves mathematics achievement for grade 5 students. In addition, the team examined how duration of intervention and intensity of use interacted with student outcomes. They also investigated the quality and fidelity of implementation.

Key Outcomes: The main findings of this efficacy study were:

• Treatment teachers implemented the RM-CC5 curriculum with adequate fidelity to the model. Across all treatment schools, students were logged into RM-CC5 for an overall estimated average of almost an hour a day.
• Mathematics achievement on the West Virginia end-of-year state assessment in the treatment group was no higher than achievement in the business-as-usual control group.
• Teachers were able to make dramatic changes in the organization of classrooms with a complete set of digital learning materials, a robust and well-designed program of teacher training and support, and enough time to implement them.
• Implementation of adaptive, digital curricula is complex and does not always reflect the needs of both the teachers and students.
• There were tensions between adaptive learning and coverage of all learning objectives measured on an end-of-year state assessment, between the degree of variation in student pace and a teacher's ability to interact with all students, and in balancing among the amount of time and attention teachers dedicated to full-group, small-group, and individualized instruction.

Products:

• The researchers produced information about the efficacy of Reasoning Mind compared to typical instructional practice, and about the implementation needs and challenges of digital curricula that emphasize adaptive, blended approaches
• These findings have been shared through peer-reviewed publications and professional conferences.

Structured Abstract

Setting: This study was completed in 46 schools with fifth-grade classrooms in West Virginia. The state had already approved use of RM-CC5, established infrastructure for widespread adoption of a digital curriculum, and was in the process of adopting new standards and curricula.

Sample: A total of 56 schools were recruited and randomized to condition, and 23 treatment and 23 control schools completed the study. All fifth-grade classrooms in the treatment and control schools participated. Across these schools, achievement data was analyzed for 1,919 fifth-grade students. Student populations at participating West Virginia schools were predominantly White, rural and low-income.

Intervention: RM-CC5 focuses on key topics on the pathway to algebra, including place value system, fractions, rational numbers, geometric measurement, and graphing points in a coordinate plane. The intervention dramatically changes the role of the teacher. Rather than present instruction to the full class, teachers focus on intervening with specific students based on data from RM-CC5. Students receive instruction from a computer and work independently in RM-CC5 to solve mathematics problems. The intervention provides dynamic feedback (e.g., accuracy of responses, hints) and implements an adaptive instruction model based on each student's performance. The system incorporates motivational components, such as games. In addition, teachers receive up to 60 hours of professional development and implementation support, such as understanding progress reports on individual students and entire classroom.

Research Design and Methods: The study was a 2-year randomized controlled trial (RCT), implemented in SY2014–15 and SY2015–16. Schools were randomly assigned to a treatment or control condition and asked to participate for two full school years. Treatment teachers implemented RM-CC5 as their core grade 5 mathematics program; control teachers implemented their business-as-usual resources (see below). The research focused on findings from the second "measurement" year.

Control Condition: Teachers of classrooms in the control condition followed their typical practice (e.g., they used existing curriculum resources and supplemental technologies), and teachers received professional development according to district policies and offerings. Control schools were asked to not adopt a blended or differentiated instruction-focused digital curriculum product during the study. For fairness across experimental groups, control schools were offered RM curricula for their second-grade classrooms (i.e., that were not participating in the research).

Key Measures: Student achievement was measured with end-of-year grade 5 mathematics assessment, the West Virginia General Student Assessment (WVGSA). The WV Department of Education provided the data, as well as students' prior year grade 4 mathematics WVGSA. Students' productive dispositions toward mathematics were also measured using the Attitudes Toward Mathematics Inventory, and teachers' mathematical knowledge for teaching was measured using either the Pedagogical Content Knowledge Test or released items from the Learning Mathematics for Teaching Project. Treatment and control teachers completed electronic teacher logs. Data measuring student problem solving, use of remote tutoring, and teachers' use of reports was gathered directly from the RM system records. Measures of implementation were collected via classroom observations, stakeholder interviews, and teacher survey.

Data Analytic Strategy: Data analyses for the main treatment effect included a hierarchical linear regression model to examine mean differences in grade 5 achievement scores between students in treatment and control conditions, nested within schools, with prior (grade 4) achievement as a covariate. Secondary analyses examined differential impact for students at different strata of prior achievement, and across genders. Exploratory analyses examined differences within and across treatment classrooms of students' use of remote tutoring; teachers' use of RM reports to adjust instruction; students' productive time on task; and implementation compliance of both students and teachers. Regression models were conducted on achievement measures to examine potential associations between achievement and use metrics; these analyses were exploratory and did not afford inferences about causality.

Products and Publications

Journal article, monograph, or newsletter

Roschelle, J., Gaudino, S., and Darling, S. (2016). Designing for Consistent Implementation of a 5th Grade Digital Math Curriculum. International Journal of Designs for Learning, 7(2).

Shechtman, N., Roschelle, J., Feng, M., and Singleton, C. (2019). An Efficacy Study of a Digital Core Curriculum for Grade 5 Mathematics. AERA Open, 5(2), 2332858419850482.

Proceeding

Herman, P., Bumgardner, K., Shechtman, N., and Feng, M. (2018). Teaching with a Fully Digital, Year-long Math Program: Learning Science Futures on the Front Line. In Kay, J. and R. Luckin (Eds.), Rethinking Learning in the Digital Age: Making the Learning Sciences Count,13th International Conference of the Learning Sciences (ICLS) 2018, Volume 3, (pp. 632-639). London, UK: International Society of the Learning Sciences.