High-Quality Instructional Materials
October 2020
Question
What does the research say about the effects of traditional and open source curriculum materials on student learning outcomes?
Ask A REL Response
Thank you for your request to our Regional Educational Laboratory (REL) Reference Desk. Ask A REL is a collaborative reference desk service provided by the 10 RELs that, by design, functions much in the same way as a technical reference library. Ask A REL provides references, referrals, and brief responses in the form of citations in response to questions about available education research.
Following an established REL Northwest research protocol, we conducted a search for evidence- based research. The sources included ERIC and other federally funded databases and organizations, research institutions, academic research databases, Google Scholar, and general Internet search engines. For more details, please see the methods section at the end of this document.
The research team has not evaluated the quality of the references and resources provided in this response; we offer them only for your reference. The search included the most commonly used research databases and search engines to produce the references presented here. References are listed in alphabetical order, not necessarily in order of relevance. The research references are not necessarily comprehensive and other relevant research references may exist. In addition to evidence-based, peer-reviewed research references, we have also included other resources that you may find useful. We provide only publicly available resources, unless there is a lack of such resources or an article is considered seminal in the topic area.
References
Agodini, R., Harris, B., Thomas, M., Murphy, R., & Gallagher, L. (2010). Achievement effects of four early elementary school math curricula: Findings for first and second graders (NCEE 2011-4001). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. https://eric.ed.gov
From the Abstract:
"National achievement data show that elementary school students in the United States, particularly those from low socioeconomic backgrounds, have weak math skills (National Center for Education Statistics 2009). In fact, data show that, even before they enter elementary school, children from disadvantaged backgrounds are behind their more advantaged peers in basic competencies such as number-line ordering and magnitude comparison (Rathburn and West 2004). Furthermore, after a year of kindergarten, disadvantaged students still have less extensive knowledge of mathematics than their more affluent peers (Denton and West 2002). This study examines whether some early elementary school math curricula are more effective than others at improving student math achievement in disadvantaged schools. A small number of curricula, which are based on different approaches for developing student math skills, dominate elementary math instruction--7 curricula make up 91 percent of those used by K-2 educators, according to a 2008 survey (Resnick et al. 2010). Little rigorous evidence exists to support one approach over another, however, which means that research does not provide educators with much useful information when choosing a math curriculum to use. The key findings in this report include the following: (1) Teachers used their assigned curriculum, and the instructional approaches of the four curriculum groups differed as expected; (2) Math instruction varied in other notable ways across the curriculum groups; (3) In terms of student math achievement, the curriculum used by the study schools mattered; and (4) The curriculum used in different contexts also mattered, and some of these findings are consistent with findings based on all students whereas others are not. Appendices include: (1) Data Collection and Response Rates; (2) Teacher-Reported Frequency of Implementing Other Curriculum-Specific Activities; (3) Glossary of Curriculum-Specific Terms; and (4) Constructing the Analyses Samples and Estimating Curriculum Effects."
Bhatt, R., & Koedel, C. (2012). Large-scale evaluations of curricular effectiveness: The case of elementary mathematics in Indiana. Educational Evaluation and Policy Analysis, 34(4), 391-412. https://eric.ed.gov4. Retrieved from https://economics.missouri.edu
From the Abstract:
"We use data from one of the few states where information on curriculum adoptions is available-Indiana-to empirically evaluate differences in performance across three elementary-mathematics curricula. The three curricula that we evaluate were popular nationally during the time of our study, and two of the three remain popular today. We find large differences in effectiveness between the curricula, most notably between the two that held the largest market shares in Indiana. Both are best characterized as traditional in pedagogy. We also show that the publisher of the least-effective curriculum did not lose market share in Indiana in the following adoption cycle; one explanation is that educational decision makers lack information about differences in curricular effectiveness."
Bhatt, R., Koedel, C., & Lehmann, D. (2013). Is curriculum quality uniform? Evidence from Florida. Economics of Education Review, 34, 107-121. https://eric.ed.gov. Retrieved from https://economics.missouri.edu
From the Abstract:
"We construct a large panel dataset of schools and districts in Florida to evaluate curricular effectiveness in elementary mathematics. A key innovation of our study is that we allow for curriculum quality to be non-uniform across various mathematics subtopics. We find evidence of variability in curricular effectiveness across different subtopics within the same curriculum. Our findings suggest that educational administrators should consider the topical performance of their various curricular alternatives when making adoption decisions."
Blazar, D., Heller, B., Kane, T. J., Polikoff, M., Staiger, D. O., Carrell, S., ... & Kurlaender, M. (2020). Curriculum reform in the Common Core era: Evaluating elementary math textbooks across six US states. Journal of Policy Analysis and Management, 39(4), 966–1019, https://eric.ed.gov. Retrieved from https://socialinnovation.usc.edu
From the Abstract:
"Can a school or district improve student achievement simply by switching to a higher quality textbook or curriculum? We conducted the first multi-textbook, multi-state effort to estimate textbook efficacy following widespread adoption of the Common Core State Standards (CCSS) and associated changes in the textbook market. Pooling textbook adoption and student test score data across six geographically and demographically diverse U.S. states, we found little evidence of differences in average achievement gains for schools using different math textbooks. We found some evidence of greater variation in achievement gains among schools using pre-CCSS editions, which may have been more varied in their content than post-CCSS editions because they were written for a broader set of standards. We also found greater variation among schools that had more exposure to a given text. However, these differences were small. Despite considerable interest and attention to textbooks as a low-cost, "silver bullet" intervention for improving student outcomes, we conclude that the adoption of a new textbook or set of curriculum materials, on its own, is unlikely to achieve this goal."
Charalambous, C. Y., & Hill, H. C. (2012). Teacher knowledge, curriculum materials, and quality of instruction: Unpacking a complex relationship. Journal of Curriculum Studies, 44(4), 443-466. Retrieved from http://www.fnbaldeo.com
From the Abstract:
"This paper draws on four case studies to perform a cross-case analysis investigating the unique and joint contribution of mathematical knowledge for teaching (MKT) and curriculum materials to instructional quality. As expected, it was found that both MKT and curriculum materials matter for instruction. The contribution of MKT was more prevalent in the richness of the mathematical language employed during instruction, the explanations offered, the avoidance of errors, and teachers' capacity to highlight key mathematical ideas and use them to weave the lesson activities. By virtue of being ambitious, the curriculum materials set the stage for engaging students in mathematical thinking and reasoning; at the same time, they amplified the demands for enactment, especially for the low-MKT teachers. The analysis also helped develop three tentative hypotheses regarding the joint contribution of MKT and the curriculum materials: when supportive "and" when followed closely, curriculum materials can lead to high-quality instruction, even for low-MKT teachers; in contrast, when unsupportive, they can lead to problematic instruction, particularly for low-MKT teachers; high-MKT teachers, on the other hand, might be able to compensate for some of the limitations of the curriculum materials and offer high-quality instruction. This paper discusses the policy implications of these findings and points to open issues warranting further investigation."
Gonzales, C., & Skarin, R. (2019). Do your materials meet English learners' needs? The Learning Professional, 40(2), 62-70. Retrieved from https://learningforward.org
From the Abstract:
"There is substantial evidence documenting the impact of high-quality, educative learning materials on teacher practices and student learning, yet this evidence has not translated widely into more equitable learning opportunities for all students. English learners tend to have less access to intellectually rich, grade-level content and instructional materials. With most English learners receiving instruction for math and English language arts alongside their non-English learner peers, it can no longer be the norm to simplify the content or rely on supplemental materials targeting English learners. Instructional materials must be well-designed to serve all students. What do materials that do this well look like, and how do teachers know if theirs measure up? The English Learners Success Forum, an organization working to enhance instructional materials to address the linguistic and cultural needs of English learners, convened national English learner experts, field-leading content providers, and educators to examine how to integrate English learner supports in English language arts and math materials. Guidelines and tools created to determine if materials measure up are presented in this article."
Hilton III, J., Larsen, R., Wiley, D., & Fischer, L. (2019). Substituting open educational resources for commercial curriculum materials: Effects on student mathematics achievement in elementary schools. Research in Mathematics Education, 21(1), 60-76. https://eric.ed.gov. Retrieved from https://www.tandfonline.com
From the Abstract:
"Open Educational Resources (OER) have the potential to replace commercial learning materials in education. An empirical examination of this potential was conducted, comparing the end-of-year mathematics test results of 12,110 elementary school students clustered within 95 schools from five school districts in the state of Washington in the United States of America. Of this group, 6796 students used open learning materials, and 5314 used commercial educational resources. When three years of test scores were considered, there were no statistically significant differences in the exam scores of students who used open versus commercial curriculum materials. The lack of statistical significance may have practical significance, demonstrating that OER can replace conventional materials without impacting student performance, while potentially reducing costs and allowing for local modification."
Hirsch, E., & Allison, C. (2020). Do your materials measure up? The Learning Professional, 41(4), 28-31. https://eric.ed.gov. Retrieved from https://learningforward.org
From the Abstract:
"Districts and states have worked around the clock for the past several months to put new structures in place for students to learn safely during an unprecedented crisis. Instructional materials matter for student success. They mattered before the COVID-19 crisis, and they will matter even more as schools transition into new teaching formats this fall. As educators prepare for an uncertain return to school, many may feel pressure to throw out what they know about curriculum and rush toward buying materials that are high-tech and digital. Since 2015, EdReports has worked with educator reviewers from across the country to publish reports for more than 700 math, English language arts, and science grade-level materials. In response to the COVID-19 crisis, EdReports has held conversations with dozens of state and district leaders, classroom teachers, and instructional coaches to identify how best to support immediate and long-term needs to accelerate learning. Based on what they heard, this article presents five recommendations to educators on how to identify, build capacity for, and leverage high-quality instructional materials to support students, whether they are learning in the classroom, at home, or in a hybrid setting this fall: (1) When it comes to curriculum, make content the top priority; (2) Leverage quality curriculum to accelerate learning; (3) Take time to analyze your remote learning options; (4) Beware of marketing hype and sales spin; and (5) High-quality professional learning will be key."
Jackson, K., & Makarin, A. (2018). Can online off-the-shelf lessons improve student outcomes? Evidence from a field experiment. American Economic Journal: Economic Policy, 10(3), 226-54. Retrieved from http://www.eief.it
From the Abstract:
"Many websites now warehouse instructional materials designed to be taught by teachers in a traditional classroom. What are the potential benefits of the new resources? We analyze an experiment in which we randomly give middle school math teachers access to existing high-quality, off-the-shelf lessons, and in some cases, support to promote their use. Teachers receiving access alone increased students' math achievement by a marginally significant 0.06 of a standard deviation. Teachers who received access and support increased students' math achievement by 0.09 of a standard deviation. Weaker teachers experience larger gains, suggesting that these lessons substitute for teacher skill or efforts. The online materials are more scalable and cost effective than most policies aimed at improving teacher quality, suggesting that, if search costs can be overcome, there is a real benefit to making high-quality instructional materials available to teachers on the Internet."
Kablan, Z., Topan, B., & Erkan, B. (2013). The effectiveness level of material use in classroom instruction: A meta-analysis study. Educational Sciences: Theory and Practice, 13(3), 1638-1644. https://eric.ed.gov
From the Abstract:
"In this study, the aim was to combine the results obtained in independent studies aiming to determine the effectiveness of material use. The main question of the study is: "Does material use in classroom instruction improve students' academic achievements?" To answer this question, the meta-analysis method was employed. Meta-analysis is the method employed in order to statistically analyze the quantitative data collected in independent and multiple studies carried out on similar topics, and to reach a general judgment regarding the results of these studies. Certain criteria were used in order to decide which researches would be included in the meta-analysis. Based on these criteria, it was decided to include 57 experimental studies in the meta-analysis. As a result of this analysis; it was determined that the material use in classroom instruction has a positive effect in terms of academic achievement, and that this effectiveness level does not differ with respect to stages of education, course types and material types."
Koedel, C., Li, D., Polikoff, M. S., Hardaway, T., & Wrabel, S. L. (2017). Mathematics curriculum effects on student achievement in California. Aera Open, 3(1), https://eric.ed.gov
From the Abstract:
"We estimate relative achievement effects of the four most commonly adopted elementary mathematics textbooks in the fall of 2008 and fall of 2009 in California. Our findings indicate that one book, Houghton Mifflin's "California Math," is more effective than the other three, raising student achievement by 0.05 to 0.08 student-level standard deviations of the Grade 3 state standardized math test. We also estimate positive effects of "California Math" relative to the other textbooks in higher elementary grades. The differential effect of "California Math" is educationally meaningful, particularly given that it is a schoolwide effect and can be had at what is effectively zero marginal cost.
Methods
Keywords and Search Strings: The following keywords, subject headings, and search strings were used to search reference databases and other sources: "Instructional materials", "Curriculum materials", (Effect OR impact), "High-quality", ("Student learning" OR "student performance" OR "academic achievement" OR "student outcomes" OR "student achievement")
Databases and Resources: We searched ERIC for relevant resources. ERIC is a free online library of more than 1.6 million citations of education research sponsored by the Institute of Education Sciences (IES). Additionally, we searched Google Scholar and EBSCO databases (Academic Search Premier, Education Research Complete, and Professional Development Collection).
Reference Search and Selection Criteria
When we were searching and reviewing resources, we considered the following criteria:
Date of publications: This search and review included references and resources published in the last 10 years.
Search priorities of reference sources: Search priority was given to study reports, briefs, and other documents that are published and/or reviewed by IES and other federal or federally funded organizations, as well as academic databases, including ERIC, EBSCO databases, and Google Scholar.
Methodology: The following methodological priorities/considerations were given in the review and selection of the references:
- Study types: randomized control trials, quasi experiments, surveys, descriptive data analyses, literature reviews, and policy briefs, generally in this order
- Target population and samples: representativeness of the target population, sample size, and whether participants volunteered or were randomly selected
- Study duration
- Limitations and generalizability of the findings and conclusions
This memorandum is one in a series of quick-turnaround responses to specific questions posed by stakeholders in Alaska, Idaho, Montana, Oregon, and Washington, which is served by the Regional Educational Laboratory (REL) Northwest. It was prepared under Contract ED-IES-17-C-0009 by REL Northwest, administered by Education Northwest. The content does not necessarily reflect the views or policies of IES or the U.S. Department of Education, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. Government.