Skip Navigation
archived information
Skip Navigation

Back to Ask A REL Archived Responses

REL Midwest Ask A REL Response

February 2020


What research is available on evaluating the quality of open educational resources?


Following an established Regional Educational Laboratory (REL) Midwest protocol, we conducted a search for research reports, descriptive studies and policy overviews on evaluating the quality of open educational resources. For details on the databases and sources, keywords, and selection criteria used to create this response, please see the Methods section at the end of this memo.

Below, we share a sampling of the publicly accessible resources on this topic. References are listed in alphabetical order, not necessarily in order of relevance. The search conducted is not comprehensive; other relevant references and resources may exist. For each reference, we provide an abstract, excerpt, or summary written by the study’s author or publisher. We have not evaluated the quality of these references, but provide them for your information only.

Research References

Achieve, Inc. (2014). Toolkit for evaluating alignment of instructional and assessment materials to the Common Core State Standards. Washington, DC: Author. Retrieved from

From the ERIC abstract: “In joint partnership, Achieve, The Council of Chief State School Officers, and Student Achievement Partners have developed a Toolkit for Evaluating the Alignment of Instructional and Assessment Materials to the Common Core State Standards. The Toolkit is a set of interrelated, freely available instruments for evaluating alignment to the CCSS; each tool in the Toolkit supports the expectations in the CCSS and derives from the Publishers’ Criteria for the Common Core State Standards.”

Council of Chief State School Officers. (2014). State of the states: Open educational resources (OER) in K–12 education. Washington, DC: Author. Retrieved from

From the executive summary: “Several states expressed an interest to the Council of Chief State School Officers (CCSSO) in exploring the development and dissemination of this digital content in the form of open educational resources (OER), which are teaching materials licensed for free use and repurposing. In response to that interest, CCSSO conducted a survey of states in May 2014 to collect information about the current ‘state of the states’ as it relates to OER. This report highlights those results and provides examples of ongoing work related to OER. This report also strives to inform and connect those chiefs and senior level state education agency (SEA) staff interested in OER to encourage cross-state understanding and collaboration with the potential to both share content and resources in this space.”

Council of Chief State School Officers. (2018). Navigating the new curriculum landscape: How states are using and sharing open educational resources. Washington, DC: Author. Retrieved from

From the description:Navigating the New Curriculum Landscape: How States are Using and Sharing Open Educational Resources provides a deeper look into how states are supporting OER implementation, and what lessons can be learned from the progress they are making. This CCSSO and New America report spotlights examples of new and different approaches for promoting and sustaining open, relevant, and high-quality instructional materials. It concludes with five key takeaways for state leaders to consider as they improve the quality and affordability of the materials teachers are using every day. While there is wide variation in approaches, states are deeply integrating OER within their broader efforts to ensure teachers and school leaders have access to high-quality curriculum and instructional materials. State strategies range from including OER within existing state reviews of curricular materials to developing and curating new openly-licensed resources from which schools or districts can choose. Across all states, CCSSO and New America we found evidence that frequent collaboration, communities of practice, and intentional communication have been key levers to spread and build upon this work.”

Hilton, J., III, Larsen, R., Wiley, D., & Fischer, L. (2019). Substituting open educational resources for commercial curriculum materials: Effects on student mathematics achievement in elementary schools. Research in Mathematics Education, 21(1), 60–76. Retrieved from

From the ERIC abstract: “Open Educational Resources (OER) have the potential to replace commercial learning materials in education. An empirical examination of this potential was conducted, comparing the end-of-year mathematics test results of 12,110 elementary school students clustered within 95 schools from five school districts in the state of Washington in the United States of America. Of this group, 6796 students used open learning materials, and 5314 used commercial educational resources. When three years of test scores were considered, there were no statistically significant differences in the exam scores of students who used open versus commercial curriculum materials. The lack of statistical significance may have practical significance, demonstrating that OER can replace conventional materials without impacting student performance, while potentially reducing costs and allowing for local modification.”

Kimmons, R. (2015). OER quality and adaptation in K-12: Comparing teacher evaluations of copyright-restricted, open, and open/adapted textbooks. International Review of Research in Open and Distributed Learning, 16(5), 39–57. Retrieved from

From the ERIC abstract: “Conducted in conjunction with an institute on open textbook adaptation, this study compares textbook evaluations from practicing K-12 classroom teachers (n = 30) on three different types of textbooks utilized in their contexts: copyright-restricted, open, and open/adapted. Copyright-restricted textbooks consisted of those textbooks already in use by the teachers in their classrooms prior to the institute, open textbooks included alternatives from CK-12 and OpenStax, and open/adapted consisted of open textbooks that the teachers devoted time to adapting to their individual needs. Results indicate that open/adapted textbooks were evaluated as having the highest quality, and that open textbooks were of higher quality than copyright-restricted textbooks. Though some factors of quality might be influenced by cost differences (e.g., timeliness and the ability to adopt updated textbooks), results reveal that open and open/adapted textbooks may do a better job of meeting the needs of K-12 teachers in a variety of ways that may not be captured through traditional approaches to quality assurance. This study marks an early step in exploring the quality of K-12 open educational resources (OER) and the use of practicing teachers as authentic evaluators of textbooks for their local contexts.”

State Education Technology Directors Association. (2015). Ensuring the quality of digital content for learning: Recommendations for K12 education. Burnie, MD: Author. Retrieved from

From the overview: “This paper complements SETDA’s prior digital transition policy briefs by examining strategies for ensuring digital content quality, including exploration of the specific quality-control challenges and opportunities associated with open educational resources (OER). Specifically, this paper describes: 1) digital content’s unique characteristics; 2) traditional state/district instructional materials quality-review practices; and 3) recommendations to inform/strengthen state strategies for ensuring digital resource quality.”

State Educational Technology Directors Association. (2019). State K12 instructional materials leadership trends snapshot. Burnie, MD: Author. Retrieved from

From the ERIC abstract: “This report summarizes current state policies and practices in the selection and implementation of digital instructional materials. It is based on the 2019 updates to the Digital Instructional Materials Acquisition Policies for States (DMAPS) online portal. Since its inception in 2015, DMAPS has been updated annually and expanded to include new topic areas. This trends snapshot includes details regarding: (1) digital learning policies and practices including state digital learning plans and standards; (2) state definitions for personalized learning; (3) policies and practices related to instructional materials acquisition, procurement and implementation; (4) professional learning opportunities states are providing teachers around the selection, creation and implementation of digital instructional materials; and (5) accessibility policies and practices.”

Southern Regional Education Board. (2017). Alignment of instructional materials: Trends in state efforts. Atlanta, GA: Author. Retrieved from

From the description: “High-quality instructional materials that are aligned to state standards can help boost student achievement and reduce variability in the quality of instruction across classrooms. In Alignment of Instructional Materials, SREB examines the level of support and leadership SREB states provided in 2014-15 and 2015-16 to help educators identify and use high-quality textbooks, lesson plans and other materials aligned to state K-12 standards.”

U.S. Department of Education, Office of Educational Technology. (2016). #GoOpen district launch packet. Washington, DC: Author. Retrieved from

From the ERIC abstract: “Across the country, districts are choosing to #GoOpen and transitioning to the use of openly licensed educational resources to improve student learning in their schools. Openly licensed educational resources enable districts to reallocate significant funds currently spent on inflexible, static learning materials to resources and activities that accelerate the transition to digital learning. These include implementing new professional learning programs for teachers, developing a robust technology infrastructure to support digital learning, and funding new leadership roles for educators who curate and create openly licensed educational materials. The #GoOpen District Launch Packet is designed for districts that have decided to implement a systematic approach to incorporating openly licensed educational resources into their curriculum by becoming a #GoOpen District.”

Whitehill, J., Aguerrebere, C., & Hylak, B. (2019, July). Do learners know what’s good for them? Crowdsourcing subjective ratings of OERs to predict learning gains. Paper presented at the International Conference on Educational Data Mining, Montreal, Canada. Retrieved from

From the ERIC abstract: “We explored how learners’ ‘subjective ratings’ of open educational resources (OERs) in terms of how much they find them ‘helpful’ can predict the actual ‘learning gains’ associated with those resources as measured with pre- and post-tests. To this end, we developed a probabilistic model called GRAM (Gaussian Rating Aggregation Model) that combines subjective ratings from multiple learners into an aggregate quality score of each resource. Based on an experiment we conducted on Mechanical Turk (n = 304 participants with m = 17 math tutorial videos as resources), we found that aggregated subjective ratings are highly (and stat. sig.) predictive of the resources’ average learning gains, with Pearson correlation of 0.78. Moreover, when predicting average learning gains of ‘new’ learners, subjective scores were still predictive (Pearson correlation of 0.49) and attained higher prediction accuracy than a model that directly uses pre- and post-test data to estimate learning gains for each resource. These results have potential implications for large-scale learning platforms (e.g., MOOCs, Khan Academy) that assign resources (tutorials, explanations, hints, etc.) to learners based on the expected learning gains.”

Additional Organizations to Consult

Achieve, Inc. –

From the website: “Achieve is an independent, nonpartisan, nonprofit education reform organization dedicated to working with states to raise academic standards and graduation requirements, improve assessments, and strengthen accountability.”

Achieve OER Institute:

Achieve OER Rubrics:

Council of Chief State School Officers (CCSSO) –

From the website: “The Council of Chief State School Officers (CCSSO) is a nonpartisan, nationwide, nonprofit organization of public officials who head departments of elementary and secondary education in the states, the District of Columbia, the Department of Defense Education Activity, the Bureau of Indian Education and the five U.S. extra-state jurisdictions.”

OER Resources for Policy Makers:

From the website: “Resources for policymakers at the district, state and national level interested in the adoption and implementation of open educational resources (OER).”

Department of Education’s Office of Educational Technology –

From the website: “The U.S. Department of Education’s #GoOpen initiative supports States and districts choosing to transition to the use of openly licensed educational resources to transform teaching and learning. Learn what it takes to be a #GoOpen District or State.”

Phase 4: Ensuring High Quality Learning Resources –

From the website: “When using static, traditional textbooks, most districts rely on publishers to ensure that the materials are high quality and aligned with rigorous college and career ready standards. In contrast, when you transition to openly licensed educational resources, the onus is on you to ensure that the resources are of a high quality and provide the learning experiences necessary to exceed learning goals for all students. This is a serious, new responsibility for a district that is accustomed to buying proprietary resources from third parties. #GoOpen Districts thoughtfully and deliberately create quality assurance processes with multiple checks to ensure that materials match the needs and expectations of all stakeholders.”

State Educational Technology Directors Association: Digital Instructional Materials –

From the website: “This online database provides state and territory policies and practices related to the acquisition and implementation of digital instructional materials in K12 education. ”

Site highlights:

  • Overview of states policies/practices
  • State trends via heat map
  • Compare up to 5 states by topic
  • Individual state profiles
  • Print and download options
  • District exemplars
  • RFPs”


Keywords and Search Strings

The following keywords and search strings were used to search the reference databases and other sources:

  • “Alignment of instructional materials”

  • “Digital instructional materials”

  • “Open educational resources”

  • “Open educational resources” “educational quality”

  • “Open educational resources” “elementary education”

  • “Open educational resources” rubric

  • “Open educational resources” “secondary education”

Databases and Search Engines

We searched ERIC for relevant resources. ERIC is a free online library of more than 1.6 million citations of education research sponsored by the Institute of Education Sciences (IES). Additionally, we searched IES and Google Scholar.

Reference Search and Selection Criteria

When we were searching and reviewing resources, we considered the following criteria:

  • Date of the publication: References and resources published over the last 15 years, from 2005 to present, were included in the search and review.

  • Search priorities of reference sources: Search priority is given to study reports, briefs, and other documents that are published or reviewed by IES and other federal or federally funded organizations.

  • Methodology: We used the following methodological priorities/considerations in the review and selection of the references: (a) study types—randomized control trials, quasi-experiments, surveys, descriptive data analyses, literature reviews, policy briefs, and so forth, generally in this order, (b) target population, samples (e.g., representativeness of the target population, sample size, volunteered or randomly selected), study duration, and so forth, and (c) limitations, generalizability of the findings and conclusions, and so forth.
This memorandum is one in a series of quick-turnaround responses to specific questions posed by educational stakeholders in the Midwest Region (Illinois, Indiana, Iowa, Michigan, Minnesota, Ohio, Wisconsin), which is served by the Regional Educational Laboratory (REL Midwest) at American Institutes for Research. This memorandum was prepared by REL Midwest under a contract with the U.S. Department of Education’s Institute of Education Sciences (IES), Contract ED-IES-17-C-0007, administered by American Institutes for Research. Its content does not necessarily reflect the views or policies of IES or the U.S. Department of Education nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. Government.