Just over five years ago the Institute of Education Sciences (IES) and the National Science Foundation (NSF) released the Common Guidelines for Education Research and Development. The Guidelines provided the expected purposes, justifications, and contributions of various types of research aimed at improving our knowledge of interventions and strategies for improving teaching and learning. Since 2013, there has been increased attention to replication and reproducibility studies and their role in building the evidence base. In response to this interest and the importance of this work, the two organizations jointly issued the new Companion Guidelines on Replication and Reproducibility in Education Research to supplement the Common Guidelines for Education Research and Development. The companion document provides guidance on the steps researchers can take to promote corroboration, ensure the integrity of research, and extend the evidence base.
The Companion Guidelines identify principles to help education stakeholders design and report reproducibility and replication studies. These principles are consistent with and draw from guidelines provided by scientific and professional organizations, advisory committees, and have emerged in consultation with the field (e.g., Dettmer, Taylor, and Chhin, 2017; Subcommittee on Replicability and Science, 2015). The principles address three main areas – (1) replication and reproducibility at the proposal stage, (2) promoting transparency and openness in designing studies, and (3) considerations in the reporting of results.
Although the importance of reproducibility and replication studies for advancing scientific knowledge has been widely acknowledged, there are several challenges for researchers in our field, including actual or perceived disincentives (e.g., publication bias; reputation and career advancement norms; emphases on novel, potentially transformative lines of inquiry), implementation difficulties (especially for direct replications), and complexities of interpreting results (e.g., lack of consensus on what it means to “replicate” findings, low statistical power for replications). Grant funding agencies such as IES and NSF as well as education researchers have a role to play in addressing these challenges, promoting reproducibility and replication studies, and ultimately moving the field forward.
Why focus on replication and reproducibility?
The original Common Guidelines document did not substantively address issues pertaining to replication and reproducibility of research. Given the interest in and importance of this work, IES and NSF are providing additional clarity to the field in terms of common definitions and principles around replication and reproducibility.
Who is the audience for the Companion Guidelines on Replication and Reproducibility?
The primary audience for this document is education researchers; however, education research funding agencies and reviewers of grant applications are additional audiences for this document.
How should this document be used by researchers intending to apply for grants to conduct a reproducibility or replication study?
This document is meant to highlight the importance of replication and reproducibility studies and to offer guidelines to education stakeholders for thinking about and promoting reproducibility and replication in education research. It does not supersede the guidance provided in the requests for applications provided by IES and NSF.
What are the guiding principles for proposing replication and reproducibility studies?
The overarching principles at the grant proposal stage are as follows:
- Clarify how reproducibility or replication studies would build on prior studies and contribute to the knowledge base.
- Clearly specify any variations from prior studies and the rationale for such variations.
- Ensure objectivity (e.g., by conducting an independent investigation, or by putting safeguards in place if the original investigator(s) is involved).
In addition to these principles, the document also lays out principles for promoting transparency, open science, and reporting results.
Read the full Companion Guidelines here.
