Standards for Excellence in Education Research

About

Since its founding in 2002, the Institute of Education Sciences (IES) has supported rigorous evidence-building about education policy, programs, and practices. IES introduced the Standards for Excellence in Education Research, or SEER, in 2018. SEER is designed to complement existing standards for internal validity developed by IES’s What Works Clearinghouse, emphasizing additional factors that can make research transformational.

The 2027 Standards (SEER 27) are organized into 4 domains. 

Domain 1. Objectivity and Scientific Integrity

Domain 2. Relevance

Domain 3. Open Science

Domain 4. Intervention Research

SEER codifies practices that IES expects to be implemented as part of IES-funded research. For each domain, we note whether the Standards are typically required for all IES-funded work, or if they typically apply to only specific types of projects. In all cases, the specific requirements that apply to funded work are those stated in an IES Request for Applications (grants and cooperative agreements) or an IES Request for Proposals (contracts). Questions about the applicability of given Standard should be directed to the cognizant IES Program Officer.

IES has iterated on SEER frequently and expects to continue to do so as we receive feedback from researchers, practitioners, and policymakers. 

To provide feedback, please write us at Contact.IES@ed.gov. 

 

Objectivity and Scientific Integrity

Rationale

IES’s authorizing legislation, the Education Sciences Reform Act of 2002, requires that the activities of the Institute “are objective, secular, neutral, and nonideological and are free of partisan political influence and racial, cultural, gender, or regional bias” (see Section 111 (b)(2)(B)). 

The Department of Education’s Scientific Integrity Policy is designed to “ensure that Scientific Activities conducted and supported by the U.S. Department of Education are of the highest quality and integrity and can be trusted by the public and contribute to sound decision-making.”

IES activities must also comply with all applicable laws and regulations. 

 

SEER Standards Applicable to All IES Research

  • Theory and Hypotheses. The theory or conceptual framework informing a research project’s design should be clearly and plainly articulated. Hypotheses arising from that theory or framework must be testable and falsifiable. The results of that testing should result in the theory or framework being refined, modified, or discarded.
  • Research Design. Research design, sampling, measurement, analysis, interpretation, and dissemination should not be shaped by pre-determined conclusions or goals. Consistent with pre-registered analysis plans, all findings (be they positive, null, or negative) should be accurately reported. 
  • Communication of Uncertainty. When describing results, researchers will be transparent about limitations of their study, and clear about the degree of uncertainty of the findings.
  • Integrity. Researchers and research administrators should adhere to professional ethics and practices when conducting, managing, using the results of, and communicating about science and scientific activities that ensure objectivity, clarity, and honesty, and that provide protection from bias, fabrication, falsification, plagiarism, interference, censorship, and inadequate procedural and research security.
  • Conflicts of Interest. Researchers will disclose at all phases of research potential conflicts of interest they may have related to the program, practice, or policy under study or affiliated personnel, their strategies for managing such conflicts of interest, and how they ensure the objectivity of the research.

 

Relevance

Rationale

IES statistics, research, and evaluation are meant to be relevant to education policy and practice, as well as to researchers and the public (see, for example Sections 133(a)(6) and 151(b)(3)(B) of ESRA). Relevance is critical to IES’s efforts to “promote the use, development, and application of knowledge gained from scientifically valid research activities” (see Section 112(3) of ESRA) and, similarly, to “improve practice in the classroom” (see Section 112(6) of ESRA).

 

SEER Standards Applicable to All IES Research

  • Relevant and Student Outcomes-Focused. Research must be tied to improving student outcomes by:
    • Addressing pressing problems of policy or practice, such as those faced by educators, educational administrators, state and local policymakers, families, or students themselves; or 
    • Advancing fundamental theories of cognition, pedagogy, or the learning sciences; or
    • Advancing high-quality methods of research and knowledge translation.   
  • Engagement. Researchers should describe how they will elicit and consider input from the public (such as educators, policymakers, learners, and families) when conceptualizing, designing, and reporting the results of their research, and when considering issues critical for implementation and scaling of interventions.
  • Generalizability. Researchers must:
    • through intentional sampling (or other means) and appropriate a priori power analysis, design studies that allow for valid estimates to be calculated for groups of interest and that permit ready generalization of its findings to populations of interest; and  
    • document and report the baseline characteristics of their analytic sample. 

 

SEER Standards Applicable Only to Development and Impact Evaluations

  • Design for Scale. Researchers must:
    • design and evaluate practices, programs, and policies with the possibility of future scaling in mind; 
    • explore factors associated with the intervention and its implementation that can inform the impact and sustainability of the intervention at scale, such as its affordability and feasibility; and
    • develop materials that could support the replication, evaluation, and/or scaling of an intervention by others, such as manuals, toolkits, or implementation guides.

 

Resources

 

Open Science

Rationale

A commitment to Open Science principles supports efforts to demonstrate how IES-funded research is scientifically valid (see Section 102(19) and Section (102)(20) of ESRA) as well as efforts to replicate IES-funded work (see Section 102(19)(B)(v) of ESRA). Open access to the knowledge arising from IES research is particularly important, supporting our mission to broadly disseminate scientifically valid education research to stakeholders nationwide (see Section 112(2) of ESRA).  

 

SEER Standards Applicable to All IES Research

  • Pre-registration. Studies must be pre-registered in a recognized study registry, documenting their confirmatory research questions and planned analytic activities. When deviations from pre-registered plans occur, researchers must update their registry entries and provide an explanation for why a change took place.
  • Open Data. Researchers must provide access to the final research data arising from their activities, while protecting the rights of study participants and the confidentiality of the data in a manner consistent with the requirements of the responsible institutional review board (IRB) as well as applicable state and Federal laws and regulations.
    • Note: Consistent with the IES Public Access to Research Policy, grantees and contractors testing the causal impacts of policies, practices, and/or interventions are required to develop a data sharing and management plan (DSMP). DSMPs must specify how final research data will be shared.
  • Open Methods. Researchers must provide detailed descriptions of research activities, including data collection, construct measurement and development, data editing and cleaning, and data analysis. Wherever possible, well-documented statistical code should be made available. 
  • Open Access. Researchers must submit the electronic versions of final manuscripts to an archive site that will make them freely available to others.
    • Note: Consistent with the IES Public Access to Research Policy, IES grantees and contractors must use ERIC for this purpose.

 

Resources

 

Intervention Research

Rationale

Conducting and supporting scientifically-valid research, development, and evaluation is a core IES function (see Sections 102(18) through 102(20) and 112(1) of ESRA). In addition to being characterized by well-designed and well-implemented causal methods, IES expects that intervention research include high-quality information about interventions’ implementation (see Section 102(19)(B) of ESRA), costs, and cost-effectiveness (see, for example, Section 133 of ESRA and Section 4611(a)(2)(B) of the Elementary and Secondary Education Act, as amended).

 

SEER Standards Applicable Only to Impact Evaluations

  • Core components. Researchers must document the components of an intervention, including its essential practices and structural elements. Researchers must offer a clear description of how the components of an intervention are hypothesized to affect outcomes. Researchers should consider exploring which components are most important in achieving impact; that is, which components are core to the intervention's efficacy.
  • Implementation and Treatment Contrast. Researchers must document (1) how, and the context within which, the treatment was implemented; and (2) the counterfactual condition(s), including its context. Researchers must measure (1) the essential elements of the treatment contrast between the treatment and control conditions; and (2) the fidelity of an intervention's implementation.
  • High Quality Measures. Researchers must clearly define the outcome constructs of interest to their study, selecting measures that are valid and reliable assessments of those constructs, are otherwise psychometrically sound, and are appropriate to the context of the intervention and the research participants. Researchers must avoid outcome measures that are overaligned to the intervention being studied. (For more information on overalignment, consult the What Works Clearinghouse Procedures and Standards Handbook.)
  • Distal Outcomes. Researchers must examine both the immediate impact of their intervention on outcomes of interest as well as its impact on relevant distal outcomes and the potential that initial impacts may fade over time.
  • Cost and Cost-Effectiveness Analysis. When intervention impacts are estimated, researchers must conduct cost-effectiveness analyses. Researchers:
    • Must document the type and quantity of resources (e.g., personnel, materials and equipment, facilities, and other inputs) required to implement the interventions they study and document the economic cost of those resources. 
    • Must: (1) identify the perspective from which the cost analysis was conducted (e.g., societal perspective, district perspective) and provide the reason for taking this perspective; (2) clearly state all the assumptions used in estimating costs and offer a justification for those assumptions; (3) describe any adjustments made to prices to amortize costs or to account for inflation, regional price differences, and the time value of money and explain a rationale for those adjustments; and (4) present cost metrics and cost breakdowns that help education decisionmakers understand the resource requirements and costs of implementing interventions.
    • Must justify and clearly describe the alternative condition and in the case of comparative cost-effectiveness analyses the programs and/or programs, against which the intervention is being contrasted.

 

Additional Recommendations

  • Implementation and Treatment Contrast. Researchers should document, and identify opportunities to learn from, adaptations of the intervention that were observed during the implementation of interventions.
  • Assumptions for Economic Cost Analysis. When feasible, researchers should provide a ‘reference case analysis’ in which they adopt a societal perspective (i.e., include costs to all stakeholders), use national average prices, and use a 3 percent discount rate to calculate present values. Even if not otherwise required, researchers should consider measuring the cost of components of the intervention relative to the control or comparison condition.

 

Resources