2012 National Board for Education Sciences Annual Report
Briefing Material for Board Members

The NCER Goal Structure

Education research awards within both NCER and National Center for Special Education Research (NCSER) utilize a 5-part goal structure.  Applicants submit their proposals under one of the five goals. 

The goal structure of the Education Research Grants program divides the research process into stages for both theoretical and practical purposes. Individually, the goals are intended to help focus the work of researchers while together they are intended to cover the range of research, development, and evaluation activities necessary for building a scientific enterprise that can provide solutions to the education problems in our nation. Under the Exploration goal, researchers generate hypotheses about the components and processes involved in learning and instruction and in the operation of education systems and develop models about how they think systems and processes function to bring about education outcomes. Practically, Exploration projects provide the empirical justification for developing or refining an intervention or assessment, or causally evaluating an existing intervention. Under Development and Innovation, investigators build on prior theoretical and empirical work to propose a theory of change for a specific intervention. The intervention, in essence, is an instantiation of the theory. Practically, researchers not only develop the intervention but also show its usability and its feasibility in a real-world education setting, and collect pilot data on its promise for improving student outcomes that may justify the intervention’s evaluation. Efficacy and Replication projects evaluate the impact of specific interventions under ideal conditions. Effectiveness projects assess the impact of specific interventions when implemented under routine practice. Both Efficacy and Replication projects and Effectiveness projects constitute tests of the theory. Results from these studies should inform further theory development and refinement. Practically, evaluations identify which programs and policies actually produce positive effects on student outcomes, which need more work, and which should be discarded.

Education has always produced new ideas, new innovations, and new approaches but only appropriate empirical evaluation can identify those that are in fact improvements. Taken together, work across the various goals should not only yield information on the practical benefits about the effects of specific interventions on education outcomes but also contribute to the bigger picture of scientific knowledge and theory on learning, instruction, and education systems.

The 5 goals are:

• Goal 1: Exploration.  Research supported under the Exploration goal identifies (1) malleable factors that are associated with education outcomes for students (student outcomes) and (2) factors and conditions that may mediate or moderate the relations between malleable factors and student outcomes. This identification is to be done through the analysis of data (collected by the project and/or using a secondary data set) or the meta-analysis of research studies. By malleable factors, the Institute means factors that can be changed by the education system such as children’s behaviors and skills, teachers’ practices, education programs and their components, school or district management practices, or education policies.  Projects under the Exploration goal are to (a) generate hypotheses regarding the potential causal relations between malleable factors and education outcomes, (b) contribute to theories of change for education interventions, (c) contribute to the development of interventions that can improve student outcomes or to identify the conditions that are associated with better implementation of interventions, and (d) identify potentially beneficial interventions.
• Goal 2: Development and Innovation.  Research supported under the Development and Innovation goal develops innovative education interventions and improves existing education interventions that are to produce beneficial impacts on student outcomes when implemented in authentic education delivery settings (e.g., classrooms, schools, districts). IES considers interventions to encompass curricula, instructional approaches, technology, education practices, programs, and policies.  An iterative development process is expected to be used including a cycle of development, implementation, observation, and revision. The cycle is to continue until the interventions can be shown to be usable by the intended end users and feasible for use within the intended authentic delivery setting. A pilot study is done to determine if there is evidence of the promise of the intervention for achieving its intended student outcomes. IES expects that a finding of strong evidence of promise will lead to further research under the Efficacy and Replication goal.
• Goal 3: Efficacy and Replication.  Research supported under the Efficacy and Replication goal determines whether or not fully developed interventions produce a beneficial impact on student outcomes (and the practical importance of that impact) relative to a counterfactual when implemented in authentic education delivery settings. Interventions can be implemented under ideal conditions which may include use of greater implementation support or a more homogeneous sample than would be expected under routine practice. The interventions tested under the Efficacy and Replication goal include newly developed interventions as well as long standing ones in widespread use. The vast majority of the education programs, practices, and policies implemented in U.S. schools have never been rigorously evaluated to determine if they are able to improve student outcomes relative to any other education intervention. Efficacy and Replication projects may provide the first evaluation of an intervention, may evaluate an already evaluated intervention but under a different set of conditions (these conditions can include a change in the sample, or a change in the intervention or how it is implemented), or may follow the longer-term impacts of a previous evaluation. Efficacy and Replication projects are to provide causal analysis, and randomized controlled trials are the favored research design though strong quasi-experimental designs can also be used. Efficacy and Replication projects also examine the fidelity of implementation of the intervention both to determine how feasible the use of the intervention is and to identify the organizational supports, tools, and procedures that may be needed for sufficient implementation of the core components of the intervention. Interventions that are difficult to implement with fidelity under ideal conditions are unlikely to be implemented well when the intervention is implemented under conditions of routine practice.
• Goal 4: Effectiveness.  Research supported under the Effectiveness goal (previously called “Scale-up Evaluation”) determines whether or not fully developed interventions with prior evidence of efficacy produce a beneficial impact on education outcomes for students (student outcomes) relative to a counterfactual when they are implemented under routine practice in authentic education delivery settings. “Routine practice” refers to the type of implementation that would occur if a school or district were to implement the intervention on its own without special support from the developer or research team. Effectiveness projects, like Efficacy and Replication projects, are to provide a causal evaluation of an intervention as well as examine the intervention’s fidelity of implementation. Before an Effectiveness project can be proposed, at least two evaluations of the intervention, that meet the requirements under the Efficacy and Replication goal, must show beneficial and practical impacts on student outcomes. In addition, the evaluation team must be independent from the developer/distributor of the intervention.
• Goal 5: Measurement.  Research supported under the Measurement goal supports (1) the development of new assessments or refinement of existing assessments and the validation of these assessments or (2) the validation of existing assessments for specific purposes, contexts, and populations. Under refinement, IES includes changing existing assessments or changing the delivery of existing assessments in order to increase efficiency, improve measurement, improve accessibility, or provide accommodation for test takers. Proposed assessments must meet the specific content and sample requirements detailed under the topic to which the application is submitted.  Measurement projects include assessments intended to assess students (e.g., for screening, progress monitoring, formative assessment, outcome assessment), education professionals (e.g., credentialing or evaluation of teachers, principals, and related service providers), and/or education systems (e.g., accountability standards). All assessments developed and/or validated must be either directly or indirectly related to measures of student academic outcomes.