National Board for Education Sciences 2006 Annual Report

In 1999 the National Research Council (NRC) published a report concluding that, "the complex world of education-unlike defense, health care, or industrial production-does not rest on a strong research base." Three years later, as the nation turned its attention to reforming federal education policy, the legislative branch and the executive branch agreed that the research base described in the NRC report was still woefully inadequate and needed more rigor and relevance.

Congress, to its credit, responded by passing the Education Sciences Reform Act of 2002 (ESRA), establishing the Institute of Education Sciences (the Institute) and its advisory board, the National Board for Education Sciences (NBES). The Senate confirmed the members of the National Board for Education Sciences at the close of the 108th Congress, and the Board held its first meeting in February 2005. ESRA mandates that the Board shall submit an annual report "that assesses the effectiveness of the Institute in carrying out its priorities and mission, especially as they relate to carrying out scientifically valid research, conducting unbiased evaluations, and collecting and reporting accurate education statistics."

For this report, the Board assessed the Institute's progress in three areas: 1) the establishment of research priorities to guide its investments into the next decade; 2) the creation of a rigorous peer review process for grants applications and reports; and 3) the Institute's efforts to ensure its work is useful to practitioners and policymakers.

Here are the main findings in this report:

• Research Priorities
  The Institute has developed clear research priorities that provide a strong, long-term guide for its work.
• Plan for Addressing the Priorities
  The Institute has made great progress in funding projects that will fill important gaps in information.
• Standards and Policies for Peer Review
  The Institute has implemented standards and procedures for reviewing and approving grants and IES reports that are of the highest merit and are comparable to those of the National Science Foundation and the National Institutes of Health.
• Actions Taken to Assure the Relevance of Its Work
  The Institute has initiated a number of activities to assure the relevance of its work for policy and practice.

NBES believes that the vision Congress had when it created the Institute of Education Sciences is being realized, and the Board is privileged to serve during this important period of transformation.

Ex Officio Members
Director of the Institute of Education Sciences
Each of the Commissioners of the National Education Centers
Director of the National Institute of Child Health and Human Development
Director of the Census
Director of the National Science Foundation

In 1996 the National Research Council (NRC) began assessing the feasibility of a long-term research program focused on the topics that would be most critical to improving learning in the nation's schools. When the panel's findings were published in 1999, the NRC concluded that, "the complex world of education-unlike defense, health care, or industrial production-does not rest on a strong research base. In no other field are personal experience and ideology so frequently relied on to make policy choices, and in no other field is the research base so inadequate and little used" (Improving Student Learning: A Strategic Plan for Education Research and Its Utilization, 1999).

Those were tough words, but three years later, as the nation turned its attention to reforming federal education policy, the legislative branch and the executive branch agreed that the research base described in the NRC report was still woefully inadequate and needed more rigor and relevance. Congress, to its credit, responded by passing the Education Sciences Reform Act of 2002 (ESRA), establishing the Institute of Education Sciences (the Institute) and its advisory board, the National Board for Education Sciences (NBES). Through the work of the Senate Committee on Health, Education, Labor and Pensions and the House Committee on Education and the Workforce, as well the House and Senate Appropriation Committees, Congress created a new entity within the Education Department that would make significant strides in the quality and quantity of the research it sponsored. Free of partisan political influence, the Institute would report to Congress yearly on the condition of education in the United States. It would provide thorough and objective evaluations of federal programs such as Title I and Reading First. It would sponsor research that would be relevant and useful to educators and others. And it would become the place to go-the gold standard-for reliable, unbiased information on what works in education-and what does not.

Members of the National Board for Education Sciences were nominated by the President and confirmed by the Senate. The Board was confirmed by the Senate at the close of the 108th Congress, and held its first meeting in February 2005. To date, the Board has met 5 times. NBES is composed of 14 researchers and practitioners who feel privileged to be a part of this groundbreaking attempt to improve the quality of education research. The ESRA mandates that the Board shall submit an annual report "that assesses the effectiveness of the Institute in carrying out its priorities and mission, especially as they relate to carrying out scientifically valid research, conducting unbiased evaluations, and collecting and reporting accurate education statistics." The Board is pleased to submit the attached report in the hopes that it will convey the extent of the progress the Institute has made as it transforms education into an evidence-based field in which decision-makers turn to rigorous data to inform policies and practices.

From its inception, the Institute made it a first priority to develop policies and procedures that foster the highest scientific standards for the research it supports. The Board has played a role in institutionalizing the commitment to these standards by fulfilling its responsibility to formally approve the Director's priorities and the procedures for peer review. To be thoughtful in its deliberations over the priorities and procedures, the Board conducted its own review of the Institute's work. As result, the Board is positioned to report on how the priorities were developed and to assess the plans to implement them. The Board also will provide information about how the peer review process was developed and how it is operating.

A. Background

ESRA 2002 requires that the Director of the Institute propose to the Board long-term research priorities for the Institute, and that upon approval of such priorities, the Director make the Institute's plan for addressing such priorities available for public comment.

B. Development and Approval of the Priorities

In July 2005, Dr. Grover "Russ" Whitehurst, Director of the Institute, developed a proposed set of priorities and published these in the Federal Register to solicit public comment. Over the summer, the Board prepared an analysis of the public comments, and shared it with Institute senior staff. Dr. Whitehurst considered the Board's analysis and the public comments and modified his proposed draft.

During the September 2005 Board meeting, following review and discussion of the revised statement of priorities, the Board approved the Institute priorities. Dr. Whitehurst summarized the overview, goals and targets of the final Institute research priorities in the following way (the entire text can be found in Attachment A):

Overview of Institute Research Priorities

• By providing an independent, scientific base of evidence and promoting and enabling its use, the Institute aims to further the transformation of education into an evidence-based field, and thereby enable the nation to educate all of its students effectively.
• In pursuit of its goals, the Institute will support research, conduct evaluations, and compile statistics in education that conform to rigorous scientific standards, and will disseminate and promote the use of research in ways that are objective, free of bias in their interpretation, and readily accessible.

Goals

• To develop or identify a substantial number of programs, practices, policies, and approaches that enhance academic achievement and that can be widely deployed;
• To identify what does not work and what is problematic or inefficient, and thereby encourage innovation and further research;
• To gain fundamental understanding of the processes that underlie variations in the effectiveness of education programs, practices, policies, and approaches; and
• To develop delivery systems for the results of education research that will be routinely used by policymakers, educators, and the general public when making education decisions.

What and Who

The Institute's over-arching priority is to support research that contributes to improved academic achievement for all students, particularly for those students whose education prospects are hindered by inadequate education services and conditions associated with poverty, race/ethnicity, limited English proficiency, disability, and family circumstance.

C. Board's Assessment of the Effectiveness of the Institute in Developing Its Priorities

The Institute did an excellent job developing its research priorities. The draft prepared for public comment was a sound document, reflecting the Institute's current work. It was more focused than many of the previously developed priority statements. That said, the public and Board comments concentrated on important new areas such as special education and the use of research. The Director and the Institute were appropriately responsive to the comments, and the final draft provides a strong long-term guide.

D. Plan to Address the Priorities

The priorities are a strategic statement for goals and areas of intent but by design they are not a strategic plan. Therefore, once the Board had approved the priorities, the Director set about developing a plan to address the priorities. Logically, the research that the Institute has funded to date is consistent with the Director's priorities. However, because these priorities are intended to guide work over the long term, this plan shows how the Institute will ensure that future work is strategically driven and that gaps are identified and filled. The Institute's primary mechanism for acting on the priorities is through research competitions to which researchers and research organizations can apply for funding. These competitions are organized by topic (e.g., reading and writing), and by center within the Institute (e.g., the National Center for Special Education Research). The plan explains how this mechanism will be used to oversee adherence to the priorities and take corrective action as necessary.

The plan has three components: opportunities for researchers, mix of grant applications, yield of grants. Each component involves an assessment of current activities to determine if action is necessary, and where necessary, to create new research activities or modify existing activities in order for the Institute's portfolio of activities to cover the priorities.

Opportunities for Researchers

The first component involves assessing whether the Institute is providing opportunities for researchers to obtain funding for work on each of the topics identified in the priorities. For example, is the Institute providing funding opportunities for research on the effects of curriculum and instruction on outcomes in mathematics and science outcomes for students in grade K-12? If such an assessment revealed a gap in the topics being funded, the corrective action would be to create new funding opportunities.

The number of topics covered in the competitions has increased each year, and numbered 20 for the 2006 fiscal year. As an example of the topic areas, the table below presents the number of grants awarded by topic area in 2006. 1

Mix of Grant Applications

The second component involves assessing whether the mix of grant applications within each topic is appropriate to the Institute's goals of determining what works and does not work, and understanding the processes that underlie variations in program effectiveness. For example, with respect to the effects of curriculum and instruction on mathematics and science outcomes, is there sufficient upstream work to generate a new generation of programs and practices? Is there sufficient downstream work in moving interventions to scale and evaluating their effectiveness? When the mix of work within a research program is unbalanced, the corrective actions include: clarifying funding announcements, increasing capacity in the research community, and enhancing incentives to pursue particular categories of research.

Yield of grants

The third component involves assessing whether the yield of grants within each topic is advancing the Institute's goals, particularly the goal of developing and identifying programs and practices that are effective in enhancing academic achievement. For example, are grants on the effects of curriculum and instruction on mathematics and science outcomes yielding promising findings that can be field-tested? Are field tests of programs and practices at scale yielding positive effects? When the yield from a research program is insufficient, the Institute can redirect investments and use different forms of funding-such as contracts and cooperative agreements-to jumpstart and actively shape the research portfolio. (The full research plan that was submitted to the Federal Register for public comment can be found in Attachment B.)

E. Board Assessment of the Institute's Plan for Addressing the Priorities

Because the Institute is relatively young, it has focused on developing new opportunities for researchers. In reviewing the plan for addressing the priorities and the grants that were funded in the last year, the Board concludes that the Institute has made great progress in filling the gaps.

The second component, the mix of types of research within research programs, will be addressed for those research topics that have been through at least two cycles of funding. Addressing the third component, yield, is just beginning to be possible at this point. The Board will be reviewing the grants funded by subject each year to assess whether the corrective actions outlined in the plan have been implemented as needed.

A. Background

ESRA requires the Director of the Institute "to establish necessary procedures for technical and scientific peer review of the activities of the Institute" to assure that ''scientifically based research standards'' are applied to the work of the Institute, including the funding of grant applications and the products that are published by the Institute. To that end, the Standards and Review Office was created and staffed within the Institute's Office of the Deputy Director for Science. The Standards and Review Office developed, implemented, and refined the peer review procedures beginning shortly after the enactment of ESRA. The review procedures apply to all reports supported by the Institute, as well as all grant proposals submitted for funding.

B. Peer Review of Products

ESRA requires that "all research, statistics, and evaluation reports conducted by, or supported through, the Institute shall be subjected to rigorous peer review before being published or otherwise made available to the public." In addition, the Act requires that Institute products be "objective, secular, neutral, and non-ideological and are free of partisan political influence and racial, cultural, gender, or regional bias." Locating the Standards and Review Office within the Office of the Deputy Director for Science, independent of any of the Institute's four operating Centers, allows Institute staff responsible for approval and scientific peer review of products (action editors) to maintain an independent, objective point of view. Action editors are senior staff members with several years of experience conducting research, publishing and carrying out scientific reviews. Center staff responsible for the oversight of research projects, research contractors, and report authoring are necessarily exposed to influences and relationships that could easily result in the appearance of conflict of interest in a review process. This carefully engineered organizational independence allows the Institute's products to undergo objective, arms-length peer review, setting a standard for rigor, and placing the Institute on par with the nation's other premier federal science agencies.

Centers submit a report to the Standards and Review Office after they have conducted their own Center-level review, and the relevant Commissioner has approved the report for submission. The relevant Center Commissioner recommends to the Deputy Director for Science whether a report should receive external or internal scientific review, or should be exempt from review. Reports that present new analyses of data are sent to external scientists for peer review. All reports with limited descriptive data analyses are reviewed internally by Standards and Review staff.

For an external scientific review, the action editors identify potential peer reviewers, locating researchers who have published in top research journals in the relevant content area, have used similar methodological approaches, and have substantial experience conducting studies similar to the one presented in the report. The Deputy Director for Science reviews names, publication lists, and examples of published work of potential reviewers, and approves the recruitment of specific reviewers. Action editors recruit at least two approved external reviewers for each report. The external reviewers are asked to focus on the significance of the work for the field of education (the research itself, and the product at-hand); the technical quality of the research design, data collection, and data analyses; the appropriateness of the conclusions; and the clarity of the presentation. They are also asked to assure that any language in the report that advances causal claims is supported by the research methods and analyses described in the report.

Standards and Review Office action editors conduct their own review of each report simultaneously with the external reviewers, and then write a disposition memorandum synthesizing the action editor's review and those of the external reviewers. In their reviews, action editors focus on issues of technical quality, and are also responsible for ensuring that the reports are neutral and objective, and do not contain policy implication statements or recommendations, or statements of advocacy for particular positions, programs, or policies. The disposition memorandum indicates whether or not the report has been approved for publication, or is in need of revision.

For an internal scientific review, the disposition memorandum is based on the action editor's review of the report. For both internal and external reviews, the disposition memorandum is reviewed and approved by the Deputy Director for Science, and then sent to the Center responsible for the report. Standards and Review action editors, in consultation with the Deputy Director for Science, are responsible for reviewing and approving revisions made to the report and for recommending final approval of the report for publication by the Institute.

The Institute understands the importance of a timely review process. In 2005, it devoted an average of 24.9 working days up to the disposition memo (representing a 12% decrease from the previous year); 29.6 days in the Standards and Review Office (a 23% decrease from 2004); and 60.7 total working days until final approval (35% decrease from 2004). The Standards and Review Office is committed to continuing to address a timely review process.

Implementation of the new peer review process has resulted in substantial changes to some Institute products. As with the review process used by top research journals, reports sometimes require more than one round of revisions prior to receiving approval for publication from the Standards and Review Office. Sometimes these revisions involve issues of technical quality, and sometimes the revisions are focused on ensuring that the work is presented in a clear, objective, neutral, and unbiased manner. As an example, all program evaluation reports approved by the new review process will clearly lay out the purpose of the report, the study context, research design, data collection procedures, outcome measures, and data analyses; report the findings in a neutral manner; and leave implications for policy and practice to the reader.

The scientific review process for reviewing products represents a departure from the way in which Department research, statistics, and evaluation products were handled prior to passage of ESRA. While products were subject to various kinds of review prior to publication or public release, there was no systematic, independent scientific peer review of research, statistics, and evaluation products, and there were no requirements that such products be neutral and objective. The new process is intended to mirror the scientific peer review process used by top research journals for the acceptance of research articles for publication.

The Institute's peer review process for reports was initiated in early 2004. In 2004, the Standards and Review Office reviewed 44 reports, and in 2005, 96 reports.

C. Peer Review of Grant Applications

Under ESRA, activities of the Institute that are carried out through grants, contracts, or cooperative agreements, at a minimum, shall be awarded on a competitive basis and, when practical, through a process of peer review. Further, the Director is required to establish a peer review procedure (involving highly qualified individuals with an in-depth knowledge of the subject to be investigated) for reviewing and evaluating all applications for grants and cooperative agreements that exceed $100,000.

The Standards and Review Office is responsible for implementing the scientific peer review of grant applications. As with the peer review of reports, a key provision of the grant application peer review system is intended to put distance between the program officers and administrators within the Institute who administer grant programs, work with grantees, and disseminate the results of research, and those who are responsible for the peer review of applications for funding under those grant programs.

In FY2002, the Institute established a new system for the scientific review of grant applications that is similar to the process of grant application peer review at the National Institutes of Health. Standards and Review Office staff identify and recruit highly qualified reviewers primarily on the basis of the quality of the research they have conducted and published in scientific peer-reviewed journals and the degree to which they are in-depth experts in the relevant research methods and subject matter.

These reviewers are assigned to various panels designated to review applications for similar research topics. Reviewers are asked to consider the significance of the work for the field of education, the quality of the research plan, the quality of the personnel, and the adequacy of the available resources for the project. Two to three primary reviewers provide independent narrative reviews and initial rating scores for each of the four criteria above, as well as an initial overall quality rating for their assigned application. Based on the initial overall scores provided by the primary reviewers, Standards and Review staff prepare a preliminary rank order of the applications assigned to each review panel, and based on this rank ordering, approximately the top 25 applications are identified for discussion by the full panels at the review panel meetings.

In FY2003, the Institute created an entirely electronic application submission and review process. This electronic system allows applicants to submit letters of intent and grant applications on-line. In addition, the electronic system also allows reviewers to access applications assigned to their panel, submit their reviews of applications, view preliminary scores and reviews submitted by other reviewers (after submitting their own reviews), and revise their own narrative comments during the panel review meeting. The electronic system allows Institute staff to closely monitor the progress of the review process, and to quickly calculate preliminary scientific merit scores that are used to triage the top ranked applications for consideration by the full panels at the panel review meetings.

Between FY2002 and FY2005, single session review panels were constituted as needed for each round of review, and each review panel considered one grant competition (topic). In FY2006, the Institute created 5 standing review panels to which panel members may be appointed for multiple, consecutive review sessions, to complement the use of single session review panels as appropriate. The standing panels are composed of principal panel members who serve staggered, 3-year terms, rotating panel members who serve for one review session, and ad hoc members who serve for one review session to review a subset of applications using specialized expertise.

The following table summarizes the growth of the review process over the last five years.

* In 2005, the National Center for Special Education Research (NCSER) was established, and the Standards and Review Office became responsible for the peer review of grant applications from this Center starting in FY2006.

The new peer review system for grants has been in place for sufficient time to consider its yield. Between FY2001 and FY2004, to examine whether the Institute was fulfilling its goal to fund rigorous research, a number of distinguished researchers were asked to rate the overall quality of newly funded research projects. The percent of newly funded research projects rated as being of high quality from FY2001 to FY2004 was 36%, 50%, 70%, and 70%, respectively.

The review processes and expectations for higher quality proposals have improved the quality of the responses from the field. In each of the fiscal years 2002 through 2004, the Institute funded about 8% of the grant applications received. In FY2005, the Institute funded about 12%.

In addition, the Institute is also keeping track of the percentage of new research proposals funded by the Institute's National Center for Education Research (NCER) that receive an average score of excellent or higher from a panel of independent reviewers. From FY2003 to FY2005 the percentage of proposals that received this score were 88%, 97% and 100%, respectively.

D. Board Assessment and Approval of Peer Review Processes

While the process for scientifically reviewing grant proposals was first implemented in 2002 and the Institute's peer review process for reports was initiated in early 2004, the Board did not formally approve these procedures until its September 2005 meeting. This time lag allowed the Board to review extensive documentation of these processes and to assess not only the quality of their design but also the fidelity of their implementation prior to granting formal approval. Board members received grant reviews and multiple iterations of report reviews. Members also attended panel meetings and thus were able to assess the quality of the deliberations.

Many of the Board members are researchers who have received funding from the National Science Foundation (NSF) and/or the National Institutes of Health (NIH) and are familiar with the processes of these federal research agencies. They were impressed with what the Institute had put in place and were able to validate that these processes would assure quality, objectivity, validity, and integrity in scientific publications.

In addition, ex-officio members Arden Bement (NSF) and Duane Alexander (NICHD) publicly acknowledged that the approved procedures were of the highest merit and comparable to those of their agencies. While the processes are exemplary, the Board and Director realize that the key to strong systems is the selection of strong panel members. Members made several recommendations related to panel composition and asked that the Director institute quality assurance procedures to insure that these well-developed processes continue to be implemented with fidelity. The Director has committed to ongoing monitoring and reporting of the quality and compositions of the panels.

Over the last six months, the Director and Institute senior staff have been considering ways to ensure that the Institute's research better inform the difficult questions that education policy-makers and practitioners are facing. In order to be more responsive to these constituencies and improve its dissemination mechanisms, the Institute has undertaken the following efforts:

• Regional Educational Labs
  In the re-competition for 10 Regional Educational Labs, the Institute has added a requirement that they be able to conduct fast response applied research and development projects for the policymakers and practitioners in their region.
• National Research and Development Centers
  Similarly, the Institute has established cooperative agreements with its 10 research and development centers to enable quick turnaround studies to inform pressing policy concerns.
• Practice Guides
  Practice Guides address practitioners' current needs for coherent guidance based on evidence. The guides will be designed to incorporate the best available evidence, allowing for evidence below the standards for scientifically-based research in NCLB and the Institute statute if such evidence is the best available. Practice Guides will clearly label the quality of evidence supporting particular recommendations. The Practice Guides will include both summary recommendations and a supporting narrative.
• Redesign of What Works Clearinghouse (WWC)
  The WWC is currently one of the Institute's most significant dissemination mechanisms. However, it is still somewhat complex for practitioners to interpret the findings from the WWC. The redesign will provide practitioners with concise documents that summarize whether the evidence supporting particular programs and practices is positive, indeterminate, or negative.
  
  The WWC is also changing a threshold for including strong designs for which sample size limitations have previously precluded inclusion in the WWC. This will increase the number of research studies in the database.
• Statewide longitudinal databases
  The Institute is providing technical assistance to states that have received a grant to design, develop and implement statewide longitudinal data systems to efficiently and accurately manage, analyze, disaggregate, and use individual student data.
• Collaborations
  The Institute is embarking on numerous collaborative activities with the Council of the Great City Schools, the Council of Chief State School Officers (CCSSO) and the National Conference of State Legislatures (NCSL). The Institute will be establishing a task force on urban education, composed of superintendents and researchers. The goal is for researchers to start addressing the questions that are important to urban school districts and for districts to buy into the value of conducting rigorous research on their policy initiatives.

Board's Assessment of the Institute's Actions Taken to Assure the Relevance of Its Work

While much of the Institute's focus to date has been on assuring the rigor of the work it supports, the Board is very pleased that the Director and staff are conducting a number of activities to assure the relevance of the work for policy and practice. The items described above are an excellent start and the Board looks forward to additional examples and to the execution and assessment of the current plans.

As this report makes clear, to date the Board has emphasized its statutory requirements to approve the Director's priorities and the Institute's peer review procedures. In the coming year, the Board will continue to monitor the processes that ensure the rigor of the Institute's work, while supporting an ongoing dialog with the field to assure its relevance. In addition, the Board will be discussing a variety of issues that may lead to recommendations for congressional actions in the following areas:

• Demonstration authority for evaluation of components of Elementary and Secondary Education Act (ESEA) that require compliance so that participating schools and districts could vary practices from those required under law in the context of evaluating the effects of those variations
• Requirement for all congressionally mandated or authorized evaluations of Department of Education programs to be carried out by the Institute
• Ways of meeting the privacy requirements of Family Educational Rights and Privacy Act that allow independent research using data collected by local and state education agencies pursuant to ESEA
• Fungibility of the .5% evaluation set aside for evaluation of small programs
• Requirement of longitudinal data for reporting of graduation and other persistence outcomes
• Rationale and funding levels for congressionally mandated national research and development centers
• Reauthorization of ESRA

Overall, we have one overarching recommendation: Since the Institute has and continues to meet the highest standards for promoting rigorous and relevant research, the National Board for Education Sciences recommends that Congress and the Administration support the Institute's role as the primary agency for federal education research.

The National Board for Education Sciences has approved research priorities for the Institute of Education Sciences (IES). Proposed by IES Director Grover (Russ) Whitehurst under terms of the Education Science Reform Act of 2002, the priorities were submitted for public comment this summer and approved at the National Board's meeting September 6-7 in Washington, D.C. The text follows:

The long-term goals associated with the Institute's priorities are fourfold: First, to develop or identify a substantial number of programs, practices, policies, and approaches that enhance academic achievement and that can be widely deployed; second, to identify what does not work and what is problematic or inefficient, and thereby encourage innovation and further research; third, to gain fundamental understanding of the processes that underlie variations in the effectiveness of education programs, practices, policies, and approaches; and fourth, to develop delivery systems for the results of education research that will be routinely used by policymakers, educators, and the general public when making education decisions. By providing an independent, scientific base of evidence and promoting and enabling its use, the Institute aims to further the transformation of education into an evidence-based field, and thereby enable the nation to educate all of its students effectively.

In pursuit of its goals, the Institute will support research, conduct evaluations, and compile statistics in education that conform to rigorous scientific standards, and will disseminate and promote the use of research in ways that are objective, free of bias in their interpretation, and readily accessible.

The Institute's over-arching priority is research that contributes to improved academic achievement for all students, and particularly for those whose education prospects are hindered by inadequate education services and conditions associated with poverty, race/ethnicity, limited English proficiency, disability, and family circumstance.

With academic achievement as the major priority, the Institute will focus on outcomes that differ by periods of education. In the infancy and preschool period, the outcomes of interest will be those that enhance readiness for schooling, for example, language skills, and for infants and toddlers with disabilities, developmental outcomes. In kindergarten through 12th grade, the core academic outcomes of reading and writing (including reading and writing in the disciplines), mathematics, and science will be emphasized, as will the behaviors and social skills that support learning in school and successful transitions to employment, independent living, and post-secondary education. At the post-secondary level, the focus will be on enrollment in and completion of programs that prepare students for successful careers and lives. The same outcomes are emphasized for students with disabilities across each of these periods, and include the functional outcomes that improve educational and transitional results. The acquisition of basic skills by adults with low levels of education is also a priority.

In conducting research on academic outcomes, the Institute will concentrate on conditions within the control of the education system, with the aim of identifying, developing, and validating effective education programs, practices, policies, and approaches as well as understanding the factors that influence variation in their effectiveness such as implementation. Conditions that are of highest priority to the Institute are in the areas of curriculum instruction, assessment (including the identification of students with disabilities), the quality of the education workforce, and the systems and policies that affect these conditions and their interrelationships (for example, accountability systems, delivery mechanisms including technology, and policies that support the ability of parents to improve educational results for their children through such means as choice of education services and provision of school-related learning opportunities in the home).

The successful pursuit of the Institute's goals and priorities requires increased capacity to produce and use rigorous education research. To that end, the Institute's priorities include support of post-doctoral training and interdisciplinary doctoral training in the education sciences, development and refinement of education research methods, and expansion and use for research of longitudinal databases that link student data to information on the conditions that affect outcomes, such as curriculum. To assure increased capacity to use and apply the results of research, the Institute will support systematic reviews of evidence, enhanced access to findings through advanced information systems, and outreach to parents, educators, students, policymakers, and the general public.

These are not exclusive or absolute priorities: To the extent that resources permit and the Institute's priorities are being adequately addressed, the Institute may address other important education issues.

The Education Sciences Reform Act of 2002 requires that the Director of the Institute of Education Sciences (Institute) propose to the National Board for Education Sciences (Board) long-term research priorities for the Institute, and that upon approval of such priorities, the Director make the Institute's plan for addressing such priorities available for public comment. The Board has approved the Director's priorities. This document describes the Institute's plan for addressing those priorities.

Review of Priorities

The Institute's research priorities (http://ies.ed.gov/director/board/priorities.asp) are driven by the goals of developing and identifying programs and practices that are effective in enhancing academic achievement, identifying existing programs and practices that do not work or are inefficient, gaining understanding of the processes that underlie variations in the effectiveness of programs and practices, and delivering results of education research to consumers in ways that will support the use of research evidence when making education decisions.

These goals are to be pursued by research activities that focus on the effects on specific student outcomes of four conditions: curriculum and instruction, assessment, workforce quality, and the systems and policies under which the first three conditions operate. The specific student outcomes of interest vary by age of the student. They are:

• Infancy and preschool
  • readiness for schooling
  • developmental outcomes for infants and toddlers with disabilities
• Kindergarten through 12th grade
  • reading and writing
  • mathematics and science
  • behaviors and social skills that support learning in school and successful transitions, and additional functional outcomes for students with disabilities
• Post-secondary level
  • enrollment in and completion of programs that prepare students for successful careers and lives
  • acquisition of basic skills by adults

Mechanism for Acting on the Priorities

The principal means by which the Institute can act on its research priorities is through research competitions to which researchers and research organizations can apply for funding. These competitions are organized by topic, e.g., reading and writing, and by center within the Institute, e.g., the National Center for Special Education Research. Annual research competitions have been held since the Institute was established in 2002. The number of topics covered in the competitions has increased each year, and numbered 20 for the 2006 fiscal year.

The Plan for Addressing Priorities

The Institute's plan for addressing its research priorities has three elements, each involving an assessment of current activities to determine if new research activities should be created or existing activities should be modified in order for the Institute's portfolio of activities to cover the priorities.

The first element involves assessing whether the Institute is providing opportunities for researchers to obtain funding for work on each of the topics identified in the priorities (where the topics are defined as the intersect of the conditions of influence, outcomes, and age periods listed in the previous section). Is the Institute, for example, providing funding opportunities for research on the effects of curriculum and instruction on outcomes in mathematics and science during the k-12 period? The action associated with this assessment is to create new funding opportunities when gaps in coverage are identified.

The second element involves assessing whether the mix of grant applications within each topic is appropriate to the Institute's goals of determining what works, what doesn't, and understanding the processes that underlie variations in program effectiveness. For example, with respect to the effects of curriculum and instruction on mathematics and science outcomes, is there sufficient upstream work to provide a new generation of programs and practices? Is there sufficient downstream work in moving interventions to scale and evaluating their effectiveness? A variety of actions can be taken when the mix of work within a research program is unbalanced. These include clarifying funding announcements, increasing capacity in the research community, and enhancing the incentives to pursue particular categories of research.

The third element involves assessing whether the yield of grants within each topic is advancing the Institute's goals, particularly the goal of developing and identifying programs and practices that are effective in enhancing academic achievement. For example, are grants on the effects of curriculum and instruction on mathematics and science outcomes yielding promising findings that can be field-tested? Are field tests of programs and practices at scale yielding positive effects? A variety of actions can be taken when the yield from a research program is insufficient. These include redirecting investments and using different forms of funding such as contracts and cooperative agreements to jumpstart and actively shape the research portfolio.

The Institute, established in November of 2002, is only a little over 3 years old. Its grants typically run for terms of 4 years. At this stage of the Institute's life the emphasis has to be on the first element of the plan, assuring that research opportunities exist for each of the Institute's priorities. The second element, the mix of types of research within research programs, can be addressed for those research topics that have been through at least two cycles of funding. Addressing the third element, yield, is just beginning to be possible.

Do research opportunities fit the priorities?

The Director's research priorities were published in the fall of 2005. Those priorities built on research competitions that were already in place. In that context, it should not be surprising that most of the conditions and outcomes identified in the priorities are presently being covered.

Kindergarten through 12th grade: Each of the priorities is being covered for the k-12 period. The effects on student outcomes of curriculum and instruction, assessment, and workforce quality are covered by the Institute's research grants in the following areas:

• reading and writing
• reading, writing, and language development (special education)
• teacher quality in reading and writing
• quality of teachers and other service providers (special education)
• mathematics and science
• mathematics and science (special education)
• teacher quality in math and science
• cognition and student learning
• individualized education programs and individualized family service plans (special education)
• response to instruction (special education)
• high school reform
• interventions for struggling adolescent and adult readers
• socialization and character development
• serious behavior disorders (special education)
• secondary and transition services (special education)
• autism spectrum disorders (special education)

The effects of systems and policies in kindergarten through 12th grade are covered by the following research programs and national research and development centers:

• education leadership
• education policy, finance, and management
• assessment for accountability (special education)
• national assessment of educational progress secondary analysis
• statewide, longitudinal data systems
• national research and development centers on:
  • rural education
  • data-driven reform
  • school choice, competition, and achievement
  • evaluation, standards, and student testing
  • English language learners
  • performance-based incentives
  • analysis of longitudinal data
  • gifted and talented education

Infancy and preschool: Many, but not all, of the priorities are covered during the infancy and preschool period by existing research programs. Each of the teacher quality and subject matter topics listed in the previous section on k-12 education includes the pre-k period. Additional research programs that focus exclusively on infancy and preschool include:

• early intervention, early childhood special education, and assessment for young children with disabilities (special education)
• national research and development center on early childhood development and education
• preschool curriculum evaluation

A notable gap in coverage is special education research on infants and toddlers with disabilities. Accordingly, the Institute has just launched the early intervention, early childhood special education and assessment for young children with disabilities program, listed above, which covers children from birth through age five. In addition, the Institute's new research program on autism spectrum disorders, listed in the previous section on k-12 education, includes the infancy and preschool period.

Postsecondary: The Institute has two research programs that focus exclusively on postsecondary education:

• national research and development center on postsecondary education and training
• postsecondary education

In addition, a number of the research programs listed in the previous section on k-12 education expressly include coverage of the postsecondary period. Specifically, interventions for struggling adolescent and adult readers, mathematics and science, teacher quality in reading and writing, and teacher quality in mathematics and science, include teaching basic skills to adults through adult and vocational education programs and developmental/bridge programs designed to help under-prepared students acquire the skills to succeed in college. The cognition and student learning program includes research on adult education and tertiary education. And the program in secondary and transition services includes the transition from secondary to postsecondary settings for students with disabilities.

However, some areas of research identified by the priorities have been underrepresented in existing research programs. These include research on the effects of different forms of student aid on enrollment and persistence, the development and evaluation of remedial programs and retention programs for students who enter college underprepared, and the development and validation of assessments that capture the value added by baccalaureate programs in preparing students for careers after graduation. The postsecondary education research program, listed above, has just been launched to address these issues.

Is the mix of research appropriate?

The Institute funds research in seven categories: (1) Identification - identify existing programs, practices, and policies that are differentially associated with student outcomes and the factors that may mediate or moderate the effects of these programs, practices, and policies; (2) Development - develop programs, practices, and policies that are potentially effective for improving outcomes; (3) Efficacy - establish the efficacy of fully developed programs, practices, or policies that either have evidence of potential efficacy or are widely used but have not been rigorously evaluated; (4) Scale-up - provide evidence on the effectiveness of programs, practices, and policies implemented at scale; (5) Assessment and Tools - develop or validate data and measurement systems and tools; (6) Training - increase the capacity to conduct rigorous and relevant research through pre-doctoral and postdoctoral training of education scientists; and (7) Centers - carry out a range of research and national leadership on topics specified by Congress.

Categories 1 through 4 are a logical and progressive ordering of research activities towards the goal of developing and identifying programs and practices that are effective in enhancing academic achievement. Category 4, scale-up research, is the logical culmination of research activities that identify and develop promising approaches (categories 1 and 2) and establish their efficacy in small-scale field tests (category 3). While there is no formula for determining the appropriate mix of research across these categories, the Institute wants to see a distribution that has the shape of a triangle, with the base consisting of identification and development activities, the second level representing small-scale field tests, and the apex representing evaluations of programs and practices at scale. Table 1 is a tabulation of applications in each of categories 1 through 4 as a percentage of the total number of applications received across categories 1 through 4 since 2002.

This distribution has the desirable triangular shape, with identification and development representing 60% of applications, small field tests at 31%, and evaluations of scaled-up programs at 9%.

In addition to examining the mix of grant applications across all funding programs, the Institute has examined the mix within four critical research programs that have existed since 2003. The results from this inventory are presented in Table 2.

The distribution of grant applications within research programs suggests two issues the Institute should address in managing these research programs. First, the teacher quality research program is making virtually no use of the identification goal, which typically involves statistical analyses of large databases to model and test associations between conditions such as teacher training and outcomes such as student test scores. With the growth and availability of state databases that incorporate longitudinal data for students and that include individual data on teachers, this is a resource that should be utilized. The Institute intends to publicize the need for this type of research, to work with its state partners in the statewide longitudinal data systems program, and with its research and development center on analysis of longitudinal data to make such research feasible.

A second issue is the absence of scale-up research within the education policy, management, finance, and leadership area. This likely reflects the lack of a substantial base of rigorous research in this field. However, evidence suggests that the effectiveness of scientifically based programs and practices is strongly influenced by the context of management and leadership in the schools and districts in which such programs are implemented. To enhance research in this area, the Institute has just launched a new research program on education leadership, listed previously in the section on k-12 education. The Institute also plans new initiatives that directly involve school districts as partners in research on policies that are of high relevance to schools districts nationwide.

Is research yielding findings that will enhance academic achievement?

As indicated previously, not enough time has passed since the Institute was established for the vast majority of its grants to have progressed to the point at which findings can be inventoried. An exception is one of the Institute's first grant programs, on cognition and student learning. Grants have been made in this area since 2002, and the initial grants are beginning to produce results. This program has been remarkably successful, as indicated by its having been the cover story of the March 2006 issue of the Association for Psychological Science's Observer (APS Observer - How We Learn).

Using educational software that incorporates design principles from cognitive psychology, one team of researchers funded by the cognition and student learning program has demonstrated four-fold increases in children's learning of science and vocabulary terms that are needed for classroom subjects. Another team of researchers has discovered what may prove to be a basic principle of instruction -- that learning is optimal when instruction on content is repeated at from 10 to 20 percent of the interval over which students are expected to retain the material. For example, mathematics content presented during the first month of a 10-month course should be reviewed in the second month of that course for optimal retention on an end-of-course examination. A third team of researchers has demonstrated that giving a test after reading prose material produces a greater benefit on a final test than many additional readings of the material.

Findings such as these, which flow from theoretically grounded, methodologically rigorous research, are going to transform education if the Institute carries out it priorities well. This document lays out a plan to address areas in which insufficient opportunities exist for researchers to address priorities and in which the current mix of applications indicates gaps in important categories of research. This is an initial plan that has been developed through a complete inventory of the Institute's investments as connected to its priorities. The plan is dynamic and will be adjusted annually based on a continuing examination of the opportunities, mix, and yield of the Institute's grant programs.